2024-04-22T06:25:31.174 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T06:25:31.174 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} --name leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi --verbose -- /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/orig.config.yaml 2024-04-22T06:25:31.231 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:25:31.326 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} email: null first_in_suite: false job_id: '7668137' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4362 sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 sleep_before_teardown: 0 subset: 222/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDqpX1tV8qLz7FHOEAaPXsOUj0+aWaXrJ228ZMBOOU9N1EmTyApBwBxcMNvotZwmp6kHpWAM8HaXOR5PmRjQVs= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+xG0cSY8uVsMvim/zyVATK+HGTebLR/+oGDEHCkRIx9f4OI9qQgyY8f7FpixNJFFKsZ+dqFo3a1l5pEK7rnRw= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhUBvAv2vykhAHKa3RdmGhLlQ+UDmD0N+2VX5/o9iSH01EPOkFlDB0qtr26IK1ElKTOyluERVaJ7jo3IWabb3U= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_05:52:53 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T06:25:31.326 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa; will attempt to use it 2024-04-22T06:25:31.327 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks 2024-04-22T06:25:31.327 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T06:25:31.329 INFO:teuthology.task.internal:Checking packages... 2024-04-22T06:25:31.352 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '86f7587a5a09af35f0895e1a2d08527638fae697' 2024-04-22T06:25:31.352 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T06:25:31.352 INFO:teuthology.packaging:ref: None 2024-04-22T06:25:31.352 INFO:teuthology.packaging:tag: None 2024-04-22T06:25:31.352 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-22T06:25:31.353 INFO:teuthology.packaging:sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:25:31.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-22T06:25:31.514 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.g86f7587a 2024-04-22T06:25:31.516 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T06:25:31.522 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T06:25:31.522 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T06:25:31.560 INFO:teuthology.task.internal:Saving configuration 2024-04-22T06:25:31.581 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T06:25:31.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:25:31.614 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:30.323444', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDqpX1tV8qLz7FHOEAaPXsOUj0+aWaXrJ228ZMBOOU9N1EmTyApBwBxcMNvotZwmp6kHpWAM8HaXOR5PmRjQVs='} 2024-04-22T06:25:31.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:30.325273', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+xG0cSY8uVsMvim/zyVATK+HGTebLR/+oGDEHCkRIx9f4OI9qQgyY8f7FpixNJFFKsZ+dqFo3a1l5pEK7rnRw='} 2024-04-22T06:25:31.666 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:30.327447', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhUBvAv2vykhAHKa3RdmGhLlQ+UDmD0N+2VX5/o9iSH01EPOkFlDB0qtr26IK1ElKTOyluERVaJ7jo3IWabb3U='} 2024-04-22T06:25:31.666 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T06:25:31.672 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-22T06:25:31.672 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-22T06:25:31.672 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-22T06:25:31.672 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T06:25:31.798 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff7f9fc80d0>, signals=[15]) 2024-04-22T06:25:31.799 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T06:25:31.805 INFO:teuthology.task.internal:Opening connections... 2024-04-22T06:25:31.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:25:31.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:25:31.901 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-04-22T06:25:31.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:25:32.001 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-04-22T06:25:32.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:25:32.086 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T06:25:32.092 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-04-22T06:25:32.110 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-04-22T06:25:32.110 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-04-22T06:25:32.168 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-04-22T06:25:32.168 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-04-22T06:25:32.168 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-04-22T06:25:32.168 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:25:32.168 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-04-22T06:25:32.168 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:25:32.169 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:25:32.170 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-04-22T06:25:32.194 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-04-22T06:25:32.212 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-04-22T06:25:32.212 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-04-22T06:25:32.269 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-04-22T06:25:32.269 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-04-22T06:25:32.269 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:25:32.270 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:25:32.270 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-04-22T06:25:32.302 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-04-22T06:25:32.319 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-04-22T06:25:32.320 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:25:32.377 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-04-22T06:25:32.378 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:25:32.378 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:25:32.378 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:25:32.378 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:25:32.378 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:25:32.378 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:25:32.378 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-04-22T06:25:32.402 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T06:25:32.417 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T06:25:32.506 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T06:25:32.507 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:25:32.509 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:25:32.512 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:25:32.528 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T06:25:32.534 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T06:25:32.534 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:25:32.566 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:25:32.568 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:25:32.665 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T06:25:32.759 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T06:25:32.774 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-22T06:25:32.774 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-22T06:25:32.774 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-22T06:25:32.774 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:25:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:25:32.791 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-04-22T06:25:32.849 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:25:32.849 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-437.el9.x86_64 2024-04-22T06:25:32.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-04-22T06:25:36.721 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-04-22T06:25:38.530 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T06:25:42.539 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-22T06:25:44.015 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-04-22T06:25:56.289 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 21 MB 00:12 2024-04-22T06:26:01.305 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T06:26:03.161 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:26:03.161 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:26:03.253 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-22T06:26:03.254 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-04-22T06:26:03.254 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-22T06:26:03.345 DEBUG:teuthology.orchestra.run.smithi079:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:26:03.843 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 06:26:01 AM UTC. 2024-04-22T06:26:03.962 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-22T06:26:03.974 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-22T06:26:03.974 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 255 k 2024-04-22T06:26:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-04-22T06:26:03.976 INFO:teuthology.orchestra.run.smithi079.stderr:Operation aborted. 2024-04-22T06:26:04.032 INFO:teuthology.orchestra.run.smithi079.stdout:Is this ok [y/N]: 2024-04-22T06:26:04.032 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum reinstall -y kernel || true 2024-04-22T06:26:04.525 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:26:01 AM UTC. 2024-04-22T06:26:04.641 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 255 k 2024-04-22T06:26:04.642 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-04-22T06:26:04.643 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-22T06:26:04.982 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T06:26:04.982 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:26:04.982 INFO:teuthology.orchestra.run.smithi079.stdout:Total 750 kB/s | 255 kB 00:00 2024-04-22T06:26:05.003 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-22T06:26:05.179 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-22T06:26:05.179 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-22T06:26:05.184 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-22T06:26:05.184 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-22T06:26:05.306 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-22T06:26:05.383 INFO:teuthology.orchestra.run.smithi079.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:26:05.383 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:26:06.149 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:26:06.149 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:26:06.837 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:26:06.837 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:26:06.837 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalled: 2024-04-22T06:26:06.838 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:26:06.838 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:26:06.838 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-22T06:26:07.028 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:26:07.075 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:26:07.075 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:26:07.075 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:26:07.075 DEBUG:teuthology.parallel:result is None 2024-04-22T06:26:07.075 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T06:26:07.082 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T06:26:07.083 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:26:07.085 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:26:07.088 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:26:07.105 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T06:26:07.112 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T06:26:07.133 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T06:26:07.133 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:26:07.140 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:26:07.145 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:26:07.227 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T06:26:07.236 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T06:26:07.236 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:26:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:07.253 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:26:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:07.269 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:26:07.271 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:26:07.295 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:26:07.307 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:07.327 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:07.332 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:07.341 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:07.349 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:07.361 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:07.363 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T06:26:07.369 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T06:26:07.370 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:26:07.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:26:07.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:26:07.430 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T06:26:07.439 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T06:26:07.440 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:26:07.442 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:26:07.461 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:26:07.488 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:07.541 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:07.595 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:07.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:07.674 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:26:07.674 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:26:07.737 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:07.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:07.823 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:07.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:07.947 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:26:07.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:26:08.011 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:08.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:08.111 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:08.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:08.237 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:26:08.237 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:26:08.302 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-04-22T06:26:08.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-04-22T06:26:08.309 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-04-22T06:26:08.335 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:26:08.341 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:26:08.382 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:26:08.776 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T06:26:08.784 INFO:teuthology.task.internal:Starting timer... 2024-04-22T06:26:08.784 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T06:26:08.836 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T06:26:08.875 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T06:26:08.876 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-04-22T06:26:08.938 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-04-22T06:26:08.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-04-22T06:26:08.994 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-04-22T06:26:08.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-04-22T06:26:09.071 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-04-22T06:26:09.073 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T06:26:09.073 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-04-22T06:26:09.094 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-04-22T06:26:09.095 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-04-22T06:26:09.130 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-04-22T06:26:09.130 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-04-22T06:26:09.161 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-04-22T06:26:09.161 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-04-22T06:26:09.162 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:26:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:09.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:26:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:09.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:26:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:09.241 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T06:26:09.241 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:26:09.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:26:09.296 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:26:09.321 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T06:26:09.334 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T06:26:09.349 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T06:26:09.351 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi190.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T06:30:06.321 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-04-22T06:30:06.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-22T06:30:06.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:30:06.410 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-04-22T06:30:06.498 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-04-22T06:30:06.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-22T06:30:06.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:30:06.581 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-04-22T06:30:06.663 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-22T06:30:06.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-04-22T06:30:06.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:30:06.749 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-04-22T06:30:06.826 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-04-22T06:30:06.826 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T06:30:06.838 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T06:30:06.838 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:30:06.838 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:30:06.841 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:30:06.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:30:06.844 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:30:06.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:30:06.871 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:30:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:30:06.886 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:30:06.888 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:30:06.893 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:30:06.906 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:30:06.911 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-04-22T06:30:06.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-04-22T06:30:06.922 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-04-22T06:30:06.924 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-04-22T06:30:06.929 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-04-22T06:30:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:30:06.937 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:30:06.940 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-04-22T06:30:06.948 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:30:06.950 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:30:06.953 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:30:06.966 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:30:07.003 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:30:07.005 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:30:07.006 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:30:07.007 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-04-22T06:30:07.007 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.007 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.007 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.007 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.009 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:30:07.009 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-04-22T06:30:07.010 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.010 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.010 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.010 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.018 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:30:07.023 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:30:07.023 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-04-22T06:30:07.023 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.023 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.023 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.023 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:07.024 INFO:teuthology.run_tasks:Running task install... 2024-04-22T06:30:07.033 DEBUG:teuthology.task.install:project ceph 2024-04-22T06:30:07.033 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697'}} 2024-04-22T06:30:07.033 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697'} 2024-04-22T06:30:07.033 INFO:teuthology.task.install:Using flavor: default 2024-04-22T06:30:07.039 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T06:30:07.039 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-22T06:30:07.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:30:07.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:30:07.042 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:30:07.042 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:30:07.043 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:30:07.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:30:07.204 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:30:07.204 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:30:07.277 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:30:07.277 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:30:07.292 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:30:07.292 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:30:07.326 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:30:07.326 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:30:07.327 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:30:07.350 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:30:07.351 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:30:07.396 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:30:07.397 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:30:07.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:30:07.415 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:30:07.422 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:30:07.422 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:30:07.474 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:30:07.474 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:30:07.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:30:07.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:30:07.488 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:30:07.499 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:30:07.499 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:30:07.552 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-04-22T06:30:07.554 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-04-22T06:30:07.562 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:30:07.564 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:30:07.626 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-04-22T06:30:07.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-04-22T06:30:07.637 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:30:07.699 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-04-22T06:30:07.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-04-22T06:30:07.776 INFO:teuthology.orchestra.run.smithi079.stdout:48 files removed 2024-04-22T06:30:07.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:30:07.851 INFO:teuthology.orchestra.run.smithi190.stdout:48 files removed 2024-04-22T06:30:07.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:30:07.925 INFO:teuthology.orchestra.run.smithi193.stdout:48 files removed 2024-04-22T06:30:07.952 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:30:08.477 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-04-22T06:30:08.555 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 199 kB/s | 83 kB 00:00 2024-04-22T06:30:08.610 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 209 kB/s | 83 kB 00:00 2024-04-22T06:30:08.911 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-22T06:30:08.943 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-04-22T06:30:08.980 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-22T06:30:09.254 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-22T06:30:09.302 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-04-22T06:30:09.406 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-22T06:30:10.078 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T06:30:12.045 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-04-22T06:30:14.340 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-22T06:30:15.261 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-04-22T06:30:18.815 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-22T06:30:19.439 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 824 kB/s | 8.0 MB 00:09 2024-04-22T06:30:19.970 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.2 MB 00:01 2024-04-22T06:30:20.305 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-04-22T06:30:21.240 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T06:30:21.363 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-04-22T06:30:22.140 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-22T06:30:23.981 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-04-22T06:30:25.850 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.2 MB 00:01 2024-04-22T06:30:27.156 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T06:30:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-22T06:30:29.064 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T06:30:29.256 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:29.257 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:29.268 INFO:teuthology.orchestra.run.smithi079.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:30:29.268 INFO:teuthology.orchestra.run.smithi079.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:30:29.269 INFO:teuthology.orchestra.run.smithi079.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:30:29.270 INFO:teuthology.orchestra.run.smithi079.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:30:29.271 INFO:teuthology.orchestra.run.smithi079.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:30:29.274 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:30:29.274 INFO:teuthology.orchestra.run.smithi079.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:30:29.334 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:30:29.343 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:30:29.344 INFO:teuthology.orchestra.run.smithi079.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:30:29.345 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:30:29.346 INFO:teuthology.orchestra.run.smithi079.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:30:29.347 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:29.348 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:30:29.349 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:30:29.350 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:30:29.351 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:30:29.352 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:30:29.353 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:30:29.354 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout:Install 148 Packages 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-04-22T06:30:29.355 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:30:29.356 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 218 M 2024-04-22T06:30:29.356 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-04-22T06:30:30.615 INFO:teuthology.orchestra.run.smithi079.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-04-22T06:30:30.685 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 21 MB 00:03 2024-04-22T06:30:30.823 INFO:teuthology.orchestra.run.smithi079.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 4.2 MB/s | 890 kB 00:00 2024-04-22T06:30:30.898 INFO:teuthology.orchestra.run.smithi079.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.9 MB/s | 145 kB 00:00 2024-04-22T06:30:31.066 INFO:teuthology.orchestra.run.smithi079.stdout:(4/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 14 MB/s | 2.4 MB 00:00 2024-04-22T06:30:31.147 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:31.148 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:31.158 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:30:31.159 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:30:31.160 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:30:31.161 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:30:31.163 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:30:31.165 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:30:31.165 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:30:31.199 INFO:teuthology.orchestra.run.smithi079.stdout:(5/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 11 MB/s | 1.5 MB 00:00 2024-04-22T06:30:31.224 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:30:31.247 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:30:31.248 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:30:31.249 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:30:31.250 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:30:31.251 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:30:31.252 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:30:31.253 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:30:31.254 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:30:31.255 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:30:31.256 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:30:31.257 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout:Install 148 Packages 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2024-04-22T06:30:31.258 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:30:31.259 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 218 M 2024-04-22T06:30:31.259 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-04-22T06:30:31.466 INFO:teuthology.orchestra.run.smithi079.stdout:(6/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 18 MB/s | 4.7 MB 00:00 2024-04-22T06:30:32.240 INFO:teuthology.orchestra.run.smithi193.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-22T06:30:32.366 INFO:teuthology.orchestra.run.smithi079.stdout:(7/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x86 19 MB/s | 17 MB 00:00 2024-04-22T06:30:32.883 INFO:teuthology.orchestra.run.smithi079.stdout:(8/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-04-22T06:30:32.990 INFO:teuthology.orchestra.run.smithi193.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 1.2 MB/s | 890 kB 00:00 2024-04-22T06:30:33.058 INFO:teuthology.orchestra.run.smithi079.stdout:(9/150): ceph-radosgw-19.0.0-3156.g86f7587a.el9 15 MB/s | 11 MB 00:00 2024-04-22T06:30:33.074 INFO:teuthology.orchestra.run.smithi193.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.7 MB/s | 145 kB 00:00 2024-04-22T06:30:33.083 INFO:teuthology.orchestra.run.smithi079.stdout:(10/150): ceph-selinux-19.0.0-3156.g86f7587a.el 126 kB/s | 25 kB 00:00 2024-04-22T06:30:33.176 INFO:teuthology.orchestra.run.smithi079.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 2.4 MB/s | 225 kB 00:00 2024-04-22T06:30:33.267 INFO:teuthology.orchestra.run.smithi079.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 296 kB/s | 27 kB 00:00 2024-04-22T06:30:33.368 INFO:teuthology.orchestra.run.smithi079.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 316 kB/s | 32 kB 00:00 2024-04-22T06:30:33.643 INFO:teuthology.orchestra.run.smithi079.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 2.7 MB/s | 747 kB 00:00 2024-04-22T06:30:33.768 INFO:teuthology.orchestra.run.smithi079.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 1.3 MB/s | 164 kB 00:00 2024-04-22T06:30:33.894 INFO:teuthology.orchestra.run.smithi079.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 1.0 MB/s | 127 kB 00:00 2024-04-22T06:30:34.177 INFO:teuthology.orchestra.run.smithi079.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 1.7 MB/s | 500 kB 00:00 2024-04-22T06:30:34.199 INFO:teuthology.orchestra.run.smithi193.stdout:(4/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-04-22T06:30:35.057 INFO:teuthology.orchestra.run.smithi193.stdout:(5/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 1.7 MB/s | 1.5 MB 00:00 2024-04-22T06:30:35.591 INFO:teuthology.orchestra.run.smithi193.stdout:(6/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 960 kB/s | 2.4 MB 00:02 2024-04-22T06:30:35.651 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-22T06:30:35.752 INFO:teuthology.orchestra.run.smithi079.stdout:(18/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 3.4 MB/s | 5.4 MB 00:01 2024-04-22T06:30:35.878 INFO:teuthology.orchestra.run.smithi079.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 362 kB/s | 45 kB 00:00 2024-04-22T06:30:35.987 INFO:teuthology.orchestra.run.smithi079.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 1.2 MB/s | 135 kB 00:00 2024-04-22T06:30:36.095 INFO:teuthology.orchestra.run.smithi079.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 1.5 MB/s | 162 kB 00:00 2024-04-22T06:30:36.358 INFO:teuthology.orchestra.run.smithi193.stdout:(7/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-04-22T06:30:36.579 INFO:teuthology.orchestra.run.smithi079.stdout:(22/150): ceph-test-19.0.0-3156.g86f7587a.el9.x 14 MB/s | 49 MB 00:03 2024-04-22T06:30:36.604 INFO:teuthology.orchestra.run.smithi079.stdout:(23/150): python3-rados-19.0.0-3156.g86f7587a.e 630 kB/s | 320 kB 00:00 2024-04-22T06:30:36.762 INFO:teuthology.orchestra.run.smithi079.stdout:(24/150): python3-rbd-19.0.0-3156.g86f7587a.el9 1.6 MB/s | 298 kB 00:00 2024-04-22T06:30:36.787 INFO:teuthology.orchestra.run.smithi079.stdout:(25/150): python3-rgw-19.0.0-3156.g86f7587a.el9 543 kB/s | 100 kB 00:00 2024-04-22T06:30:36.813 INFO:teuthology.orchestra.run.smithi079.stdout:(26/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-22T06:30:36.858 INFO:teuthology.orchestra.run.smithi193.stdout:(8/150): ceph-common-19.0.0-3156.g86f7587a.el9. 4.4 MB/s | 21 MB 00:04 2024-04-22T06:30:36.863 INFO:teuthology.orchestra.run.smithi079.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 3.3 MB/s | 171 kB 00:00 2024-04-22T06:30:36.891 INFO:teuthology.orchestra.run.smithi193.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 759 kB/s | 25 kB 00:00 2024-04-22T06:30:36.913 INFO:teuthology.orchestra.run.smithi079.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 685 kB/s | 34 kB 00:00 2024-04-22T06:30:36.980 INFO:teuthology.orchestra.run.smithi079.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 2.1 MB/s | 142 kB 00:00 2024-04-22T06:30:37.347 INFO:teuthology.orchestra.run.smithi079.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 9.7 MB/s | 3.5 MB 00:00 2024-04-22T06:30:37.481 INFO:teuthology.orchestra.run.smithi079.stdout:(31/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 4.5 MB/s | 3.1 MB 00:00 2024-04-22T06:30:37.623 INFO:teuthology.orchestra.run.smithi079.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 1.7 MB/s | 249 kB 00:00 2024-04-22T06:30:37.740 INFO:teuthology.orchestra.run.smithi079.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 423 kB/s | 49 kB 00:00 2024-04-22T06:30:37.761 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:37.762 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:37.772 INFO:teuthology.orchestra.run.smithi190.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:30:37.773 INFO:teuthology.orchestra.run.smithi190.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:30:37.774 INFO:teuthology.orchestra.run.smithi190.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:30:37.775 INFO:teuthology.orchestra.run.smithi190.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:30:37.776 INFO:teuthology.orchestra.run.smithi190.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:30:37.778 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:30:37.779 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:30:37.831 INFO:teuthology.orchestra.run.smithi079.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.g8 180 kB/s | 16 kB 00:00 2024-04-22T06:30:37.839 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:30:37.849 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:30:37.850 INFO:teuthology.orchestra.run.smithi190.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:30:37.851 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:30:37.852 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:37.853 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:30:37.854 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:30:37.855 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:30:37.856 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:30:37.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:30:37.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-04-22T06:30:37.859 INFO:teuthology.orchestra.run.smithi190.stdout:Install 148 Packages 2024-04-22T06:30:37.860 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-04-22T06:30:37.860 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:30:37.860 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 218 M 2024-04-22T06:30:37.860 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-04-22T06:30:37.957 INFO:teuthology.orchestra.run.smithi079.stdout:(35/150): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-22T06:30:37.982 INFO:teuthology.orchestra.run.smithi079.stdout:(36/150): ceph-volume-19.0.0-3156.g86f7587a.el9 1.8 MB/s | 273 kB 00:00 2024-04-22T06:30:38.049 INFO:teuthology.orchestra.run.smithi079.stdout:(37/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 8.1 MB/s | 763 kB 00:00 2024-04-22T06:30:38.074 INFO:teuthology.orchestra.run.smithi079.stdout:(38/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 199 kB/s | 18 kB 00:00 2024-04-22T06:30:38.233 INFO:teuthology.orchestra.run.smithi193.stdout:(10/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 5.6 MB/s | 11 MB 00:01 2024-04-22T06:30:38.283 INFO:teuthology.orchestra.run.smithi193.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 4.4 MB/s | 225 kB 00:00 2024-04-22T06:30:38.334 INFO:teuthology.orchestra.run.smithi193.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 542 kB/s | 27 kB 00:00 2024-04-22T06:30:38.383 INFO:teuthology.orchestra.run.smithi193.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 637 kB/s | 32 kB 00:00 2024-04-22T06:30:38.509 INFO:teuthology.orchestra.run.smithi193.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 5.8 MB/s | 747 kB 00:00 2024-04-22T06:30:38.567 INFO:teuthology.orchestra.run.smithi193.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 2.7 MB/s | 164 kB 00:00 2024-04-22T06:30:38.618 INFO:teuthology.orchestra.run.smithi193.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 2.5 MB/s | 127 kB 00:00 2024-04-22T06:30:38.686 INFO:teuthology.orchestra.run.smithi190.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-22T06:30:38.718 INFO:teuthology.orchestra.run.smithi193.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 4.9 MB/s | 500 kB 00:00 2024-04-22T06:30:38.850 INFO:teuthology.orchestra.run.smithi079.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 100 kB/s | 80 kB 00:00 2024-04-22T06:30:39.242 INFO:teuthology.orchestra.run.smithi079.stdout:(40/150): ceph-common-19.0.0-3156.g86f7587a.el9 2.4 MB/s | 21 MB 00:08 2024-04-22T06:30:39.386 INFO:teuthology.orchestra.run.smithi190.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 1.2 MB/s | 890 kB 00:00 2024-04-22T06:30:39.428 INFO:teuthology.orchestra.run.smithi193.stdout:(18/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 7.6 MB/s | 5.4 MB 00:00 2024-04-22T06:30:39.478 INFO:teuthology.orchestra.run.smithi193.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 906 kB/s | 45 kB 00:00 2024-04-22T06:30:39.486 INFO:teuthology.orchestra.run.smithi190.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.4 MB/s | 145 kB 00:00 2024-04-22T06:30:39.536 INFO:teuthology.orchestra.run.smithi193.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 2.3 MB/s | 135 kB 00:00 2024-04-22T06:30:39.603 INFO:teuthology.orchestra.run.smithi193.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 2.4 MB/s | 162 kB 00:00 2024-04-22T06:30:39.703 INFO:teuthology.orchestra.run.smithi193.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 3.1 MB/s | 320 kB 00:00 2024-04-22T06:30:39.804 INFO:teuthology.orchestra.run.smithi193.stdout:(23/150): python3-rbd-19.0.0-3156.g86f7587a.el9 2.9 MB/s | 298 kB 00:00 2024-04-22T06:30:39.841 INFO:teuthology.orchestra.run.smithi079.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 454 kB/s | 803 kB 00:01 2024-04-22T06:30:39.862 INFO:teuthology.orchestra.run.smithi193.stdout:(24/150): python3-rgw-19.0.0-3156.g86f7587a.el9 1.7 MB/s | 100 kB 00:00 2024-04-22T06:30:39.921 INFO:teuthology.orchestra.run.smithi193.stdout:(25/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-22T06:30:39.942 INFO:teuthology.orchestra.run.smithi079.stdout:(42/150): libquadmath-11.4.1-3.el9.x86_64.rpm 179 kB/s | 195 kB 00:01 2024-04-22T06:30:40.192 INFO:teuthology.orchestra.run.smithi079.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 274 kB/s | 253 kB 00:00 2024-04-22T06:30:40.420 INFO:teuthology.orchestra.run.smithi190.stdout:(4/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 2.5 MB/s | 2.4 MB 00:00 2024-04-22T06:30:40.580 INFO:teuthology.orchestra.run.smithi193.stdout:(26/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 4.7 MB/s | 3.1 MB 00:00 2024-04-22T06:30:40.671 INFO:teuthology.orchestra.run.smithi193.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-22T06:30:40.759 INFO:teuthology.orchestra.run.smithi079.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 130 kB/s | 106 kB 00:00 2024-04-22T06:30:40.763 INFO:teuthology.orchestra.run.smithi193.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 372 kB/s | 34 kB 00:00 2024-04-22T06:30:40.845 INFO:teuthology.orchestra.run.smithi190.stdout:(5/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-04-22T06:30:40.855 INFO:teuthology.orchestra.run.smithi193.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 1.5 MB/s | 142 kB 00:00 2024-04-22T06:30:40.953 INFO:teuthology.orchestra.run.smithi190.stdout:(6/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-04-22T06:30:40.975 INFO:teuthology.orchestra.run.smithi079.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 179 kB/s | 135 kB 00:00 2024-04-22T06:30:41.509 INFO:teuthology.orchestra.run.smithi079.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 166 kB/s | 125 kB 00:00 2024-04-22T06:30:41.662 INFO:teuthology.orchestra.run.smithi190.stdout:(7/150): ceph-common-19.0.0-3156.g86f7587a.el9. 6.6 MB/s | 21 MB 00:03 2024-04-22T06:30:41.826 INFO:teuthology.orchestra.run.smithi079.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-22T06:30:41.851 INFO:teuthology.orchestra.run.smithi079.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 245 kB/s | 215 kB 00:00 2024-04-22T06:30:41.954 INFO:teuthology.orchestra.run.smithi190.stdout:(8/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 4.2 MB/s | 4.7 MB 00:01 2024-04-22T06:30:41.955 INFO:teuthology.orchestra.run.smithi193.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 3.2 MB/s | 3.5 MB 00:01 2024-04-22T06:30:41.977 INFO:teuthology.orchestra.run.smithi079.stdout:(49/150): python3-cryptography-36.0.1-4.el9.x86 588 kB/s | 1.2 MB 00:02 2024-04-22T06:30:42.029 INFO:teuthology.orchestra.run.smithi190.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 336 kB/s | 25 kB 00:00 2024-04-22T06:30:42.123 INFO:teuthology.orchestra.run.smithi193.stdout:(31/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x8 2.6 MB/s | 17 MB 00:06 2024-04-22T06:30:42.327 INFO:teuthology.orchestra.run.smithi079.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 61 kB/s | 21 kB 00:00 2024-04-22T06:30:42.331 INFO:teuthology.orchestra.run.smithi193.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 1.2 MB/s | 249 kB 00:00 2024-04-22T06:30:42.386 INFO:teuthology.orchestra.run.smithi079.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 200 kB/s | 107 kB 00:00 2024-04-22T06:30:42.406 INFO:teuthology.orchestra.run.smithi193.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 657 kB/s | 49 kB 00:00 2024-04-22T06:30:42.444 INFO:teuthology.orchestra.run.smithi079.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 561 kB/s | 33 kB 00:00 2024-04-22T06:30:42.490 INFO:teuthology.orchestra.run.smithi193.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.g8 198 kB/s | 16 kB 00:00 2024-04-22T06:30:42.528 INFO:teuthology.orchestra.run.smithi079.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 313 kB 00:00 2024-04-22T06:30:42.586 INFO:teuthology.orchestra.run.smithi079.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-22T06:30:42.645 INFO:teuthology.orchestra.run.smithi079.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 192 kB/s | 11 kB 00:00 2024-04-22T06:30:42.703 INFO:teuthology.orchestra.run.smithi079.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-22T06:30:42.732 INFO:teuthology.orchestra.run.smithi193.stdout:(35/150): ceph-volume-19.0.0-3156.g86f7587a.el9 1.1 MB/s | 273 kB 00:00 2024-04-22T06:30:42.762 INFO:teuthology.orchestra.run.smithi079.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 779 kB/s | 45 kB 00:00 2024-04-22T06:30:42.887 INFO:teuthology.orchestra.run.smithi079.stdout:(58/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-22T06:30:43.005 INFO:teuthology.orchestra.run.smithi079.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-22T06:30:43.054 INFO:teuthology.orchestra.run.smithi079.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-04-22T06:30:43.113 INFO:teuthology.orchestra.run.smithi079.stdout:(61/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 813 kB/s | 47 kB 00:00 2024-04-22T06:30:43.146 INFO:teuthology.orchestra.run.smithi190.stdout:(10/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 7.1 MB/s | 11 MB 00:01 2024-04-22T06:30:43.172 INFO:teuthology.orchestra.run.smithi079.stdout:(62/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-22T06:30:43.222 INFO:teuthology.orchestra.run.smithi079.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.8 MB/s | 3.0 MB 00:00 2024-04-22T06:30:43.238 INFO:teuthology.orchestra.run.smithi190.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 2.4 MB/s | 225 kB 00:00 2024-04-22T06:30:43.247 INFO:teuthology.orchestra.run.smithi079.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-22T06:30:43.272 INFO:teuthology.orchestra.run.smithi079.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-22T06:30:43.321 INFO:teuthology.orchestra.run.smithi190.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 324 kB/s | 27 kB 00:00 2024-04-22T06:30:43.324 INFO:teuthology.orchestra.run.smithi193.stdout:(36/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 1.3 MB/s | 763 kB 00:00 2024-04-22T06:30:43.397 INFO:teuthology.orchestra.run.smithi190.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 421 kB/s | 32 kB 00:00 2024-04-22T06:30:43.407 INFO:teuthology.orchestra.run.smithi193.stdout:(37/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 219 kB/s | 18 kB 00:00 2024-04-22T06:30:43.616 INFO:teuthology.orchestra.run.smithi193.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 386 kB/s | 80 kB 00:00 2024-04-22T06:30:43.647 INFO:teuthology.orchestra.run.smithi190.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 2.9 MB/s | 747 kB 00:00 2024-04-22T06:30:43.665 INFO:teuthology.orchestra.run.smithi079.stdout:(66/150): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-04-22T06:30:43.731 INFO:teuthology.orchestra.run.smithi190.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 1.9 MB/s | 164 kB 00:00 2024-04-22T06:30:43.749 INFO:teuthology.orchestra.run.smithi193.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-04-22T06:30:43.765 INFO:teuthology.orchestra.run.smithi079.stdout:(67/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.4 MB/s | 4.1 MB 00:00 2024-04-22T06:30:43.807 INFO:teuthology.orchestra.run.smithi079.stdout:(68/150): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-22T06:30:43.815 INFO:teuthology.orchestra.run.smithi190.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 1.5 MB/s | 127 kB 00:00 2024-04-22T06:30:43.816 INFO:teuthology.orchestra.run.smithi193.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-22T06:30:43.865 INFO:teuthology.orchestra.run.smithi079.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-04-22T06:30:43.875 INFO:teuthology.orchestra.run.smithi193.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-22T06:30:43.891 INFO:teuthology.orchestra.run.smithi079.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-22T06:30:43.965 INFO:teuthology.orchestra.run.smithi190.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 3.3 MB/s | 500 kB 00:00 2024-04-22T06:30:43.967 INFO:teuthology.orchestra.run.smithi193.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-22T06:30:43.991 INFO:teuthology.orchestra.run.smithi079.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-22T06:30:44.036 INFO:teuthology.orchestra.run.smithi193.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 106 kB 00:00 2024-04-22T06:30:44.050 INFO:teuthology.orchestra.run.smithi079.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-22T06:30:44.120 INFO:teuthology.orchestra.run.smithi193.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-22T06:30:44.178 INFO:teuthology.orchestra.run.smithi193.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-22T06:30:44.245 INFO:teuthology.orchestra.run.smithi193.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-22T06:30:44.266 INFO:teuthology.orchestra.run.smithi079.stdout:(73/150): bison-3.7.4-5.el9.x86_64.rpm 387 kB/s | 943 kB 00:02 2024-04-22T06:30:44.333 INFO:teuthology.orchestra.run.smithi079.stdout:(74/150): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-04-22T06:30:44.346 INFO:teuthology.orchestra.run.smithi193.stdout:(47/150): ceph-mgr-diskprediction-local-19.0.0- 3.1 MB/s | 7.4 MB 00:02 2024-04-22T06:30:44.504 INFO:teuthology.orchestra.run.smithi193.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-04-22T06:30:44.554 INFO:teuthology.orchestra.run.smithi193.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-22T06:30:44.597 INFO:teuthology.orchestra.run.smithi193.stdout:(50/150): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-04-22T06:30:44.622 INFO:teuthology.orchestra.run.smithi193.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-04-22T06:30:44.664 INFO:teuthology.orchestra.run.smithi193.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 785 kB/s | 33 kB 00:00 2024-04-22T06:30:44.689 INFO:teuthology.orchestra.run.smithi193.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-22T06:30:44.731 INFO:teuthology.orchestra.run.smithi193.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 438 kB/s | 18 kB 00:00 2024-04-22T06:30:44.773 INFO:teuthology.orchestra.run.smithi193.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 268 kB/s | 11 kB 00:00 2024-04-22T06:30:44.823 INFO:teuthology.orchestra.run.smithi193.stdout:(56/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-04-22T06:30:44.857 INFO:teuthology.orchestra.run.smithi193.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-04-22T06:30:44.874 INFO:teuthology.orchestra.run.smithi193.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-04-22T06:30:44.932 INFO:teuthology.orchestra.run.smithi193.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-22T06:30:44.966 INFO:teuthology.orchestra.run.smithi193.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-22T06:30:44.973 INFO:teuthology.orchestra.run.smithi190.stdout:(18/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 5.3 MB/s | 5.4 MB 00:01 2024-04-22T06:30:44.983 INFO:teuthology.orchestra.run.smithi193.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-04-22T06:30:45.000 INFO:teuthology.orchestra.run.smithi193.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-22T06:30:45.033 INFO:teuthology.orchestra.run.smithi193.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-22T06:30:45.065 INFO:teuthology.orchestra.run.smithi190.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 493 kB/s | 45 kB 00:00 2024-04-22T06:30:45.067 INFO:teuthology.orchestra.run.smithi193.stdout:(64/150): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-04-22T06:30:45.092 INFO:teuthology.orchestra.run.smithi193.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-22T06:30:45.157 INFO:teuthology.orchestra.run.smithi190.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 1.4 MB/s | 135 kB 00:00 2024-04-22T06:30:45.232 INFO:teuthology.orchestra.run.smithi190.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 2.1 MB/s | 162 kB 00:00 2024-04-22T06:30:45.251 INFO:teuthology.orchestra.run.smithi193.stdout:(66/150): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-04-22T06:30:45.318 INFO:teuthology.orchestra.run.smithi193.stdout:(67/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-04-22T06:30:45.324 INFO:teuthology.orchestra.run.smithi190.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 3.4 MB/s | 320 kB 00:00 2024-04-22T06:30:45.376 INFO:teuthology.orchestra.run.smithi193.stdout:(68/150): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-22T06:30:45.410 INFO:teuthology.orchestra.run.smithi193.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-04-22T06:30:45.425 INFO:teuthology.orchestra.run.smithi190.stdout:(23/150): python3-rbd-19.0.0-3156.g86f7587a.el9 2.9 MB/s | 298 kB 00:00 2024-04-22T06:30:45.444 INFO:teuthology.orchestra.run.smithi193.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-22T06:30:45.517 INFO:teuthology.orchestra.run.smithi190.stdout:(24/150): python3-rgw-19.0.0-3156.g86f7587a.el9 1.1 MB/s | 100 kB 00:00 2024-04-22T06:30:45.544 INFO:teuthology.orchestra.run.smithi193.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-22T06:30:45.603 INFO:teuthology.orchestra.run.smithi193.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-22T06:30:45.617 INFO:teuthology.orchestra.run.smithi190.stdout:(25/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 855 kB/s | 85 kB 00:00 2024-04-22T06:30:45.678 INFO:teuthology.orchestra.run.smithi193.stdout:(73/150): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-22T06:30:45.728 INFO:teuthology.orchestra.run.smithi193.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-22T06:30:45.779 INFO:teuthology.orchestra.run.smithi193.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-22T06:30:45.871 INFO:teuthology.orchestra.run.smithi193.stdout:(76/150): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-04-22T06:30:45.896 INFO:teuthology.orchestra.run.smithi193.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-04-22T06:30:45.930 INFO:teuthology.orchestra.run.smithi193.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-04-22T06:30:45.955 INFO:teuthology.orchestra.run.smithi193.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-22T06:30:45.980 INFO:teuthology.orchestra.run.smithi193.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-22T06:30:46.030 INFO:teuthology.orchestra.run.smithi193.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-22T06:30:46.089 INFO:teuthology.orchestra.run.smithi193.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-22T06:30:46.139 INFO:teuthology.orchestra.run.smithi193.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-22T06:30:46.206 INFO:teuthology.orchestra.run.smithi193.stdout:(84/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-04-22T06:30:46.224 INFO:teuthology.orchestra.run.smithi079.stdout:(75/150): python3-devel-3.9.18-3.el9.x86_64.rpm 125 kB/s | 244 kB 00:01 2024-04-22T06:30:46.231 INFO:teuthology.orchestra.run.smithi193.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 461 kB/s | 42 kB 00:00 2024-04-22T06:30:46.257 INFO:teuthology.orchestra.run.smithi193.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-22T06:30:46.283 INFO:teuthology.orchestra.run.smithi079.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 440 kB/s | 26 kB 00:00 2024-04-22T06:30:46.290 INFO:teuthology.orchestra.run.smithi193.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-22T06:30:46.316 INFO:teuthology.orchestra.run.smithi193.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-22T06:30:46.333 INFO:teuthology.orchestra.run.smithi193.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-22T06:30:46.342 INFO:teuthology.orchestra.run.smithi079.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-04-22T06:30:46.358 INFO:teuthology.orchestra.run.smithi193.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 54 kB 00:00 2024-04-22T06:30:46.400 INFO:teuthology.orchestra.run.smithi193.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-22T06:30:46.400 INFO:teuthology.orchestra.run.smithi079.stdout:(78/150): python3-jinja2-2.11.3-5.el9.noarch.rp 120 kB/s | 248 kB 00:02 2024-04-22T06:30:46.434 INFO:teuthology.orchestra.run.smithi190.stdout:(26/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 3.8 MB/s | 3.1 MB 00:00 2024-04-22T06:30:46.442 INFO:teuthology.orchestra.run.smithi193.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-22T06:30:46.460 INFO:teuthology.orchestra.run.smithi079.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-04-22T06:30:46.485 INFO:teuthology.orchestra.run.smithi079.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-22T06:30:46.492 INFO:teuthology.orchestra.run.smithi193.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-22T06:30:46.509 INFO:teuthology.orchestra.run.smithi190.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-22T06:30:46.526 INFO:teuthology.orchestra.run.smithi193.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 1.1 MB/s | 36 kB 00:00 2024-04-22T06:30:46.559 INFO:teuthology.orchestra.run.smithi193.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-22T06:30:46.592 INFO:teuthology.orchestra.run.smithi190.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 411 kB/s | 34 kB 00:00 2024-04-22T06:30:46.610 INFO:teuthology.orchestra.run.smithi193.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-22T06:30:46.643 INFO:teuthology.orchestra.run.smithi193.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2024-04-22T06:30:46.668 INFO:teuthology.orchestra.run.smithi190.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 1.9 MB/s | 142 kB 00:00 2024-04-22T06:30:46.693 INFO:teuthology.orchestra.run.smithi079.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 2.3 MB/s | 6.0 MB 00:02 2024-04-22T06:30:46.694 INFO:teuthology.orchestra.run.smithi193.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-22T06:30:46.718 INFO:teuthology.orchestra.run.smithi079.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-04-22T06:30:46.727 INFO:teuthology.orchestra.run.smithi193.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-04-22T06:30:46.752 INFO:teuthology.orchestra.run.smithi079.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-22T06:30:46.777 INFO:teuthology.orchestra.run.smithi079.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-22T06:30:46.811 INFO:teuthology.orchestra.run.smithi079.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 723 kB/s | 42 kB 00:00 2024-04-22T06:30:46.869 INFO:teuthology.orchestra.run.smithi079.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-22T06:30:46.878 INFO:teuthology.orchestra.run.smithi193.stdout:(100/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-22T06:30:46.895 INFO:teuthology.orchestra.run.smithi193.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 268 kB/s | 45 kB 00:00 2024-04-22T06:30:46.936 INFO:teuthology.orchestra.run.smithi079.stdout:(87/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-22T06:30:46.962 INFO:teuthology.orchestra.run.smithi079.stdout:(88/150): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-04-22T06:30:46.987 INFO:teuthology.orchestra.run.smithi079.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-04-22T06:30:47.012 INFO:teuthology.orchestra.run.smithi079.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-04-22T06:30:47.037 INFO:teuthology.orchestra.run.smithi079.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 710 kB/s | 54 kB 00:00 2024-04-22T06:30:47.071 INFO:teuthology.orchestra.run.smithi079.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 713 kB/s | 42 kB 00:00 2024-04-22T06:30:47.096 INFO:teuthology.orchestra.run.smithi079.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 756 kB/s | 44 kB 00:00 2024-04-22T06:30:47.130 INFO:teuthology.orchestra.run.smithi079.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T06:30:47.153 INFO:teuthology.orchestra.run.smithi193.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-04-22T06:30:47.155 INFO:teuthology.orchestra.run.smithi079.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 613 kB/s | 36 kB 00:00 2024-04-22T06:30:47.189 INFO:teuthology.orchestra.run.smithi079.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-04-22T06:30:47.247 INFO:teuthology.orchestra.run.smithi079.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 990 kB/s | 58 kB 00:00 2024-04-22T06:30:47.281 INFO:teuthology.orchestra.run.smithi079.stdout:(98/150): socat-1.7.4.1-5.el9.x86_64.rpm 2.4 MB/s | 305 kB 00:00 2024-04-22T06:30:47.315 INFO:teuthology.orchestra.run.smithi079.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-22T06:30:47.340 INFO:teuthology.orchestra.run.smithi079.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T06:30:47.373 INFO:teuthology.orchestra.run.smithi079.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-04-22T06:30:47.449 INFO:teuthology.orchestra.run.smithi079.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-04-22T06:30:47.535 INFO:teuthology.orchestra.run.smithi190.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 4.1 MB/s | 3.5 MB 00:00 2024-04-22T06:30:47.662 INFO:teuthology.orchestra.run.smithi193.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-04-22T06:30:47.782 INFO:teuthology.orchestra.run.smithi079.stdout:(103/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi193.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 865 kB/s | 548 kB 00:00 2024-04-22T06:30:47.816 INFO:teuthology.orchestra.run.smithi079.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-04-22T06:30:47.888 INFO:teuthology.orchestra.run.smithi193.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-22T06:30:47.950 INFO:teuthology.orchestra.run.smithi079.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-04-22T06:30:47.975 INFO:teuthology.orchestra.run.smithi079.stdout:(106/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-22T06:30:48.033 INFO:teuthology.orchestra.run.smithi079.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-04-22T06:30:48.059 INFO:teuthology.orchestra.run.smithi079.stdout:(108/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-22T06:30:48.084 INFO:teuthology.orchestra.run.smithi079.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-04-22T06:30:48.121 INFO:teuthology.orchestra.run.smithi193.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 673 kB/s | 309 kB 00:00 2024-04-22T06:30:48.126 INFO:teuthology.orchestra.run.smithi079.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-22T06:30:48.185 INFO:teuthology.orchestra.run.smithi079.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-22T06:30:48.221 INFO:teuthology.orchestra.run.smithi193.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-22T06:30:48.260 INFO:teuthology.orchestra.run.smithi079.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-04-22T06:30:48.285 INFO:teuthology.orchestra.run.smithi079.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-04-22T06:30:48.319 INFO:teuthology.orchestra.run.smithi079.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 740 kB/s | 43 kB 00:00 2024-04-22T06:30:48.336 INFO:teuthology.orchestra.run.smithi079.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-04-22T06:30:48.369 INFO:teuthology.orchestra.run.smithi079.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-22T06:30:48.394 INFO:teuthology.orchestra.run.smithi079.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-22T06:30:48.430 INFO:teuthology.orchestra.run.smithi193.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-04-22T06:30:48.553 INFO:teuthology.orchestra.run.smithi079.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-04-22T06:30:48.563 INFO:teuthology.orchestra.run.smithi193.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 506 kB/s | 67 kB 00:00 2024-04-22T06:30:48.578 INFO:teuthology.orchestra.run.smithi079.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-04-22T06:30:48.585 INFO:teuthology.orchestra.run.smithi190.stdout:(31/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x8 2.2 MB/s | 17 MB 00:07 2024-04-22T06:30:48.612 INFO:teuthology.orchestra.run.smithi079.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 605 kB/s | 35 kB 00:00 2024-04-22T06:30:48.637 INFO:teuthology.orchestra.run.smithi079.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-04-22T06:30:48.696 INFO:teuthology.orchestra.run.smithi079.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-22T06:30:48.710 INFO:teuthology.orchestra.run.smithi190.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 2.0 MB/s | 249 kB 00:00 2024-04-22T06:30:48.746 INFO:teuthology.orchestra.run.smithi079.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 15 MB/s | 2.0 MB 00:00 2024-04-22T06:30:48.771 INFO:teuthology.orchestra.run.smithi079.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-22T06:30:48.777 INFO:teuthology.orchestra.run.smithi190.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 743 kB/s | 49 kB 00:00 2024-04-22T06:30:48.805 INFO:teuthology.orchestra.run.smithi079.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-22T06:30:48.830 INFO:teuthology.orchestra.run.smithi079.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-04-22T06:30:48.852 INFO:teuthology.orchestra.run.smithi190.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.g8 219 kB/s | 16 kB 00:00 2024-04-22T06:30:48.864 INFO:teuthology.orchestra.run.smithi079.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-04-22T06:30:48.889 INFO:teuthology.orchestra.run.smithi079.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-04-22T06:30:48.948 INFO:teuthology.orchestra.run.smithi079.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-22T06:30:48.973 INFO:teuthology.orchestra.run.smithi079.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-22T06:30:48.980 INFO:teuthology.orchestra.run.smithi193.stdout:(110/150): ceph-test-19.0.0-3156.g86f7587a.el9. 4.1 MB/s | 49 MB 00:12 2024-04-22T06:30:49.005 INFO:teuthology.orchestra.run.smithi193.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-04-22T06:30:49.007 INFO:teuthology.orchestra.run.smithi079.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-22T06:30:49.032 INFO:teuthology.orchestra.run.smithi079.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-04-22T06:30:49.055 INFO:teuthology.orchestra.run.smithi190.stdout:(35/150): ceph-mgr-diskprediction-local-19.0.0- 4.9 MB/s | 7.4 MB 00:01 2024-04-22T06:30:49.074 INFO:teuthology.orchestra.run.smithi079.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-22T06:30:49.080 INFO:teuthology.orchestra.run.smithi190.stdout:(36/150): ceph-volume-19.0.0-3156.g86f7587a.el9 1.2 MB/s | 273 kB 00:00 2024-04-22T06:30:49.099 INFO:teuthology.orchestra.run.smithi079.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-04-22T06:30:49.141 INFO:teuthology.orchestra.run.smithi079.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-22T06:30:49.163 INFO:teuthology.orchestra.run.smithi190.stdout:(37/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 219 kB/s | 18 kB 00:00 2024-04-22T06:30:49.166 INFO:teuthology.orchestra.run.smithi079.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-22T06:30:49.172 INFO:teuthology.orchestra.run.smithi193.stdout:(112/150): libarrow-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-04-22T06:30:49.225 INFO:teuthology.orchestra.run.smithi079.stdout:(137/150): libarrow-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-04-22T06:30:49.250 INFO:teuthology.orchestra.run.smithi079.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-04-22T06:30:49.272 INFO:teuthology.orchestra.run.smithi190.stdout:(38/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 3.4 MB/s | 763 kB 00:00 2024-04-22T06:30:49.273 INFO:teuthology.orchestra.run.smithi193.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-04-22T06:30:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-22T06:30:49.298 INFO:teuthology.orchestra.run.smithi193.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 1.8 MB/s | 548 kB 00:00 2024-04-22T06:30:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 780 kB/s | 59 kB 00:00 2024-04-22T06:30:49.326 INFO:teuthology.orchestra.run.smithi079.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-22T06:30:49.352 INFO:teuthology.orchestra.run.smithi079.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-22T06:30:49.377 INFO:teuthology.orchestra.run.smithi079.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-22T06:30:49.381 INFO:teuthology.orchestra.run.smithi190.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 370 kB/s | 80 kB 00:00 2024-04-22T06:30:49.398 INFO:teuthology.orchestra.run.smithi193.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-04-22T06:30:49.402 INFO:teuthology.orchestra.run.smithi079.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-22T06:30:49.424 INFO:teuthology.orchestra.run.smithi193.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-04-22T06:30:49.427 INFO:teuthology.orchestra.run.smithi079.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-22T06:30:49.444 INFO:teuthology.orchestra.run.smithi079.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-04-22T06:30:49.469 INFO:teuthology.orchestra.run.smithi079.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-22T06:30:49.498 INFO:teuthology.orchestra.run.smithi190.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-22T06:30:49.515 INFO:teuthology.orchestra.run.smithi193.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-22T06:30:49.549 INFO:teuthology.orchestra.run.smithi193.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-22T06:30:49.553 INFO:teuthology.orchestra.run.smithi079.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-22T06:30:49.556 INFO:teuthology.orchestra.run.smithi190.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-04-22T06:30:49.599 INFO:teuthology.orchestra.run.smithi193.stdout:(119/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-04-22T06:30:49.624 INFO:teuthology.orchestra.run.smithi193.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-22T06:30:49.650 INFO:teuthology.orchestra.run.smithi193.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-04-22T06:30:49.708 INFO:teuthology.orchestra.run.smithi193.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-04-22T06:30:49.858 INFO:teuthology.orchestra.run.smithi193.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-04-22T06:30:49.951 INFO:teuthology.orchestra.run.smithi193.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-04-22T06:30:49.976 INFO:teuthology.orchestra.run.smithi193.stdout:(125/150): python3-grpcio-tools-1.46.7-10.el9.x 443 kB/s | 144 kB 00:00 2024-04-22T06:30:50.078 INFO:teuthology.orchestra.run.smithi079.stdout:(149/150): librados2-19.0.0-3156.g86f7587a.el9. 5.4 MB/s | 3.4 MB 00:00 2024-04-22T06:30:50.084 INFO:teuthology.orchestra.run.smithi193.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-04-22T06:30:50.193 INFO:teuthology.orchestra.run.smithi193.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 183 kB/s | 20 kB 00:00 2024-04-22T06:30:50.243 INFO:teuthology.orchestra.run.smithi193.stdout:(128/150): python3-grpcio-1.46.7-10.el9.x86_64. 3.3 MB/s | 2.0 MB 00:00 2024-04-22T06:30:50.268 INFO:teuthology.orchestra.run.smithi193.stdout:(129/150): python3-jaraco-functools-3.5.0-2.el9 66 kB/s | 19 kB 00:00 2024-04-22T06:30:50.335 INFO:teuthology.orchestra.run.smithi193.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-04-22T06:30:50.377 INFO:teuthology.orchestra.run.smithi193.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-22T06:30:50.403 INFO:teuthology.orchestra.run.smithi193.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 587 kB/s | 79 kB 00:00 2024-04-22T06:30:50.436 INFO:teuthology.orchestra.run.smithi193.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 572 kB/s | 58 kB 00:00 2024-04-22T06:30:50.487 INFO:teuthology.orchestra.run.smithi193.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-22T06:30:50.512 INFO:teuthology.orchestra.run.smithi193.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-22T06:30:50.537 INFO:teuthology.orchestra.run.smithi193.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 894 kB/s | 90 kB 00:00 2024-04-22T06:30:50.596 INFO:teuthology.orchestra.run.smithi193.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-22T06:30:50.621 INFO:teuthology.orchestra.run.smithi193.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-04-22T06:30:50.646 INFO:teuthology.orchestra.run.smithi193.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-22T06:30:50.688 INFO:teuthology.orchestra.run.smithi193.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-04-22T06:30:50.713 INFO:teuthology.orchestra.run.smithi193.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 387 kB/s | 36 kB 00:00 2024-04-22T06:30:50.737 INFO:teuthology.orchestra.run.smithi079.stdout:(150/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-04-22T06:30:50.739 INFO:teuthology.orchestra.run.smithi193.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-22T06:30:50.740 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:30:50.740 INFO:teuthology.orchestra.run.smithi079.stdout:Total 10 MB/s | 218 MB 00:21 2024-04-22T06:30:50.806 INFO:teuthology.orchestra.run.smithi193.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-22T06:30:50.831 INFO:teuthology.orchestra.run.smithi193.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 765 kB/s | 90 kB 00:00 2024-04-22T06:30:50.856 INFO:teuthology.orchestra.run.smithi193.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-22T06:30:50.898 INFO:teuthology.orchestra.run.smithi193.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-04-22T06:30:50.932 INFO:teuthology.orchestra.run.smithi193.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-22T06:30:51.199 INFO:teuthology.orchestra.run.smithi193.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-04-22T06:30:51.698 INFO:teuthology.orchestra.run.smithi190.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 115 kB/s | 253 kB 00:02 2024-04-22T06:30:51.807 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-04-22T06:30:51.932 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-04-22T06:30:51.932 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-04-22T06:30:52.641 INFO:teuthology.orchestra.run.smithi193.stdout:(149/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-04-22T06:30:53.374 INFO:teuthology.orchestra.run.smithi193.stdout:(150/150): librados2-19.0.0-3156.g86f7587a.el9. 1.4 MB/s | 3.4 MB 00:02 2024-04-22T06:30:53.378 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:30:53.378 INFO:teuthology.orchestra.run.smithi193.stdout:Total 9.8 MB/s | 218 MB 00:22 2024-04-22T06:30:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-04-22T06:30:53.660 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-04-22T06:30:53.681 INFO:teuthology.orchestra.run.smithi190.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 54 kB/s | 106 kB 00:01 2024-04-22T06:30:53.748 INFO:teuthology.orchestra.run.smithi190.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 299 kB/s | 1.2 MB 00:04 2024-04-22T06:30:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 853 kB/s | 135 kB 00:00 2024-04-22T06:30:53.865 INFO:teuthology.orchestra.run.smithi190.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-04-22T06:30:53.899 INFO:teuthology.orchestra.run.smithi190.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-22T06:30:54.174 INFO:teuthology.orchestra.run.smithi190.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-04-22T06:30:54.258 INFO:teuthology.orchestra.run.smithi190.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-22T06:30:54.316 INFO:teuthology.orchestra.run.smithi190.stdout:(50/150): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 943 kB 00:00 2024-04-22T06:30:54.342 INFO:teuthology.orchestra.run.smithi190.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 257 kB/s | 21 kB 00:00 2024-04-22T06:30:54.406 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-04-22T06:30:54.409 INFO:teuthology.orchestra.run.smithi190.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-04-22T06:30:54.450 INFO:teuthology.orchestra.run.smithi190.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-22T06:30:54.533 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-04-22T06:30:54.533 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-04-22T06:30:54.534 INFO:teuthology.orchestra.run.smithi190.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 218 kB/s | 18 kB 00:00 2024-04-22T06:30:54.626 INFO:teuthology.orchestra.run.smithi190.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-22T06:30:54.701 INFO:teuthology.orchestra.run.smithi190.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-22T06:30:54.777 INFO:teuthology.orchestra.run.smithi190.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 604 kB/s | 45 kB 00:00 2024-04-22T06:30:55.027 INFO:teuthology.orchestra.run.smithi190.stdout:(58/150): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-04-22T06:30:55.277 INFO:teuthology.orchestra.run.smithi190.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 2.3 MB/s | 582 kB 00:00 2024-04-22T06:30:55.352 INFO:teuthology.orchestra.run.smithi190.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 339 kB/s | 25 kB 00:00 2024-04-22T06:30:55.428 INFO:teuthology.orchestra.run.smithi190.stdout:(61/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 633 kB/s | 47 kB 00:00 2024-04-22T06:30:55.553 INFO:teuthology.orchestra.run.smithi190.stdout:(62/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-22T06:30:55.695 INFO:teuthology.orchestra.run.smithi190.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-04-22T06:30:55.720 INFO:teuthology.orchestra.run.smithi190.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-04-22T06:30:55.762 INFO:teuthology.orchestra.run.smithi190.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 524 kB/s | 35 kB 00:00 2024-04-22T06:30:56.242 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-04-22T06:30:56.242 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-04-22T06:30:56.654 INFO:teuthology.orchestra.run.smithi190.stdout:(66/150): openblas-openmp-0.3.21-2.el9.x86_64.r 5.1 MB/s | 4.7 MB 00:00 2024-04-22T06:30:56.871 INFO:teuthology.orchestra.run.smithi190.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-22T06:30:56.955 INFO:teuthology.orchestra.run.smithi190.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.5 MB/s | 4.1 MB 00:01 2024-04-22T06:30:57.022 INFO:teuthology.orchestra.run.smithi190.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-22T06:30:57.047 INFO:teuthology.orchestra.run.smithi190.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-22T06:30:57.197 INFO:teuthology.orchestra.run.smithi190.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-04-22T06:30:57.264 INFO:teuthology.orchestra.run.smithi190.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-22T06:30:57.948 INFO:teuthology.orchestra.run.smithi190.stdout:(73/150): python3-babel-2.9.1-2.el9.noarch.rpm 8.7 MB/s | 6.0 MB 00:00 2024-04-22T06:30:58.023 INFO:teuthology.orchestra.run.smithi190.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-22T06:30:58.107 INFO:teuthology.orchestra.run.smithi190.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-22T06:30:58.182 INFO:teuthology.orchestra.run.smithi190.stdout:(76/150): postgresql-server-13.14-1.el9.x86_64. 5.0 MB/s | 5.8 MB 00:01 2024-04-22T06:30:58.207 INFO:teuthology.orchestra.run.smithi190.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-22T06:30:58.241 INFO:teuthology.orchestra.run.smithi190.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-04-22T06:30:58.274 INFO:teuthology.orchestra.run.smithi190.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-22T06:30:58.291 INFO:teuthology.orchestra.run.smithi190.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-04-22T06:30:58.366 INFO:teuthology.orchestra.run.smithi190.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-22T06:30:58.442 INFO:teuthology.orchestra.run.smithi190.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-22T06:30:58.525 INFO:teuthology.orchestra.run.smithi190.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 929 kB/s | 77 kB 00:00 2024-04-22T06:30:58.601 INFO:teuthology.orchestra.run.smithi190.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 563 kB/s | 42 kB 00:00 2024-04-22T06:30:58.684 INFO:teuthology.orchestra.run.smithi190.stdout:(85/150): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-04-22T06:30:58.776 INFO:teuthology.orchestra.run.smithi190.stdout:(86/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-22T06:30:58.801 INFO:teuthology.orchestra.run.smithi190.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-22T06:30:58.852 INFO:teuthology.orchestra.run.smithi190.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-22T06:30:58.960 INFO:teuthology.orchestra.run.smithi190.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 322 kB/s | 51 kB 00:00 2024-04-22T06:30:59.119 INFO:teuthology.orchestra.run.smithi190.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 201 kB/s | 54 kB 00:00 2024-04-22T06:30:59.178 INFO:teuthology.orchestra.run.smithi190.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 717 kB/s | 42 kB 00:00 2024-04-22T06:30:59.236 INFO:teuthology.orchestra.run.smithi190.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 760 kB/s | 44 kB 00:00 2024-04-22T06:30:59.303 INFO:teuthology.orchestra.run.smithi190.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-22T06:30:59.362 INFO:teuthology.orchestra.run.smithi190.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-04-22T06:30:59.420 INFO:teuthology.orchestra.run.smithi190.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-04-22T06:30:59.438 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-04-22T06:30:59.487 INFO:teuthology.orchestra.run.smithi190.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-22T06:30:59.513 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:30:59.547 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:30:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 867 kB/s | 58 kB 00:00 2024-04-22T06:30:59.589 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:30:59.613 INFO:teuthology.orchestra.run.smithi190.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-22T06:30:59.672 INFO:teuthology.orchestra.run.smithi190.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T06:30:59.730 INFO:teuthology.orchestra.run.smithi190.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 768 kB/s | 45 kB 00:00 2024-04-22T06:30:59.830 INFO:teuthology.orchestra.run.smithi190.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-04-22T06:30:59.918 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:31:00.048 INFO:teuthology.orchestra.run.smithi190.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-22T06:31:00.056 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:00.160 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:00.199 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:00.256 INFO:teuthology.orchestra.run.smithi190.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-04-22T06:31:00.298 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:00.341 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:00.370 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:31:00.396 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:31:00.550 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:31:00.601 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:31:00.629 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:00.731 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:00.755 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:00.817 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:00.839 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:00.865 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:01.065 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:01.097 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:01.115 INFO:teuthology.orchestra.run.smithi190.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 360 kB/s | 309 kB 00:00 2024-04-22T06:31:01.204 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:31:01.206 INFO:teuthology.orchestra.run.smithi190.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 473 kB/s | 548 kB 00:01 2024-04-22T06:31:01.232 INFO:teuthology.orchestra.run.smithi190.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-04-22T06:31:01.303 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:31:01.338 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:31:01.340 INFO:teuthology.orchestra.run.smithi190.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-22T06:31:01.373 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:31:01.448 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:31:01.449 INFO:teuthology.orchestra.run.smithi190.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-22T06:31:01.506 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:31:01.549 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:31:01.581 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:31:01.609 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:31:01.649 INFO:teuthology.orchestra.run.smithi190.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2024-04-22T06:31:01.689 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:31:01.750 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:31:01.779 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:31:01.834 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:31:01.917 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:31:01.951 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:31:01.994 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:31:01.995 INFO:teuthology.orchestra.run.smithi190.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 438 kB/s | 151 kB 00:00 2024-04-22T06:31:02.022 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:31:02.096 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:31:02.154 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:31:02.230 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:31:02.369 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:31:02.394 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-04-22T06:31:02.424 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:31:02.468 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:31:02.501 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:31:02.544 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:31:02.881 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:31:02.979 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:31:02.994 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:03.071 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:31:03.105 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:31:03.115 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:03.145 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:03.148 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:31:03.186 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:31:03.219 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:31:03.236 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:03.276 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:31:03.279 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:03.307 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:31:03.333 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:31:03.350 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:31:03.350 INFO:teuthology.orchestra.run.smithi190.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 619 kB/s | 838 kB 00:01 2024-04-22T06:31:03.386 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:31:03.423 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:31:03.471 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:31:03.492 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:31:03.506 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:31:03.546 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:31:03.549 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:31:03.575 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:03.592 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:31:03.679 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:03.709 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:03.730 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:31:03.755 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:03.777 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:03.780 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:31:03.801 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:03.810 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:31:03.852 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:31:03.949 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:31:03.976 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:31:04.002 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:04.016 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:31:04.035 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:04.042 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:31:04.066 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:31:04.143 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:31:04.143 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:31:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 658 kB/s | 548 kB 00:00 2024-04-22T06:31:04.242 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:31:04.270 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:31:04.284 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:31:04.300 INFO:teuthology.orchestra.run.smithi190.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-04-22T06:31:04.307 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:31:04.319 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:31:04.335 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:31:04.370 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:31:04.395 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:31:04.422 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:31:04.435 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:31:04.450 INFO:teuthology.orchestra.run.smithi190.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 215 kB/s | 32 kB 00:00 2024-04-22T06:31:04.460 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:31:04.488 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:31:04.499 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:31:04.529 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:31:04.529 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:31:04.556 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:31:04.563 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:31:04.597 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:31:04.628 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:31:04.639 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:31:04.667 INFO:teuthology.orchestra.run.smithi190.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 69 kB/s | 15 kB 00:00 2024-04-22T06:31:04.673 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:31:04.688 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:31:04.717 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:31:04.762 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:31:04.792 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:31:04.821 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:31:04.868 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:31:04.915 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:31:04.976 INFO:teuthology.orchestra.run.smithi190.stdout:(116/150): python3-cheroot-8.6.0-4.el9.noarch.r 557 kB/s | 172 kB 00:00 2024-04-22T06:31:05.024 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:31:05.043 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:31:05.078 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:31:05.080 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:31:05.131 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:31:05.160 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:31:05.232 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:31:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:31:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:31:05.489 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:31:05.561 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:31:05.592 INFO:teuthology.orchestra.run.smithi190.stdout:(117/150): python3-cherrypy-18.6.1-2.el9.noarch 581 kB/s | 358 kB 00:00 2024-04-22T06:31:05.832 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:31:05.883 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:31:05.916 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:31:06.020 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:31:06.051 INFO:teuthology.orchestra.run.smithi190.stdout:(118/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 661 kB/s | 303 kB 00:00 2024-04-22T06:31:06.084 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:31:06.114 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:31:06.168 INFO:teuthology.orchestra.run.smithi190.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 303 kB/s | 35 kB 00:00 2024-04-22T06:31:06.172 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:31:06.199 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:31:06.210 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:31:06.235 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:31:06.286 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:31:06.341 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:31:06.374 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:31:06.386 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:31:06.416 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:31:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:31:06.493 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:31:06.503 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:31:06.541 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:31:06.543 INFO:teuthology.orchestra.run.smithi190.stdout:(120/150): python3-google-auth-2.28.2-1.el9.noa 541 kB/s | 203 kB 00:00 2024-04-22T06:31:06.578 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:31:06.626 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:31:06.661 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:31:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:31:06.755 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:31:06.893 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:31:06.943 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:31:06.973 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:31:07.015 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:31:07.111 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:31:07.139 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:31:07.159 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:31:07.180 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:31:07.213 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:31:07.238 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:31:07.283 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:31:07.314 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:31:07.410 INFO:teuthology.orchestra.run.smithi190.stdout:(121/150): libarrow-9.0.0-10.el9.x86_64.rpm 729 kB/s | 4.4 MB 00:06 2024-04-22T06:31:07.450 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:31:07.487 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:31:07.515 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:31:07.549 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:31:07.593 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:31:07.639 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:31:07.671 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:31:07.685 INFO:teuthology.orchestra.run.smithi190.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 525 kB/s | 144 kB 00:00 2024-04-22T06:31:07.700 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:31:07.735 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:31:07.768 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:31:07.794 INFO:teuthology.orchestra.run.smithi190.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-22T06:31:07.810 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:31:07.861 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:31:07.903 INFO:teuthology.orchestra.run.smithi190.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-22T06:31:07.949 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:31:07.992 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:31:08.028 INFO:teuthology.orchestra.run.smithi190.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-04-22T06:31:08.031 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:31:08.079 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:31:08.162 INFO:teuthology.orchestra.run.smithi190.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-04-22T06:31:08.187 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:31:08.241 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:31:08.295 INFO:teuthology.orchestra.run.smithi190.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-04-22T06:31:08.679 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:31:08.770 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:31:08.798 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:31:08.831 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:31:08.985 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:31:09.038 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:31:09.071 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:31:09.144 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:31:09.174 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:31:09.210 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:09.230 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:31:09.300 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:09.319 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:31:09.331 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:31:09.367 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:31:09.373 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:31:09.404 INFO:teuthology.orchestra.run.smithi190.stdout:(128/150): python3-grpcio-1.46.7-10.el9.x86_64. 729 kB/s | 2.0 MB 00:02 2024-04-22T06:31:09.538 INFO:teuthology.orchestra.run.smithi190.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 348 kB/s | 46 kB 00:00 2024-04-22T06:31:09.552 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:31:09.579 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:31:09.666 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:31:09.762 INFO:teuthology.orchestra.run.smithi190.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 351 kB/s | 79 kB 00:00 2024-04-22T06:31:09.847 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:31:09.879 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:09.951 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:09.979 INFO:teuthology.orchestra.run.smithi190.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 266 kB/s | 58 kB 00:00 2024-04-22T06:31:09.981 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:31:10.013 INFO:teuthology.orchestra.run.smithi190.stdout:(132/150): python3-kubernetes-26.1.0-2.el9.noar 609 kB/s | 1.0 MB 00:01 2024-04-22T06:31:10.138 INFO:teuthology.orchestra.run.smithi190.stdout:(133/150): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-04-22T06:31:10.326 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:31:10.363 INFO:teuthology.orchestra.run.smithi190.stdout:(134/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 400 kB/s | 90 kB 00:00 2024-04-22T06:31:10.477 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:31:10.497 INFO:teuthology.orchestra.run.smithi190.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 212 kB/s | 28 kB 00:00 2024-04-22T06:31:10.522 INFO:teuthology.orchestra.run.smithi190.stdout:(136/150): python3-pecan-1.4.2-3.el9.noarch.rpm 502 kB/s | 272 kB 00:00 2024-04-22T06:31:10.631 INFO:teuthology.orchestra.run.smithi190.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 230 kB/s | 31 kB 00:00 2024-04-22T06:31:10.873 INFO:teuthology.orchestra.run.smithi190.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 245 kB/s | 59 kB 00:00 2024-04-22T06:31:11.006 INFO:teuthology.orchestra.run.smithi190.stdout:(139/150): python3-tempora-5.0.0-2.el9.noarch.r 268 kB/s | 36 kB 00:00 2024-04-22T06:31:11.032 INFO:teuthology.orchestra.run.smithi190.stdout:(140/150): python3-routes-2.5.1-5.el9.noarch.rp 370 kB/s | 188 kB 00:00 2024-04-22T06:31:11.257 INFO:teuthology.orchestra.run.smithi190.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 204 kB/s | 51 kB 00:00 2024-04-22T06:31:11.499 INFO:teuthology.orchestra.run.smithi190.stdout:(142/150): python3-websocket-client-1.2.3-2.el9 371 kB/s | 90 kB 00:00 2024-04-22T06:31:11.524 INFO:teuthology.orchestra.run.smithi190.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 467 kB/s | 230 kB 00:00 2024-04-22T06:31:11.658 INFO:teuthology.orchestra.run.smithi190.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-04-22T06:31:11.745 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:31:11.824 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:31:11.852 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:31:11.885 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:31:12.142 INFO:teuthology.orchestra.run.smithi190.stdout:(145/150): re2-20211101-3.el9.x86_64.rpm 396 kB/s | 192 kB 00:00 2024-04-22T06:31:12.196 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:31:12.240 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:12.284 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:12.330 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:12.333 INFO:teuthology.orchestra.run.smithi190.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 512 kB/s | 427 kB 00:00 2024-04-22T06:31:12.335 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:12.361 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:31:12.397 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:31:12.883 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:31:12.887 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:12.909 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:12.930 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:31:13.024 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:13.024 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:31:13.024 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:31:13.024 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:13.056 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:14.583 INFO:teuthology.orchestra.run.smithi190.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 663 kB/s | 1.6 MB 00:02 2024-04-22T06:31:15.277 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:15.315 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:15.450 INFO:teuthology.orchestra.run.smithi190.stdout:(148/150): librados2-19.0.0-3156.g86f7587a.el9. 1.1 MB/s | 3.4 MB 00:03 2024-04-22T06:31:15.869 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:15.910 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:15.967 INFO:teuthology.orchestra.run.smithi190.stdout:(149/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 2.3 MB/s | 3.1 MB 00:01 2024-04-22T06:31:15.990 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:15.990 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:31:15.990 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:31:15.990 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:16.020 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-04-22T06:31:24.584 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-04-22T06:31:24.585 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-04-22T06:31:24.585 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:24.759 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:31:24.774 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:31:24.774 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:24.774 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:31:24.774 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:25.899 INFO:teuthology.orchestra.run.smithi190.stdout:(150/150): ceph-test-19.0.0-3156.g86f7587a.el9. 1.1 MB/s | 49 MB 00:43 2024-04-22T06:31:25.904 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:31:25.904 INFO:teuthology.orchestra.run.smithi190.stdout:Total 4.5 MB/s | 218 MB 00:48 2024-04-22T06:31:26.236 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:31:26.262 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:31:26.262 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:26.262 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:31:26.262 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:31:26.262 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:31:26.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:26.487 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:31:26.510 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:31:26.510 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:26.510 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:31:26.510 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:31:26.510 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:31:26.510 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-04-22T06:31:26.954 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:31:26.977 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:31:26.977 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:26.977 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:31:26.977 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:31:26.977 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:31:26.977 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-04-22T06:31:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-04-22T06:31:27.085 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:31:27.142 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:31:27.197 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:31:27.206 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:27.206 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-04-22T06:31:27.206 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-04-22T06:31:27.207 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-04-22T06:31:27.207 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-04-22T06:31:27.207 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-04-22T06:31:27.207 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-04-22T06:31:27.207 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-04-22T06:31:27.207 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:27.380 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:31:27.392 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:31:27.395 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:31:27.395 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:27.395 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:31:27.395 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:27.438 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:31:27.517 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:31:27.606 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:31:27.755 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:31:27.797 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:31:27.829 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:31:27.856 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:31:28.259 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:31:28.289 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:31:28.320 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:31:28.382 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:31:28.411 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:31:28.795 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-04-22T06:31:28.796 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-04-22T06:31:28.866 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:31:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:31:28.892 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:28.892 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:31:28.892 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:31:28.892 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:31:28.892 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:29.117 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:31:29.139 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:31:29.139 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:29.140 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:31:29.140 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:31:29.140 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:31:29.140 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:29.467 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:31:29.505 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:31:29.576 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:31:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:31:29.598 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:29.598 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:31:29.598 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:31:29.598 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:31:29.598 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:29.698 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:31:29.755 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:31:29.810 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:31:30.005 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:31:30.051 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:31:30.129 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:31:30.235 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:31:30.375 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:31:30.418 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:31:30.450 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:31:30.478 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:31:30.479 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:31:30.505 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:31:30.609 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:31:30.768 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:31:30.807 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:31:30.830 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:31:30.830 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:30.830 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:31:30.830 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:31:30.830 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:31:30.830 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:30.874 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:31:30.886 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:31:30.887 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:31:30.918 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:31:30.950 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:31:30.985 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:31:31.010 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:31:31.031 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:31:31.040 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:31:31.961 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:31:31.993 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:31:32.016 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:31:32.016 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:32.016 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:31:32.016 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:31:32.017 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:31:32.017 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:32.078 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:31:32.078 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:31:32.099 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:31:32.099 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:32.100 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:31:32.100 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:32.118 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:31:32.153 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:31:32.175 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:31:32.175 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:32.175 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:31:32.175 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:31:32.176 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:31:32.176 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:32.478 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:31:32.501 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:31:32.502 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:32.502 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:31:32.502 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:31:32.502 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:31:32.502 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:33.082 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:31:33.118 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:31:33.220 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:31:33.390 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:31:33.420 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:31:33.443 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:31:33.443 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:33.444 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:31:33.444 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:31:33.444 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:31:33.444 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:33.487 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:31:33.498 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:31:33.596 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:31:33.645 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:31:34.578 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:31:34.606 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:31:34.629 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:31:34.629 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:34.629 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:31:34.630 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:31:34.630 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:31:34.630 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:34.648 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-04-22T06:31:34.682 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:31:34.703 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:31:34.704 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:34.704 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:31:34.704 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:34.726 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:31:34.749 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:31:34.760 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:31:34.771 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:31:34.771 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:34.772 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:31:34.772 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:31:34.772 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:31:34.772 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:34.803 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:31:35.075 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:31:35.097 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:31:35.097 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:35.097 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:31:35.097 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:31:35.097 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:31:35.097 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:35.134 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:31:35.244 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:35.520 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:35.549 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:31:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:31:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:31:35.788 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:31:35.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:35.920 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:35.943 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:36.005 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:36.027 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:36.047 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:36.269 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:36.302 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:36.411 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:31:36.516 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:31:36.568 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:31:36.603 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:31:36.678 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:31:36.719 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:31:36.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:31:36.804 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:31:36.831 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:31:36.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:31:36.962 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:31:36.992 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:31:37.036 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:31:37.114 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:31:37.142 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:31:37.146 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:31:37.183 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:31:37.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:31:37.218 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:31:37.224 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:31:37.256 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:31:37.284 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:31:37.290 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:31:37.333 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:31:37.417 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:31:37.473 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:31:37.521 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:31:37.551 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:31:37.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:31:37.612 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:31:38.148 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:31:38.177 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:31:38.223 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:31:38.240 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:31:38.244 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:31:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:31:38.311 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:31:38.349 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:31:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:31:38.438 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:31:38.511 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:31:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:31:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:31:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:31:38.677 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:31:38.721 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:31:38.762 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:31:38.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:31:38.943 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:31:38.973 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:31:39.015 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:31:39.032 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:31:39.110 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:31:39.111 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:31:39.147 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:31:39.152 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:31:39.188 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:31:39.213 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:31:39.236 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:31:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:31:39.265 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:31:39.265 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:31:39.314 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:31:39.349 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:31:39.349 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:31:39.442 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:31:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:31:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:31:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:31:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:31:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:31:39.662 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:31:39.686 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:31:39.692 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:31:39.726 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:31:39.727 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:31:39.759 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:31:39.770 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:31:39.800 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:31:39.818 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:31:39.828 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:31:39.852 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:31:39.940 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:31:39.991 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:31:40.022 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:31:40.024 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:31:40.066 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:31:40.069 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:31:40.120 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:31:40.169 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:31:40.224 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:31:40.703 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:31:40.743 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:31:40.762 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:31:40.971 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:31:41.012 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:31:41.046 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:31:41.147 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:31:41.196 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:31:41.292 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:31:41.340 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:31:41.527 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:31:41.548 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:31:41.562 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:31:41.630 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:31:41.640 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:31:41.671 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:31:41.754 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:31:41.784 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:31:41.784 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:31:41.868 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:31:41.868 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:31:42.300 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:31:42.420 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:31:43.693 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:31:43.774 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:31:43.802 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:31:43.835 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:31:44.147 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:31:44.181 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:44.271 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:44.302 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:31:44.339 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:31:44.816 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:31:44.842 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:44.914 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:44.944 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:31:45.337 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:31:45.337 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:31:45.337 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:31:45.337 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:31:45.338 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:31:45.339 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:31:45.352 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:31:45.353 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:31:45.354 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:31:45.355 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:31:45.356 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:31:45.357 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:31:45.358 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:31:45.359 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:31:45.360 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:31:45.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:31:45.362 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:31:47.242 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:47.273 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:47.417 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:31:47.417 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:47.417 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-04-22T06:31:47.417 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.418 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.419 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:31:47.420 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:31:47.421 INFO:teuthology.orchestra.run.smithi079.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:31:47.422 INFO:teuthology.orchestra.run.smithi079.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:31:47.423 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:31:47.423 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:31:47.423 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:31:47.423 INFO:teuthology.orchestra.run.smithi079.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:31:47.425 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:31:47.426 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:31:47.427 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:31:47.428 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:31:47.429 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.430 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:31:47.431 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-04-22T06:31:47.784 DEBUG:teuthology.parallel:result is None 2024-04-22T06:31:47.823 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:47.869 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:47.959 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:31:47.959 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:31:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:31:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:48.036 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:31:48.036 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:31:48.036 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:31:48.037 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:31:48.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:31:48.039 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:31:48.040 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:31:48.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:31:48.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:31:48.043 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:31:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:31:48.045 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:31:48.046 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:31:48.047 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:31:48.047 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:31:48.047 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:31:50.050 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:31:50.050 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:50.050 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-04-22T06:31:50.050 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.050 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.050 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.051 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.052 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:31:50.053 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:31:50.054 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:31:50.055 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:31:50.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:31:50.057 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:31:50.058 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:31:50.059 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:31:50.060 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:31:50.061 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:31:50.062 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:31:50.062 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-04-22T06:31:50.429 DEBUG:teuthology.parallel:result is None 2024-04-22T06:31:59.252 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:59.252 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-04-22T06:31:59.252 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-04-22T06:31:59.252 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-04-22T06:31:59.253 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-04-22T06:31:59.253 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-04-22T06:31:59.253 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-04-22T06:31:59.253 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-04-22T06:31:59.253 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:31:59.430 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:31:59.446 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:31:59.446 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:31:59.446 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:31:59.446 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:00.942 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:00.968 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:00.968 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:00.968 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:32:00.968 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:00.968 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:00.968 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:01.192 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:01.215 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:32:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:01.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:01.660 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:01.683 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:01.683 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:01.683 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:32:01.683 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:01.683 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:01.683 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:01.790 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:32:01.846 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:32:01.902 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:32:02.087 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:32:02.143 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:32:02.222 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:32:02.310 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:32:02.449 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:32:02.494 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:32:02.526 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:32:02.553 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:32:02.964 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:02.993 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:03.026 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:32:03.096 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:32:03.133 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:04.173 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:04.210 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:05.177 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:05.202 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:05.305 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:05.467 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:32:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:05.519 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:05.519 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:05.519 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:32:05.519 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:05.519 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:05.520 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:05.571 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:05.583 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:05.680 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:32:05.736 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:32:06.674 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:32:06.707 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:06.730 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:06.730 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:06.730 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:32:06.730 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:06.730 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:06.730 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:06.775 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:06.796 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:06.797 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:06.797 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:32:06.797 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:06.842 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:06.865 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:06.865 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:06.865 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:32:06.865 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:06.865 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:06.865 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:07.168 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:07.190 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:07.190 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:07.190 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:32:07.191 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:07.191 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:07.191 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:11.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:32:11.812 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:32:11.862 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:32:11.910 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:32:11.946 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:32:12.134 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:32:12.175 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:32:12.205 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:32:12.830 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:12.869 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:12.889 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:13.673 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:32:13.757 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:32:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:32:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:32:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:32:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:19.965 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:32:19.966 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:32:19.967 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:32:19.968 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:32:19.969 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:32:19.970 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:32:19.971 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:32:19.972 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:32:19.973 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:32:19.974 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:32:19.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:32:19.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:32:19.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:32:19.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:32:19.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:32:19.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:32:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:32:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:32:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:32:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:32:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:32:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:32:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:32:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:32:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:32:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-22T06:32:22.054 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-04-22T06:32:22.439 DEBUG:teuthology.parallel:result is None 2024-04-22T06:32:22.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:22.598 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:32:22.624 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:22.625 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:22.625 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:32:22.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:22.777 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:32:22.797 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:22.798 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:22.798 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:32:22.800 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:23.026 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:32:23.052 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:23.052 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:23.052 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:32:23.054 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T06:32:23.054 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:32:23.054 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:32:23.085 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:32:23.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:32:23.110 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:32:23.110 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:32:23.139 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T06:32:23.139 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:32:23.140 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:32:23.165 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:32:23.231 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:32:23.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:32:23.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:32:23.322 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:32:23.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:32:23.350 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:32:23.418 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T06:32:23.419 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:32:23.419 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:32:23.444 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:32:23.509 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:32:23.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:32:23.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:32:23.596 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:32:23.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:32:23.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:32:23.685 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T06:32:23.685 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:32:23.685 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:32:23.709 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:32:23.773 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:32:23.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:32:23.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:32:23.860 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:32:23.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:32:23.883 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:32:23.947 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T06:32:24.050 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T06:32:24.050 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:24.051 INFO:tasks.cephadm:Cluster fsid is 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:32:24.051 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T06:32:24.051 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.79', 'mon.b': '172.21.15.190', 'mon.c': '172.21.15.193'} 2024-04-22T06:32:24.051 INFO:tasks.cephadm:First mon is mon.a on smithi079 2024-04-22T06:32:24.051 INFO:tasks.cephadm:First mgr is x 2024-04-22T06:32:24.051 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T06:32:24.051 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-04-22T06:32:24.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-04-22T06:32:24.100 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-04-22T06:32:24.123 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-22T06:32:24.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:24.279 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78786/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-g86f7587a', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.g86f7587a'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-21 22:42:52.445656', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78786/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-g86f7587a', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.g86f7587a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-21 20:57:03.749940', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T06:32:24.437 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 86f7587a5a09af35f0895e1a2d08527638fae697 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:24.438 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm 2024-04-22T06:32:24.439 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:25.084 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:32 /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:25.084 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:25.935 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:32 /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:25.935 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:26.519 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:32 /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:26.520 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:26.538 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:26.557 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:26.592 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 on all hosts... 2024-04-22T06:32:26.592 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:32:26.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:32:26.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:32:28.170 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:32:28.173 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:32:28.221 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:33:06.593 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-22T06:33:06.594 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:33:06.594 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:33:06.594 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-04-22T06:33:06.594 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:33:06.594 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-04-22T06:33:06.594 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-22T06:33:37.915 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-22T06:33:37.915 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:33:37.915 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:33:37.915 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-04-22T06:33:37.915 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:33:37.915 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-22T06:33:37.916 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:34:04.488 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:34:04.514 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-04-22T06:34:04.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-04-22T06:34:04.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-04-22T06:34:04.617 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-04-22T06:34:04.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-04-22T06:34:04.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-04-22T06:34:04.693 INFO:tasks.cephadm:Writing seed config... 2024-04-22T06:34:04.694 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T06:34:04.695 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T06:34:04.696 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T06:34:04.697 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T06:34:04.698 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T06:34:04.698 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T06:34:04.698 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T06:34:04.698 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-22T06:34:04.699 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:34:04.699 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T06:34:04.714 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 146994a8-0072-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T06:34:04.715 DEBUG:teuthology.orchestra.run.smithi079:mon.a> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T06:34:04.757 DEBUG:teuthology.orchestra.run.smithi079:mgr.x> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T06:34:04.800 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T06:34:04.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 -v bootstrap --fsid 146994a8-0072-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:34:04.989 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:34:04.990 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697', '-v', 'bootstrap', '--fsid', '146994a8-0072-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-04-22T06:34:04.990 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T06:34:04.990 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-04-22T06:34:05.015 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T06:34:05.015 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-04-22T06:34:05.015 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-04-22T06:34:05.022 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T06:34:05.022 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T06:34:05.028 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T06:34:05.028 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-04-22T06:34:05.034 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-04-22T06:34:05.040 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-04-22T06:34:05.040 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-04-22T06:34:05.040 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-04-22T06:34:05.063 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T06:34:05.063 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T06:34:05.063 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-04-22T06:34:05.063 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-04-22T06:34:05.070 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T06:34:05.070 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T06:34:05.075 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T06:34:05.075 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-04-22T06:34:05.081 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-04-22T06:34:05.088 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-04-22T06:34:05.088 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-04-22T06:34:05.088 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-04-22T06:34:05.088 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:05.088 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 140034900704416 on /run/cephadm/146994a8-0072-11ef-bc93-c7b262605968.lock 2024-04-22T06:34:05.088 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 140034900704416 acquired on /run/cephadm/146994a8-0072-11ef-bc93-c7b262605968.lock 2024-04-22T06:34:05.089 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-04-22T06:34:05.089 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-04-22T06:34:05.089 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-04-22T06:34:05.092 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-04-22T06:34:05.092 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-04-22T06:34:05.095 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T06:34:05.095 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T06:34:05.095 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T06:34:05.097 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T06:34:05.097 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T06:34:05.097 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T06:34:05.097 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T06:34:05.097 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::29a7:525d:ae18:4d4e/64 scope link noprefixroute 2024-04-22T06:34:05.098 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T06:34:05.098 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-04-22T06:34:05.098 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-04-22T06:34:05.098 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T06:34:05.099 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T06:34:05.099 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:3f93b098358ed33bdcfea942070678da0220cbea06d5b0225ae8923eb855cc02 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 2024-04-22T06:34:05.532 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T06:34:06.505 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:34:06.506 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:34:06.506 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T06:34:07.396 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-04-22T06:34:07.396 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-04-22T06:34:08.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDfBCZmpWIYKxAATL5dngDUl7FQPjWfsI8LRw== 2024-04-22T06:34:09.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDgBCZm9SlsJBAA/yY66Wk0EexuJjUv796c0Q== 2024-04-22T06:34:15.755 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDhBCZmSUywIRAACIi8kYO5x1f2PiRtirr0pg== 2024-04-22T06:34:15.755 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-04-22T06:34:20.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T06:34:20.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-22T06:34:20.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:20.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T06:34:20.488 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for a [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T06:34:20.488 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = quincy 2024-04-22T06:34:20.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:20.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T06:34:20.488 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:34:20.488 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-04-22T06:34:21.851 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.a on 2024-04-22T06:34:22.072 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T06:34:22.208 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T06:34:22.379 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-146994a8-0072-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-146994a8-0072-11ef-bc93-c7b262605968.target. 2024-04-22T06:34:22.379 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-146994a8-0072-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-146994a8-0072-11ef-bc93-c7b262605968.target. 2024-04-22T06:34:22.607 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a 2024-04-22T06:34:22.607 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T06:34:22.767 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-146994a8-0072-11ef-bc93-c7b262605968.target.wants/ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-146994a8-0072-11ef-bc93-c7b262605968@.service. 2024-04-22T06:34:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:23 smithi079 ceph-mon[27573]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:34:23.268 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:34:23.268 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-04-22T06:34:23.275 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:34:23.275 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-04-22T06:34:23.275 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-04-22T06:34:23.275 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T06:34:23.276 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-04-22T06:34:23.276 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-04-22T06:34:23.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:23 smithi079 bash[27553]: c2846b76801f138dfb32e8d74658461846b14b1ff5ac6ac1f32d9590804e4c91 2024-04-22T06:34:23.540 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:23 smithi079 systemd[1]: Started Ceph mon.a for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: monmap epoch 1 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: last_changed 2024-04-22T06:34:18.390130+0000 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: min_mon_release 19 (squid) 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: election_strategy: 1 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: fsmap 2024-04-22T06:34:24.346 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T06:34:24.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: mgrmap e1: no daemons active 2024-04-22T06:34:24.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:24 smithi079 ceph-mon[27573]: from='client.? 172.21.15.79:0/228604015' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.590824s) 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T06:34:24.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T06:34:24.359 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:24.359 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-04-22T06:34:24.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T06:34:24.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T06:34:24.364 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.731+0000 7f32a33d7700 1 Processor -- start 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.731+0000 7f32a33d7700 1 -- start start 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.731+0000 7f32a33d7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.731+0000 7f32a33d7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f329c107d40 con 0x7f329c107350 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f32a1173700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f32a1173700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42958/0 (socket says 172.21.15.79:42958) 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f32a1173700 1 -- 172.21.15.79:0/2494676422 learned_addr learned my addr 172.21.15.79:0/2494676422 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f32a1173700 1 -- 172.21.15.79:0/2494676422 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c108550 con 0x7f329c107350 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f32a1173700 1 --2- 172.21.15.79:0/2494676422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c107770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f328c0098a0 tx=0x7f328c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199831d0477c73ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f3293fff700 1 -- 172.21.15.79:0/2494676422 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f328c004160 con 0x7f329c107350 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f3293fff700 1 -- 172.21.15.79:0/2494676422 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328c004300 con 0x7f329c107350 2024-04-22T06:34:24.365 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.732+0000 7f3293fff700 1 -- 172.21.15.79:0/2494676422 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f328c0036a0 con 0x7f329c107350 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 -- 172.21.15.79:0/2494676422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 msgr2=0x7f329c107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 --2- 172.21.15.79:0/2494676422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c107770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f328c0098a0 tx=0x7f328c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 -- 172.21.15.79:0/2494676422 shutdown_connections 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 --2- 172.21.15.79:0/2494676422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c107770 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 -- 172.21.15.79:0/2494676422 >> 172.21.15.79:0/2494676422 conn(0x7f329c102910 msgr2=0x7f329c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 -- 172.21.15.79:0/2494676422 shutdown_connections 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.733+0000 7f32a33d7700 1 -- 172.21.15.79:0/2494676422 wait complete. 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a33d7700 1 Processor -- start 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a33d7700 1 -- start start 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a33d7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c07ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a33d7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f329c108a80 con 0x7f329c107350 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a1173700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c07ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a1173700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c07ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42960/0 (socket says 172.21.15.79:42960) 2024-04-22T06:34:24.366 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a1173700 1 -- 172.21.15.79:0/228604015 learned_addr learned my addr 172.21.15.79:0/228604015 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a1173700 1 -- 172.21.15.79:0/228604015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c19e8b0 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32a1173700 1 --2- 172.21.15.79:0/228604015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c07ba60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f328c000c00 tx=0x7f328c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.734+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f328c003db0 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.735+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328c011630 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.735+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329c19ea80 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.735+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f328c011950 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.735+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f329c19ee20 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.735+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f328c011af0 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.735+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f328c01c450 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.736+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329c0640a0 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.738+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f328c032020 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.808+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f329c0008d0 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.808+0000 7f32927fc700 1 -- 172.21.15.79:0/228604015 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f328c004260 con 0x7f329c107350 2024-04-22T06:34:24.367 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 msgr2=0x7f329c07ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 --2- 172.21.15.79:0/228604015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c07ba60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f328c000c00 tx=0x7f328c004750 comp rx=0 tx=0).stop 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 shutdown_connections 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 --2- 172.21.15.79:0/228604015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f329c107350 0x7f329c07ba60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 >> 172.21.15.79:0/228604015 conn(0x7f329c102910 msgr2=0x7f329c191700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 shutdown_connections 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.809+0000 7f32a33d7700 1 -- 172.21.15.79:0/228604015 wait complete. 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-04-22T06:34:24.368 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T06:34:25.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:25 smithi079 ceph-mon[27573]: from='client.? 172.21.15.79:0/3943566757' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T06:34:25.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:25 smithi079 ceph-mon[27573]: from='client.? 172.21.15.79:0/3943566757' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T06:34:26.492 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.745+0000 7f0a9309e700 1 Processor -- start 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.745+0000 7f0a9309e700 1 -- start start 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.745+0000 7f0a9309e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:26.493 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.745+0000 7f0a9309e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c108e30 con 0x7f0a8c108440 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a90e3a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a90e3a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42972/0 (socket says 172.21.15.79:42972) 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a90e3a700 1 -- 172.21.15.79:0/2431008102 learned_addr learned my addr 172.21.15.79:0/2431008102 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a90e3a700 1 -- 172.21.15.79:0/2431008102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c1095f0 con 0x7f0a8c108440 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a90e3a700 1 --2- 172.21.15.79:0/2431008102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c108860 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0a800098a0 tx=0x7f0a80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a601efd4fcd4d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a8b7fe700 1 -- 172.21.15.79:0/2431008102 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a80004160 con 0x7f0a8c108440 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a8b7fe700 1 -- 172.21.15.79:0/2431008102 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a80004300 con 0x7f0a8c108440 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.746+0000 7f0a8b7fe700 1 -- 172.21.15.79:0/2431008102 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a800036a0 con 0x7f0a8c108440 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- 172.21.15.79:0/2431008102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 msgr2=0x7f0a8c108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 --2- 172.21.15.79:0/2431008102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c108860 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0a800098a0 tx=0x7f0a80009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- 172.21.15.79:0/2431008102 shutdown_connections 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 --2- 172.21.15.79:0/2431008102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c108860 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- 172.21.15.79:0/2431008102 >> 172.21.15.79:0/2431008102 conn(0x7f0a8c103a80 msgr2=0x7f0a8c105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:26.494 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- 172.21.15.79:0/2431008102 shutdown_connections 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- 172.21.15.79:0/2431008102 wait complete. 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 Processor -- start 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- start start 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.747+0000 7f0a9309e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c109b20 con 0x7f0a8c108440 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a90e3a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a90e3a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42986/0 (socket says 172.21.15.79:42986) 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a90e3a700 1 -- 172.21.15.79:0/3943566757 learned_addr learned my addr 172.21.15.79:0/3943566757 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a90e3a700 1 -- 172.21.15.79:0/3943566757 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c19c070 con 0x7f0a8c108440 2024-04-22T06:34:26.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a90e3a700 1 --2- 172.21.15.79:0/3943566757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c19bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0a80000c00 tx=0x7f0a80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a80003db0 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a80011630 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c19c300 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a80011950 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c19eff0 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.748+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0a80011af0 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.749+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0a8001c450 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.749+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c0640a0 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.751+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0a80032020 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.819+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0a8c02e100 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.822+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f0a80004260 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.823+0000 7f0a89ffb700 1 -- 172.21.15.79:0/3943566757 <== mon.0 v2:172.21.15.79:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0a8001c6a0 con 0x7f0a8c108440 2024-04-22T06:34:26.496 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.823+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 msgr2=0x7f0a8c19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.823+0000 7f0a9309e700 1 --2- 172.21.15.79:0/3943566757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c19bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0a80000c00 tx=0x7f0a80004750 comp rx=0 tx=0).stop 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.823+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 shutdown_connections 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.824+0000 7f0a9309e700 1 --2- 172.21.15.79:0/3943566757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c108440 0x7f0a8c19bb30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.824+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 >> 172.21.15.79:0/3943566757 conn(0x7f0a8c103a80 msgr2=0x7f0a8c1927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.824+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 shutdown_connections 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.824+0000 7f0a9309e700 1 -- 172.21.15.79:0/3943566757 wait complete. 2024-04-22T06:34:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-04-22T06:34:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:27 smithi079 ceph-mon[27573]: from='client.? 172.21.15.79:0/3347075925' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cea88700 1 Processor -- start 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cea88700 1 -- start start 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cea88700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c8107150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cea88700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f16c807be40 con 0x7f16c8106d30 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cc824700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c8107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cc824700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c8107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42996/0 (socket says 172.21.15.79:42996) 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.959+0000 7f16cc824700 1 -- 172.21.15.79:0/1907322802 learned_addr learned my addr 172.21.15.79:0/1907322802 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16cc824700 1 -- 172.21.15.79:0/1907322802 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c8107690 con 0x7f16c8106d30 2024-04-22T06:34:27.591 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16cc824700 1 --2- 172.21.15.79:0/1907322802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c8107150 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0098a0 tx=0x7f16bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c02406e3fa19703 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16c77fe700 1 -- 172.21.15.79:0/1907322802 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16bc004160 con 0x7f16c8106d30 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16c77fe700 1 -- 172.21.15.79:0/1907322802 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f16bc004300 con 0x7f16c8106d30 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16c77fe700 1 -- 172.21.15.79:0/1907322802 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16bc011af0 con 0x7f16c8106d30 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16cea88700 1 -- 172.21.15.79:0/1907322802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 msgr2=0x7f16c8107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.960+0000 7f16cea88700 1 --2- 172.21.15.79:0/1907322802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c8107150 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0098a0 tx=0x7f16bc009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 -- 172.21.15.79:0/1907322802 shutdown_connections 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 --2- 172.21.15.79:0/1907322802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c8107150 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 -- 172.21.15.79:0/1907322802 >> 172.21.15.79:0/1907322802 conn(0x7f16c8102950 msgr2=0x7f16c8104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 -- 172.21.15.79:0/1907322802 shutdown_connections 2024-04-22T06:34:27.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 -- 172.21.15.79:0/1907322802 wait complete. 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 Processor -- start 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 -- start start 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.961+0000 7f16cea88700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81086c0 con 0x7f16c8106d30 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cc824700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cc824700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43010/0 (socket says 172.21.15.79:43010) 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cc824700 1 -- 172.21.15.79:0/3347075925 learned_addr learned my addr 172.21.15.79:0/3347075925 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cc824700 1 -- 172.21.15.79:0/3347075925 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c819eb10 con 0x7f16c8106d30 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cc824700 1 --2- 172.21.15.79:0/3347075925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c819e5d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16bc011ac0 tx=0x7f16bc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16bc003970 con 0x7f16c8106d30 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c819eda0 con 0x7f16c8106d30 2024-04-22T06:34:27.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f16bc003b10 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16bc024610 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c81a1a90 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f16bc024c20 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.962+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f16bc019700 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.963+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c80640a0 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:26.965+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f16bc043e50 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.032+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f16c80008d0 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.033+0000 7f16c5ffb700 1 -- 172.21.15.79:0/3347075925 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f16bc032030 con 0x7f16c8106d30 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 msgr2=0x7f16c819e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 --2- 172.21.15.79:0/3347075925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c819e5d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16bc011ac0 tx=0x7f16bc011ba0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 shutdown_connections 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 --2- 172.21.15.79:0/3347075925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f16c8106d30 0x7f16c819e5d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 >> 172.21.15.79:0/3347075925 conn(0x7f16c8102950 msgr2=0x7f16c8195240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:27.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 shutdown_connections 2024-04-22T06:34:27.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.034+0000 7f16cea88700 1 -- 172.21.15.79:0/3347075925 wait complete. 2024-04-22T06:34:27.595 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-04-22T06:34:27.863 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:27 smithi079 systemd[1]: Stopping Ceph mon.a for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a[27569]: 2024-04-22T06:34:27.863+0000 7f279b3e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a[27569]: 2024-04-22T06:34:27.863+0000 7f279b3e1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:27 smithi079 podman[27817]: 2024-04-22 06:34:27.95624072 +0000 UTC m=+0.203264129 container died c2846b76801f138dfb32e8d74658461846b14b1ff5ac6ac1f32d9590804e4c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T06:34:28.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 podman[27817]: 2024-04-22 06:34:28.330892966 +0000 UTC m=+0.577916373 container cleanup c2846b76801f138dfb32e8d74658461846b14b1ff5ac6ac1f32d9590804e4c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T06:34:28.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 bash[27817]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a 2024-04-22T06:34:28.873 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 podman[27831]: 2024-04-22 06:34:28.54848893 +0000 UTC m=+0.592448310 container remove c2846b76801f138dfb32e8d74658461846b14b1ff5ac6ac1f32d9590804e4c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:34:28.873 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T06:34:28.873 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 systemd[1]: Stopped Ceph mon.a for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:28.873 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 systemd[1]: Starting Ceph mon.a for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:34:29.121 INFO:teuthology.orchestra.run.smithi079.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T06:34:29.136 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 podman[27927]: 2024-04-22 06:34:28.874582073 +0000 UTC m=+0.109052824 container create 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:34:29.136 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:28 smithi079 podman[27927]: 2024-04-22 06:34:28.782872686 +0000 UTC m=+0.017343404 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:34:29.136 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 podman[27927]: 2024-04-22 06:34:29.043759538 +0000 UTC m=+0.278230248 container init 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:34:29.136 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 podman[27927]: 2024-04-22 06:34:29.046567664 +0000 UTC m=+0.281038370 container start 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev), process ceph-mon, pid 2 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: pidfile_write: ignore empty --pid-file 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: load: jerasure load: lrc 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: RocksDB version: 7.9.2 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Git sha 0 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Compile date 2024-04-21 19:54:05 2024-04-22T06:34:29.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: DB SUMMARY 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: DB Session ID: 3GFJ1QQ6X0P6BY13CW8B 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: CURRENT file: CURRENT 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105120 ; 2024-04-22T06:34:29.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.error_if_exists: 0 2024-04-22T06:34:29.140 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.create_if_missing: 0 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.paranoid_checks: 1 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.env: 0x55a5641cecc0 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.info_log: 0x55a565691be0 2024-04-22T06:34:29.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.statistics: (nil) 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.use_fsync: 0 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_log_file_size: 0 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T06:34:29.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.allow_fallocate: 1 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.use_direct_reads: 0 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.db_log_dir: 2024-04-22T06:34:29.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.wal_dir: 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T06:34:29.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.write_buffer_manager: 0x55a5656250e0 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T06:34:29.145 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.unordered_write: 0 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.row_cache: None 2024-04-22T06:34:29.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.wal_filter: None 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.two_write_queues: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.wal_compression: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.atomic_flush: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.log_readahead_size: 0 2024-04-22T06:34:29.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_background_jobs: 2 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_background_compactions: -1 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_subcompactions: 1 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T06:34:29.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_open_files: -1 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_background_flushes: -1 2024-04-22T06:34:29.149 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Compression algorithms supported: 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kZSTD supported: 0 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kXpressCompression supported: 0 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kZlibCompression supported: 1 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kSnappyCompression supported: 1 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kLZ4Compression supported: 1 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: kBZip2Compression supported: 0 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-22T06:34:29.150 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.merge_operator: 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_filter: None 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a565691dc0) 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T06:34:29.151 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_type: 0 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_index_type: 0 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_shortening: 1 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: checksum: 4 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: no_block_cache: 0 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache: 0x55a5656571f0 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_options: 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: capacity : 536870912 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_shard_bits : 4 2024-04-22T06:34:29.152 INFO:journalctl@ceph.mon.a.smithi079.stdout: strict_capacity_limit : 0 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_compressed: (nil) 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: persistent_cache: (nil) 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size: 4096 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size_deviation: 10 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_restart_interval: 16 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_block_restart_interval: 1 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: metadata_block_size: 4096 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: partition_filters: 0 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: use_delta_encoding: 1 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: filter_policy: bloomfilter 2024-04-22T06:34:29.153 INFO:journalctl@ceph.mon.a.smithi079.stdout: whole_key_filtering: 1 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: verify_compression: 0 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: format_version: 5 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_index_compression: 1 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_align: 0 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: max_auto_readahead_size: 262144 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: prepopulate_block_cache: 0 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: initial_auto_readahead_size: 8192 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T06:34:29.154 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression: NoCompression 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.num_levels: 7 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:34:29.155 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T06:34:29.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T06:34:29.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T06:34:29.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.inplace_update_support: 0 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T06:34:29.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.bloom_locality: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.max_successive_merges: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.ttl: 2592000 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T06:34:29.160 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.enable_blob_files: false 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.min_blob_size: 0 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T06:34:29.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1970bcc1-570a-4982-8926-570294777dc1 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767669075159, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767669076115, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100287, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98172, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14169, "raw_average_key_size": 49, "raw_value_size": 90456, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713767669, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1970bcc1-570a-4982-8926-570294777dc1", "db_session_id": "3GFJ1QQ6X0P6BY13CW8B", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767669076172, "job": 1, "event": "recovery_finished"} 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a56576c000 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: DB pointer 0x55a565756000 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T06:34:29.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** DB Stats ** 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:34:29.163 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: L0 2/0 99.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: Sum 2/0 99.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:29.164 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative compaction: 0.00 GB write, 32.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval compaction: 0.00 GB write, 32.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache BinnedLRUCache@0x55a5656571f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2024-04-22T06:34:29.165 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: starting mon.a rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???) e1 preinit fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).mds e1 new map 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).mds e1 print_map 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout: e1 2024-04-22T06:34:29.166 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout: legacy client fscid: -1 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout: No filesystems configured 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 bash[27927]: 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e 2024-04-22T06:34:29.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 systemd[1]: Started Ceph mon.a for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: monmap epoch 1 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: last_changed 2024-04-22T06:34:18.390130+0000 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: min_mon_release 19 (squid) 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: election_strategy: 1 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: fsmap 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T06:34:29.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:29 smithi079 ceph-mon[27946]: mgrmap e1: no daemons active 2024-04-22T06:34:30.190 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.546+0000 7f7f57e11700 1 Processor -- start 2024-04-22T06:34:30.190 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.546+0000 7f7f57e11700 1 -- start start 2024-04-22T06:34:30.190 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f57e11700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f50108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:30.190 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f57e11700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50108e30 con 0x7f7f50108440 2024-04-22T06:34:30.190 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f55bad700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f50108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f55bad700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f50108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43012/0 (socket says 172.21.15.79:43012) 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f55bad700 1 -- 172.21.15.79:0/3740181249 learned_addr learned my addr 172.21.15.79:0/3740181249 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f55bad700 1 -- 172.21.15.79:0/3740181249 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f501095f0 con 0x7f7f50108440 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.547+0000 7f7f55bad700 1 --2- 172.21.15.79:0/3740181249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f50108860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f400098a0 tx=0x7f7f40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514d09946c2d1c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f54bab700 1 -- 172.21.15.79:0/3740181249 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f40004160 con 0x7f7f50108440 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f54bab700 1 -- 172.21.15.79:0/3740181249 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7f40004300 con 0x7f7f50108440 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f54bab700 1 -- 172.21.15.79:0/3740181249 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f40011af0 con 0x7f7f50108440 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 -- 172.21.15.79:0/3740181249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 msgr2=0x7f7f50108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 --2- 172.21.15.79:0/3740181249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f50108860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f400098a0 tx=0x7f7f40009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 -- 172.21.15.79:0/3740181249 shutdown_connections 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 --2- 172.21.15.79:0/3740181249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f50108860 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 -- 172.21.15.79:0/3740181249 >> 172.21.15.79:0/3740181249 conn(0x7f7f50103a00 msgr2=0x7f7f50105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 -- 172.21.15.79:0/3740181249 shutdown_connections 2024-04-22T06:34:30.191 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.548+0000 7f7f57e11700 1 -- 172.21.15.79:0/3740181249 wait complete. 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f57e11700 1 Processor -- start 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f57e11700 1 -- start start 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f57e11700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f5019c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f57e11700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5010c9a0 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f55bad700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f5019c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f55bad700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f5019c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43016/0 (socket says 172.21.15.79:43016) 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f55bad700 1 -- 172.21.15.79:0/873614085 learned_addr learned my addr 172.21.15.79:0/873614085 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.549+0000 7f7f55bad700 1 -- 172.21.15.79:0/873614085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5019c8a0 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f55bad700 1 --2- 172.21.15.79:0/873614085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f5019c360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f40003f40 tx=0x7f7f40011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f400038b0 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7f40003a50 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5019cb30 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f4001a5c0 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5019f820 con 0x7f7f50108440 2024-04-22T06:34:30.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f4001ac50 con 0x7f7f50108440 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.550+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f40019700 con 0x7f7f50108440 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.551+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f500640a0 con 0x7f7f50108440 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.553+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7f40019900 con 0x7f7f50108440 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.621+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f7f5019f9f0 con 0x7f7f50108440 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.623+0000 7f7f46ffd700 1 -- 172.21.15.79:0/873614085 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f7f40019ae0 con 0x7f7f50108440 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 msgr2=0x7f7f5019c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 --2- 172.21.15.79:0/873614085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f5019c360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f40003f40 tx=0x7f7f40011890 comp rx=0 tx=0).stop 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 shutdown_connections 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 --2- 172.21.15.79:0/873614085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7f50108440 0x7f7f5019c360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 >> 172.21.15.79:0/873614085 conn(0x7f7f50103a00 msgr2=0x7f7f501044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 shutdown_connections 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.624+0000 7f7f57e11700 1 -- 172.21.15.79:0/873614085 wait complete. 2024-04-22T06:34:30.193 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T06:34:30.215 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:34:30.215 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-04-22T06:34:30.216 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T06:34:30.216 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T06:34:30.460 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x 2024-04-22T06:34:30.460 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-22T06:34:30.601 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-146994a8-0072-11ef-bc93-c7b262605968.target.wants/ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-146994a8-0072-11ef-bc93-c7b262605968@.service. 2024-04-22T06:34:30.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:30 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/873614085' entity='client.admin' 2024-04-22T06:34:31.104 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 podman[28159]: 2024-04-22 06:34:31.025312064 +0000 UTC m=+0.304197033 container start 56d7b282d42a7c8a4c32bc9ac9452cfac5e9b4b2493c8a84d45de0df073f0fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:34:31.119 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:34:31.119 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-04-22T06:34:31.127 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:34:31.127 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-04-22T06:34:31.127 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-04-22T06:34:31.127 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T06:34:31.137 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:34:31.137 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-04-22T06:34:31.146 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:34:31.146 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-04-22T06:34:31.146 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-04-22T06:34:31.146 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-22T06:34:31.146 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-04-22T06:34:31.146 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-04-22T06:34:31.418 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 bash[28159]: 56d7b282d42a7c8a4c32bc9ac9452cfac5e9b4b2493c8a84d45de0df073f0fc8 2024-04-22T06:34:31.418 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 systemd[1]: Started Ceph mgr.x for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:31.418 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:31.183+0000 7fef00126200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:34:31.418 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:31.365+0000 7fef00126200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:31.724 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:31.471+0000 7fef00126200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:34:31.988 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2448701550' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:31.989 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:31.787+0000 7fef00126200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:34:31.989 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:31 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:31.875+0000 7fef00126200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:34:32.592 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:32.593 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:32.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:32.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:32.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:32.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:32.603 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:32.604 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:23.219994+0000", 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:32.605 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.826+0000 7fa687da9700 1 Processor -- start 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.826+0000 7fa687da9700 1 -- start start 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.826+0000 7fa687da9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa680106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.826+0000 7fa687da9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa6801070e0 con 0x7fa6801066f0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa685b45700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa680106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa685b45700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa680106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43038/0 (socket says 172.21.15.79:43038) 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa685b45700 1 -- 172.21.15.79:0/651702667 learned_addr learned my addr 172.21.15.79:0/651702667 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa685b45700 1 -- 172.21.15.79:0/651702667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa680109720 con 0x7fa6801066f0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa685b45700 1 --2- 172.21.15.79:0/651702667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa680106b10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa67c0098a0 tx=0x7fa67c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37eb78f6a8f8190f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa684b43700 1 -- 172.21.15.79:0/651702667 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa67c004160 con 0x7fa6801066f0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa684b43700 1 -- 172.21.15.79:0/651702667 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa67c004300 con 0x7fa6801066f0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.827+0000 7fa684b43700 1 -- 172.21.15.79:0/651702667 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa67c011af0 con 0x7fa6801066f0 2024-04-22T06:34:32.606 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 -- 172.21.15.79:0/651702667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 msgr2=0x7fa680106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 --2- 172.21.15.79:0/651702667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa680106b10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa67c0098a0 tx=0x7fa67c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 -- 172.21.15.79:0/651702667 shutdown_connections 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 --2- 172.21.15.79:0/651702667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa680106b10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 -- 172.21.15.79:0/651702667 >> 172.21.15.79:0/651702667 conn(0x7fa680102080 msgr2=0x7fa6801044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 -- 172.21.15.79:0/651702667 shutdown_connections 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.828+0000 7fa687da9700 1 -- 172.21.15.79:0/651702667 wait complete. 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa687da9700 1 Processor -- start 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa687da9700 1 -- start start 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa687da9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa68019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa687da9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa68010c950 con 0x7fa6801066f0 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa685b45700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa68019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa685b45700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa68019c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43050/0 (socket says 172.21.15.79:43050) 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa685b45700 1 -- 172.21.15.79:0/2448701550 learned_addr learned my addr 172.21.15.79:0/2448701550 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.829+0000 7fa685b45700 1 -- 172.21.15.79:0/2448701550 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68019c950 con 0x7fa6801066f0 2024-04-22T06:34:32.607 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa685b45700 1 --2- 172.21.15.79:0/2448701550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa68019c410 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009fb0 tx=0x7fa67c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa67c0036a0 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa68019cbe0 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa67c01a430 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa67c019550 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa68019f8d0 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa67c019b90 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.830+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa67c02d730 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.831+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6800640a0 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.833+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa67c032070 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.905+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa68019fe70 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.906+0000 7fa676ffd700 1 -- 172.21.15.79:0/2448701550 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa67c01a5e0 con 0x7fa6801066f0 2024-04-22T06:34:32.608 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 msgr2=0x7fa68019c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 --2- 172.21.15.79:0/2448701550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa68019c410 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009fb0 tx=0x7fa67c011940 comp rx=0 tx=0).stop 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 shutdown_connections 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 --2- 172.21.15.79:0/2448701550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6801066f0 0x7fa68019c410 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 >> 172.21.15.79:0/2448701550 conn(0x7fa680102080 msgr2=0x7fa680102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 shutdown_connections 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:31.907+0000 7fa687da9700 1 -- 172.21.15.79:0/2448701550 wait complete. 2024-04-22T06:34:32.609 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-04-22T06:34:33.416 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:33 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:33.084+0000 7fef00126200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:34:33.416 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:33 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:33.249+0000 7fef00126200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:33.416 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:33 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:33.417+0000 7fef00126200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:34:33.994 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:33 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:33.715+0000 7fef00126200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:34:33.994 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:33 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:33.812+0000 7fef00126200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:34:34.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:34 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:34.020+0000 7fef00126200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:34:34.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:34 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:34.218+0000 7fef00126200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:34.931 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:34 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:34.829+0000 7fef00126200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:34:35.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2416399463' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:35.571 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:35 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:35.270+0000 7fef00126200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:34:35.571 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:35 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:35.370+0000 7fef00126200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:34:35.571 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:35 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:35.572+0000 7fef00126200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:34:35.689 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:35.690 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:35.691 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:35.692 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:23.219994+0000", 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.032+0000 7f1cfe3ec700 1 Processor -- start 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.032+0000 7f1cfe3ec700 1 -- start start 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.032+0000 7f1cfe3ec700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8107c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.032+0000 7f1cfe3ec700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8108240 con 0x7f1cf8107850 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8107c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8107c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42834/0 (socket says 172.21.15.79:42834) 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf7fff700 1 -- 172.21.15.79:0/3360432647 learned_addr learned my addr 172.21.15.79:0/3360432647 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf7fff700 1 -- 172.21.15.79:0/3360432647 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf8108570 con 0x7f1cf8107850 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf7fff700 1 --2- 172.21.15.79:0/3360432647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8107c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098a0 tx=0x7f1ce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8ef83cd45ebe72 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:35.694 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf6ffd700 1 -- 172.21.15.79:0/3360432647 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8004160 con 0x7f1cf8107850 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf6ffd700 1 -- 172.21.15.79:0/3360432647 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1ce8004300 con 0x7f1cf8107850 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.033+0000 7f1cf6ffd700 1 -- 172.21.15.79:0/3360432647 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8011af0 con 0x7f1cf8107850 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/3360432647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 msgr2=0x7f1cf8107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 --2- 172.21.15.79:0/3360432647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8107c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098a0 tx=0x7f1ce8009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/3360432647 shutdown_connections 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 --2- 172.21.15.79:0/3360432647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8107c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/3360432647 >> 172.21.15.79:0/3360432647 conn(0x7f1cf8102110 msgr2=0x7f1cf8104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/3360432647 shutdown_connections 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.034+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/3360432647 wait complete. 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cfe3ec700 1 Processor -- start 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cfe3ec700 1 -- start start 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cfe3ec700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8106e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cfe3ec700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf810b920 con 0x7f1cf8107850 2024-04-22T06:34:35.695 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cf7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42850/0 (socket says 172.21.15.79:42850) 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cf7fff700 1 -- 172.21.15.79:0/2416399463 learned_addr learned my addr 172.21.15.79:0/2416399463 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cf7fff700 1 -- 172.21.15.79:0/2416399463 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf819f0f0 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.035+0000 7f1cf7fff700 1 --2- 172.21.15.79:0/2416399463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8106e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8000c00 tx=0x7f1ce8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8003b20 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1ce801a8c0 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf819f2c0 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8024940 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf819f660 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1ce8022020 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.036+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1ce802ddf0 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.037+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf80640a0 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.039+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1ce801aa70 con 0x7f1cf8107850 2024-04-22T06:34:35.696 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.111+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1cf819f960 con 0x7f1cf8107850 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.112+0000 7f1cf57fa700 1 -- 172.21.15.79:0/2416399463 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1ce801a430 con 0x7f1cf8107850 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 msgr2=0x7f1cf8106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 --2- 172.21.15.79:0/2416399463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8106e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8000c00 tx=0x7f1ce8011b70 comp rx=0 tx=0).stop 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 shutdown_connections 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 --2- 172.21.15.79:0/2416399463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1cf8107850 0x7f1cf8106e30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 >> 172.21.15.79:0/2416399463 conn(0x7f1cf8102110 msgr2=0x7f1cf8102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 shutdown_connections 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:35.113+0000 7f1cfe3ec700 1 -- 172.21.15.79:0/2416399463 wait complete. 2024-04-22T06:34:35.697 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (2/15)... 2024-04-22T06:34:36.419 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:36 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:36.093+0000 7fef00126200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:34:36.419 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:36 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:36.182+0000 7fef00126200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:34:36.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:36 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:36.420+0000 7fef00126200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:34:36.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:36 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:36.551+0000 7fef00126200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:34:36.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:36 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:36.643+0000 7fef00126200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:34:37.051 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:36 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:36.758+0000 7fef00126200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:37.052 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:37.053+0000 7fef00126200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:34:37.918 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:37.644+0000 7fef00126200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:34:37.919 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:37.864+0000 7fef00126200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:34:38.228 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: Activating manager daemon x 2024-04-22T06:34:38.228 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: mgrmap e2: x(active, starting, since 0.00314781s) 2024-04-22T06:34:38.228 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:34:38.228 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:34:38.228 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: Manager daemon x is now available 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' 2024-04-22T06:34:38.229 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:37 smithi079 ceph-mon[27946]: from='mgr.14100 172.21.15.79:0/251926028' entity='mgr.x' 2024-04-22T06:34:38.778 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:38.781 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:38.781 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:38.781 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:38.781 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-22T06:34:38.782 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:38.783 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:38.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:38.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:38.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:23.219994+0000", 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:38.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.153+0000 7f2f3551e700 1 Processor -- start 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.153+0000 7f2f3551e700 1 -- start start 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.153+0000 7f2f3551e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f30106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.153+0000 7f2f3551e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2f301074a0 con 0x7f2f30106ab0 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f30106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f30106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42928/0 (socket says 172.21.15.79:42928) 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2effd700 1 -- 172.21.15.79:0/1551605151 learned_addr learned my addr 172.21.15.79:0/1551605151 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2effd700 1 -- 172.21.15.79:0/1551605151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f30107cb0 con 0x7f2f30106ab0 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2effd700 1 --2- 172.21.15.79:0/1551605151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f30106ed0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000bb40 tx=0x7f2f2000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c4797d88f80976 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:38.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2dffb700 1 -- 172.21.15.79:0/1551605151 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f2000dad0 con 0x7f2f30106ab0 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2dffb700 1 -- 172.21.15.79:0/1551605151 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2f200044c0 con 0x7f2f30106ab0 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.154+0000 7f2f2dffb700 1 -- 172.21.15.79:0/1551605151 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f20003de0 con 0x7f2f30106ab0 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 -- 172.21.15.79:0/1551605151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 msgr2=0x7f2f30106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 --2- 172.21.15.79:0/1551605151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f30106ed0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000bb40 tx=0x7f2f2000be50 comp rx=0 tx=0).stop 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 -- 172.21.15.79:0/1551605151 shutdown_connections 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 --2- 172.21.15.79:0/1551605151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f30106ed0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 -- 172.21.15.79:0/1551605151 >> 172.21.15.79:0/1551605151 conn(0x7f2f30102110 msgr2=0x7f2f30104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 -- 172.21.15.79:0/1551605151 shutdown_connections 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 -- 172.21.15.79:0/1551605151 wait complete. 2024-04-22T06:34:38.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 Processor -- start 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.155+0000 7f2f3551e700 1 -- start start 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f3551e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f3019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f3551e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2f301083d0 con 0x7f2f30106ab0 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f2effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f3019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f2effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f3019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42930/0 (socket says 172.21.15.79:42930) 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f2effd700 1 -- 172.21.15.79:0/3665214181 learned_addr learned my addr 172.21.15.79:0/3665214181 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f2effd700 1 -- 172.21.15.79:0/3665214181 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3019a6d0 con 0x7f2f30106ab0 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f2effd700 1 --2- 172.21.15.79:0/3665214181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f3019a190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f20019040 tx=0x7f2f20003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f20025020 con 0x7f2f30106ab0 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2f200179d0 con 0x7f2f30106ab0 2024-04-22T06:34:38.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3019a960 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f20021870 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.156+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3019d650 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.157+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7f2f20028020 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.157+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f300640a0 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.157+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f2004fb30 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.160+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2f2002f020 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.229+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2f3002e290 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.229+0000 7f2f17fff700 1 -- 172.21.15.79:0/3665214181 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2f20012070 con 0x7f2f30106ab0 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 msgr2=0x7f2f3019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:38.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 --2- 172.21.15.79:0/3665214181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f3019a190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f20019040 tx=0x7f2f20003ee0 comp rx=0 tx=0).stop 2024-04-22T06:34:38.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 shutdown_connections 2024-04-22T06:34:38.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 --2- 172.21.15.79:0/3665214181 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f30106ab0 0x7f2f3019a190 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:38.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 >> 172.21.15.79:0/3665214181 conn(0x7f2f30102110 msgr2=0x7f2f30190db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:38.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 shutdown_connections 2024-04-22T06:34:38.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:38.230+0000 7f2f3551e700 1 -- 172.21.15.79:0/3665214181 wait complete. 2024-04-22T06:34:38.799 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (3/15)... 2024-04-22T06:34:39.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:38 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3665214181' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:39.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:38 smithi079 ceph-mon[27946]: mgrmap e3: x(active, since 1.00654s) 2024-04-22T06:34:41.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:40 smithi079 ceph-mon[27946]: mgrmap e4: x(active, since 2s) 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:42.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:42.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:42.105 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:42.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:42.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:23.219994+0000", 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a44fd3700 1 Processor -- start 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a44fd3700 1 -- start start 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a44fd3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a40108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a44fd3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40108e30 con 0x7f6a40108440 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a40108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a40108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42942/0 (socket says 172.21.15.79:42942) 2024-04-22T06:34:42.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a3e59c700 1 -- 172.21.15.79:0/2827173460 learned_addr learned my addr 172.21.15.79:0/2827173460 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.241+0000 7f6a3e59c700 1 -- 172.21.15.79:0/2827173460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a401095f0 con 0x7f6a40108440 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a3e59c700 1 --2- 172.21.15.79:0/2827173460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a40108860 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a340098a0 tx=0x7f6a34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fe5e7799e1e316 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a3d59a700 1 -- 172.21.15.79:0/2827173460 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a34004160 con 0x7f6a40108440 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a3d59a700 1 -- 172.21.15.79:0/2827173460 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6a34004300 con 0x7f6a40108440 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a3d59a700 1 -- 172.21.15.79:0/2827173460 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a34011af0 con 0x7f6a40108440 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a44fd3700 1 -- 172.21.15.79:0/2827173460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 msgr2=0x7f6a40108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a44fd3700 1 --2- 172.21.15.79:0/2827173460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a40108860 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a340098a0 tx=0x7f6a34009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a44fd3700 1 -- 172.21.15.79:0/2827173460 shutdown_connections 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a44fd3700 1 --2- 172.21.15.79:0/2827173460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a40108860 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a44fd3700 1 -- 172.21.15.79:0/2827173460 >> 172.21.15.79:0/2827173460 conn(0x7f6a40103ae0 msgr2=0x7f6a40105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.242+0000 7f6a44fd3700 1 -- 172.21.15.79:0/2827173460 shutdown_connections 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.243+0000 7f6a44fd3700 1 -- 172.21.15.79:0/2827173460 wait complete. 2024-04-22T06:34:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.243+0000 7f6a44fd3700 1 Processor -- start 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.243+0000 7f6a44fd3700 1 -- start start 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.243+0000 7f6a44fd3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a4019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.243+0000 7f6a44fd3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4010c9a0 con 0x7f6a40108440 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a4019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a3e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a4019c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42952/0 (socket says 172.21.15.79:42952) 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a3e59c700 1 -- 172.21.15.79:0/1311929877 learned_addr learned my addr 172.21.15.79:0/1311929877 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a3e59c700 1 -- 172.21.15.79:0/1311929877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4019c8c0 con 0x7f6a40108440 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a3e59c700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a4019c380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a34003f40 tx=0x7f6a34011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a340038b0 con 0x7f6a40108440 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4019cb50 con 0x7f6a40108440 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6a34003a50 con 0x7f6a40108440 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a3401a5c0 con 0x7f6a40108440 2024-04-22T06:34:42.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.244+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4019f840 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.245+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f6a3401a8c0 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.245+0000 7f6a2f7fe700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6a28039510 0x7f6a2803b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.245+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a3404dd70 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.245+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a400640a0 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.245+0000 7f6a3dd9b700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6a28039510 0x7f6a2803b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.246+0000 7f6a3dd9b700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6a28039510 0x7f6a2803b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6a30006fd0 tx=0x7f6a30006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.251+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a3400b3e0 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.527+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6a4019fe10 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.527+0000 7f6a2f7fe700 1 -- 172.21.15.79:0/1311929877 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6a34019db0 con 0x7f6a40108440 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6a28039510 msgr2=0x7f6a2803b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6a28039510 0x7f6a2803b9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6a30006fd0 tx=0x7f6a30006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 msgr2=0x7f6a4019c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a4019c380 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a34003f40 tx=0x7f6a34011890 comp rx=0 tx=0).stop 2024-04-22T06:34:42.111 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 shutdown_connections 2024-04-22T06:34:42.112 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6a28039510 0x7f6a2803b9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:42.112 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 --2- 172.21.15.79:0/1311929877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6a40108440 0x7f6a4019c380 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:42.112 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 >> 172.21.15.79:0/1311929877 conn(0x7f6a40103ae0 msgr2=0x7f6a401045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:42.112 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 shutdown_connections 2024-04-22T06:34:42.112 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:41.531+0000 7f6a44fd3700 1 -- 172.21.15.79:0/1311929877 wait complete. 2024-04-22T06:34:42.112 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-04-22T06:34:42.208 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:41 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1311929877' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:43.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:42 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2542654746' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T06:34:43.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:43.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T06:34:43.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T06:34:43.316 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e5d9e2700 1 Processor -- start 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e5d9e2700 1 -- start start 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e5d9e2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e58108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e5d9e2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3e58108e30 con 0x7f3e58108440 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e56ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e58108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e56ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e58108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43202/0 (socket says 172.21.15.79:43202) 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e56ffd700 1 -- 172.21.15.79:0/435415521 learned_addr learned my addr 172.21.15.79:0/435415521 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e56ffd700 1 -- 172.21.15.79:0/435415521 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e581095f0 con 0x7f3e58108440 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.492+0000 7f3e56ffd700 1 --2- 172.21.15.79:0/435415521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e58108860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e480098a0 tx=0x7f3e48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79df94caa3c6a8f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e55ffb700 1 -- 172.21.15.79:0/435415521 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e48004160 con 0x7f3e58108440 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e55ffb700 1 -- 172.21.15.79:0/435415521 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3e48004300 con 0x7f3e58108440 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e55ffb700 1 -- 172.21.15.79:0/435415521 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e48011af0 con 0x7f3e58108440 2024-04-22T06:34:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/435415521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 msgr2=0x7f3e58108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 --2- 172.21.15.79:0/435415521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e58108860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e480098a0 tx=0x7f3e48009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/435415521 shutdown_connections 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 --2- 172.21.15.79:0/435415521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e58108860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/435415521 >> 172.21.15.79:0/435415521 conn(0x7f3e58103ae0 msgr2=0x7f3e58105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/435415521 shutdown_connections 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.493+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/435415521 wait complete. 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e5d9e2700 1 Processor -- start 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e5d9e2700 1 -- start start 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e5d9e2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e5807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e5d9e2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5810c9a0 con 0x7f3e58108440 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e56ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e5807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e56ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e5807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43206/0 (socket says 172.21.15.79:43206) 2024-04-22T06:34:43.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e56ffd700 1 -- 172.21.15.79:0/2542654746 learned_addr learned my addr 172.21.15.79:0/2542654746 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.494+0000 7f3e56ffd700 1 -- 172.21.15.79:0/2542654746 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5807e0c0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.495+0000 7f3e56ffd700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e5807db80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e48003f40 tx=0x7f3e48011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.495+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e480038b0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.495+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3e48003a50 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.495+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5807c2b0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.495+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e4801a5c0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.495+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5807c7f0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.496+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f3e4801a8c0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.496+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e580640a0 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.496+0000 7f3e5c9e0700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f3e40039510 0x7f3e4003b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.496+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3e4804db80 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.496+0000 7f3e567fc700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f3e40039510 0x7f3e4003b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.496+0000 7f3e567fc700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f3e40039510 0x7f3e4003b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.502+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e48019c70 con 0x7f3e58108440 2024-04-22T06:34:43.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.697+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f3e5807ca70 con 0x7f3e58108440 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.698+0000 7f3e5c9e0700 1 -- 172.21.15.79:0/2542654746 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f3e4804d5a0 con 0x7f3e58108440 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f3e40039510 msgr2=0x7f3e4003b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f3e40039510 0x7f3e4003b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c006fd0 tx=0x7f3e4c006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 msgr2=0x7f3e5807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e5807db80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e48003f40 tx=0x7f3e48011890 comp rx=0 tx=0).stop 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 shutdown_connections 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f3e40039510 0x7f3e4003b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 --2- 172.21.15.79:0/2542654746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e58108440 0x7f3e5807db80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 >> 172.21.15.79:0/2542654746 conn(0x7f3e58103ae0 msgr2=0x7f3e581045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 shutdown_connections 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:42.701+0000 7f3e5d9e2700 1 -- 172.21.15.79:0/2542654746 wait complete. 2024-04-22T06:34:43.320 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-04-22T06:34:44.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1992252816' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T06:34:44.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: ignoring --setuser ceph since I am not root 2024-04-22T06:34:44.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: ignoring --setgroup ceph since I am not root 2024-04-22T06:34:44.241 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:44.214+0000 7f6dcdecc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:34:44.622 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:44.398+0000 7f6dcdecc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:44.622 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:44.500+0000 7f6dcdecc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:34:44.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.727+0000 7fd10ecb8700 1 Processor -- start 2024-04-22T06:34:44.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.727+0000 7fd10ecb8700 1 -- start start 2024-04-22T06:34:44.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ecb8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10810b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:44.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ecb8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd10807bec0 con 0x7fd108108e60 2024-04-22T06:34:44.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ca54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10810b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:44.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ca54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10810b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43218/0 (socket says 172.21.15.79:43218) 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ca54700 1 -- 172.21.15.79:0/1760625552 learned_addr learned my addr 172.21.15.79:0/1760625552 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ca54700 1 -- 172.21.15.79:0/1760625552 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10810b790 con 0x7fd108108e60 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd10ca54700 1 --2- 172.21.15.79:0/1760625552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10810b250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0098a0 tx=0x7fd0fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827dd2591b04aa1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd1077fe700 1 -- 172.21.15.79:0/1760625552 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc004160 con 0x7fd108108e60 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd1077fe700 1 -- 172.21.15.79:0/1760625552 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd0fc004300 con 0x7fd108108e60 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.728+0000 7fd1077fe700 1 -- 172.21.15.79:0/1760625552 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc011af0 con 0x7fd108108e60 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1760625552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 msgr2=0x7fd10810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 --2- 172.21.15.79:0/1760625552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10810b250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0098a0 tx=0x7fd0fc009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1760625552 shutdown_connections 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 --2- 172.21.15.79:0/1760625552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10810b250 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1760625552 >> 172.21.15.79:0/1760625552 conn(0x7fd108102910 msgr2=0x7fd108104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1760625552 shutdown_connections 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.729+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1760625552 wait complete. 2024-04-22T06:34:44.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ecb8700 1 Processor -- start 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ecb8700 1 -- start start 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ecb8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ecb8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd10810c760 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ca54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ca54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43228/0 (socket says 172.21.15.79:43228) 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ca54700 1 -- 172.21.15.79:0/1992252816 learned_addr learned my addr 172.21.15.79:0/1992252816 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ca54700 1 -- 172.21.15.79:0/1992252816 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10819c8c0 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ca54700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10819c380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009fb0 tx=0x7fd0fc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc0036a0 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd0fc01a430 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10819cb50 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.730+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc019550 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.731+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10819f840 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.731+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fd0fc01a8e0 con 0x7fd108108e60 2024-04-22T06:34:44.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.732+0000 7fd105ffb700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7fd0f8039510 0x7fd0f803b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.732+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1080640a0 con 0x7fd108108e60 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.732+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd0fc04d8b0 con 0x7fd108108e60 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.732+0000 7fd107fff700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7fd0f8039510 0x7fd0f803b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.732+0000 7fd107fff700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7fd0f8039510 0x7fd0f803b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4006fd0 tx=0x7fd0f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.737+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0fc00b3e0 con 0x7fd108108e60 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:43.988+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fd10819fa10 con 0x7fd108108e60 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.040+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fd10819fa10 con 0x7fd108108e60 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.041+0000 7fd105ffb700 1 -- 172.21.15.79:0/1992252816 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fd0fc003840 con 0x7fd108108e60 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7fd0f8039510 msgr2=0x7fd0f803b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7fd0f8039510 0x7fd0f803b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4006fd0 tx=0x7fd0f4006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 msgr2=0x7fd10819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10819c380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009fb0 tx=0x7fd0fc011940 comp rx=0 tx=0).stop 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 shutdown_connections 2024-04-22T06:34:44.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7fd0f8039510 0x7fd0f803b9d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:44.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 --2- 172.21.15.79:0/1992252816 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd108108e60 0x7fd10819c380 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:44.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 >> 172.21.15.79:0/1992252816 conn(0x7fd108102910 msgr2=0x7fd108103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:44.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 shutdown_connections 2024-04-22T06:34:44.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:44.043+0000 7fd10ecb8700 1 -- 172.21.15.79:0/1992252816 wait complete. 2024-04-22T06:34:44.875 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:44.831+0000 7f6dcdecc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:34:45.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:45 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1992252816' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T06:34:45.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:45 smithi079 ceph-mon[27946]: mgrmap e5: x(active, since 6s) 2024-04-22T06:34:45.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:44.917+0000 7f6dcdecc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661f0c0700 1 Processor -- start 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661f0c0700 1 -- start start 2024-04-22T06:34:45.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661f0c0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f6618107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661f0c0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6618107ce0 con 0x7f66181072f0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661ce5c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f6618107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661ce5c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f6618107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43252/0 (socket says 172.21.15.79:43252) 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661ce5c700 1 -- 172.21.15.79:0/1735687826 learned_addr learned my addr 172.21.15.79:0/1735687826 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661ce5c700 1 -- 172.21.15.79:0/1735687826 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66181084f0 con 0x7f66181072f0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.069+0000 7f661ce5c700 1 --2- 172.21.15.79:0/1735687826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f6618107710 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f660c00bb40 tx=0x7f660c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96b09f19f0b2576 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f66177fe700 1 -- 172.21.15.79:0/1735687826 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c00dbe0 con 0x7f66181072f0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f66177fe700 1 -- 172.21.15.79:0/1735687826 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f660c0044c0 con 0x7f66181072f0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f66177fe700 1 -- 172.21.15.79:0/1735687826 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c010470 con 0x7f66181072f0 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f661f0c0700 1 -- 172.21.15.79:0/1735687826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 msgr2=0x7f6618107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f661f0c0700 1 --2- 172.21.15.79:0/1735687826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f6618107710 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f660c00bb40 tx=0x7f660c00be50 comp rx=0 tx=0).stop 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f661f0c0700 1 -- 172.21.15.79:0/1735687826 shutdown_connections 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f661f0c0700 1 --2- 172.21.15.79:0/1735687826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f6618107710 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.070+0000 7f661f0c0700 1 -- 172.21.15.79:0/1735687826 >> 172.21.15.79:0/1735687826 conn(0x7f6618102910 msgr2=0x7f6618104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:45.947 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661f0c0700 1 -- 172.21.15.79:0/1735687826 shutdown_connections 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661f0c0700 1 -- 172.21.15.79:0/1735687826 wait complete. 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661f0c0700 1 Processor -- start 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661f0c0700 1 -- start start 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661f0c0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f661819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661f0c0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f661807afe0 con 0x7f66181072f0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661ce5c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f661819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661ce5c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f661819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43268/0 (socket says 172.21.15.79:43268) 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.071+0000 7f661ce5c700 1 -- 172.21.15.79:0/990193633 learned_addr learned my addr 172.21.15.79:0/990193633 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f661ce5c700 1 -- 172.21.15.79:0/990193633 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661819a6f0 con 0x7f66181072f0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f661ce5c700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f661819a1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f660c00d650 tx=0x7f660c003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c027070 con 0x7f66181072f0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661819a980 con 0x7f66181072f0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f660c010c00 con 0x7f66181072f0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c022ba0 con 0x7f66181072f0 2024-04-22T06:34:45.948 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.072+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661819d620 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.073+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f660c02a070 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.073+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66180640a0 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.073+0000 7f6615ffb700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 0x7f660803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.074+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f660c01f070 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.074+0000 7f6617fff700 1 -- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 msgr2=0x7f660803ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.074+0000 7f6617fff700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 0x7f660803ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.079+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f660c022e20 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.275+0000 7f6617fff700 1 -- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 msgr2=0x7f660803ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.275+0000 7f6617fff700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 0x7f660803ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.376+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f661819d7f0 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.377+0000 7f6615ffb700 1 -- 172.21.15.79:0/990193633 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f661819d7f0 con 0x7f66181072f0 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 msgr2=0x7f660803ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 0x7f660803ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:45.949 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 msgr2=0x7f661819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f661819a1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f660c00d650 tx=0x7f660c003ff0 comp rx=0 tx=0).stop 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 shutdown_connections 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f6608039560 0x7f660803ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 --2- 172.21.15.79:0/990193633 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66181072f0 0x7f661819a1b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 >> 172.21.15.79:0/990193633 conn(0x7f6618102910 msgr2=0x7f66181907d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 shutdown_connections 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:45.379+0000 7f661f0c0700 1 -- 172.21.15.79:0/990193633 wait complete. 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-04-22T06:34:45.950 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 5... 2024-04-22T06:34:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:46 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/990193633' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T06:34:46.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:46.127+0000 7f6dcdecc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:34:46.590 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:46.299+0000 7f6dcdecc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:46.591 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:46.471+0000 7f6dcdecc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:34:46.847 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:46.756+0000 7f6dcdecc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:34:46.847 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:46.848+0000 7f6dcdecc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:34:47.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:47.047+0000 7f6dcdecc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:34:47.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:47.241+0000 7f6dcdecc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:48.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:47.832+0000 7f6dcdecc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:34:48.533 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:48.243+0000 7f6dcdecc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:34:48.533 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:48.337+0000 7f6dcdecc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:34:48.533 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:48.534+0000 7f6dcdecc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:34:49.368 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.045+0000 7f6dcdecc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:34:49.368 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.134+0000 7f6dcdecc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:34:49.703 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.369+0000 7f6dcdecc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:34:49.703 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.499+0000 7f6dcdecc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:34:49.703 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.591+0000 7f6dcdecc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:34:49.703 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.704+0000 7f6dcdecc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:50.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:49.998+0000 7f6dcdecc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:34:50.845 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:50.583+0000 7f6dcdecc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:34:50.845 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:34:50.794+0000 7f6dcdecc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:34:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: Active manager daemon x restarted 2024-04-22T06:34:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: Activating manager daemon x 2024-04-22T06:34:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T06:34:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: mgrmap e6: x(active, starting, since 0.00323806s) 2024-04-22T06:34:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:34:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: Manager daemon x is now available 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:51.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:50 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:34:52.129 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:51 smithi079 ceph-mon[27946]: Found migration_current of "None". Setting to last migration. 2024-04-22T06:34:52.129 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:51 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:34:52.130 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:51 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:52.130 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:51 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:52.130 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:51 smithi079 ceph-mon[27946]: mgrmap e7: x(active, since 1.00551s) 2024-04-22T06:34:52.412 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.386+0000 7f1925f04700 1 Processor -- start 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.386+0000 7f1925f04700 1 -- start start 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.386+0000 7f1925f04700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192010b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f1925f04700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f192007be80 con 0x7f1920108f00 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192010b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192010b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43276/0 (socket says 172.21.15.79:43276) 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191f7fe700 1 -- 172.21.15.79:0/2626311436 learned_addr learned my addr 172.21.15.79:0/2626311436 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191f7fe700 1 -- 172.21.15.79:0/2626311436 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192010b830 con 0x7f1920108f00 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191f7fe700 1 --2- 172.21.15.79:0/2626311436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192010b2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19140098a0 tx=0x7f1914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a372aad0f01002 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:52.413 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191e7fc700 1 -- 172.21.15.79:0/2626311436 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1914004160 con 0x7f1920108f00 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191e7fc700 1 -- 172.21.15.79:0/2626311436 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1914004300 con 0x7f1920108f00 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.387+0000 7f191e7fc700 1 -- 172.21.15.79:0/2626311436 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1914011af0 con 0x7f1920108f00 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 -- 172.21.15.79:0/2626311436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 msgr2=0x7f192010b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 --2- 172.21.15.79:0/2626311436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192010b2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19140098a0 tx=0x7f1914009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 -- 172.21.15.79:0/2626311436 shutdown_connections 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 --2- 172.21.15.79:0/2626311436 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192010b2f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 -- 172.21.15.79:0/2626311436 >> 172.21.15.79:0/2626311436 conn(0x7f1920102950 msgr2=0x7f1920104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 -- 172.21.15.79:0/2626311436 shutdown_connections 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.388+0000 7f1925f04700 1 -- 172.21.15.79:0/2626311436 wait complete. 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f1925f04700 1 Processor -- start 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f1925f04700 1 -- start start 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f1925f04700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f1925f04700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f192010c7f0 con 0x7f1920108f00 2024-04-22T06:34:52.414 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f191f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f191f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192019c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43288/0 (socket says 172.21.15.79:43288) 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f191f7fe700 1 -- 172.21.15.79:0/584265334 learned_addr learned my addr 172.21.15.79:0/584265334 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f191f7fe700 1 -- 172.21.15.79:0/584265334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192019c930 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.389+0000 7f191f7fe700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192019c3f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1914009fb0 tx=0x7f1914011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.390+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19140036a0 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.390+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f191401a430 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.390+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192019cbc0 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.390+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1914019550 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.390+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f192019f8b0 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.390+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f1914019800 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.391+0000 7f191cff9700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.391+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f191404db90 con 0x7f1920108f00 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.391+0000 7f191effd700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 msgr2=0x7f190803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:52.415 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.391+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.391+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 --> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f19200640a0 con 0x7f19080395b0 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.592+0000 7f191effd700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 msgr2=0x7f190803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.592+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.993+0000 7f191effd700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 msgr2=0x7f190803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:46.993+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:47.794+0000 7f191effd700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 msgr2=0x7f190803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:47.794+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.395+0000 7f191effd700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 msgr2=0x7f190803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/874344387 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.395+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.797+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f191402d960 con 0x7f1920108f00 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.797+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 msgr2=0x7f190803ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.797+0000 7f191cff9700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.798+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f191404f110 con 0x7f1920108f00 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.798+0000 7f191cff9700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f190803ccb0 0x7f190803f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.416 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.799+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f191402ee50 con 0x7f190803ccb0 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.799+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f190803ccb0 0x7f190803f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.799+0000 7f191effd700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f190803ccb0 0x7f190803f0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1910003a10 tx=0x7f19100092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.800+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f191402ee50 con 0x7f190803ccb0 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.806+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1920103bf0 con 0x7f190803ccb0 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f191cff9700 1 -- 172.21.15.79:0/584265334 <== mgr.14120 v2:172.21.15.79:6800/3692925838 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f1920103bf0 con 0x7f190803ccb0 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f190803ccb0 msgr2=0x7f190803f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f190803ccb0 0x7f190803f0a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1910003a10 tx=0x7f19100092b0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 msgr2=0x7f192019c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192019c3f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1914009fb0 tx=0x7f1914011940 comp rx=0 tx=0).stop 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 shutdown_connections 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f190803ccb0 0x7f190803f0a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1920108f00 0x7f192019c3f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 --2- 172.21.15.79:0/584265334 >> [v2:172.21.15.79:6800/874344387,v1:172.21.15.79:6801/874344387] conn(0x7f19080395b0 0x7f190803ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.417 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 >> 172.21.15.79:0/584265334 conn(0x7f1920102950 msgr2=0x7f19201034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:52.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 shutdown_connections 2024-04-22T06:34:52.418 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:51.807+0000 7f1925f04700 1 -- 172.21.15.79:0/584265334 wait complete. 2024-04-22T06:34:52.418 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 5 is available 2024-04-22T06:34:52.418 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: [22/Apr/2024:06:34:51] ENGINE Bus STARTING 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: [22/Apr/2024:06:34:51] ENGINE Serving on http://172.21.15.79:8765 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: [22/Apr/2024:06:34:51] ENGINE Serving on https://172.21.15.79:7150 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: [22/Apr/2024:06:34:51] ENGINE Bus STARTED 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:52 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:53.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2f11f700 1 Processor -- start 2024-04-22T06:34:53.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2f11f700 1 -- start start 2024-04-22T06:34:53.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2f11f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc28106b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:53.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2f11f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28107100 con 0x7fcc28106710 2024-04-22T06:34:53.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2cebb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc28106b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2cebb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc28106b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36954/0 (socket says 172.21.15.79:36954) 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2cebb700 1 -- 172.21.15.79:0/2164702677 learned_addr learned my addr 172.21.15.79:0/2164702677 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.808+0000 7fcc2cebb700 1 -- 172.21.15.79:0/2164702677 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc28107280 con 0x7fcc28106710 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2cebb700 1 --2- 172.21.15.79:0/2164702677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc28106b30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0098a0 tx=0x7fcc1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925d0e7bed520585 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc277fe700 1 -- 172.21.15.79:0/2164702677 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc1c004160 con 0x7fcc28106710 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc277fe700 1 -- 172.21.15.79:0/2164702677 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc1c004300 con 0x7fcc28106710 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc277fe700 1 -- 172.21.15.79:0/2164702677 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc1c011af0 con 0x7fcc28106710 2024-04-22T06:34:53.641 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2164702677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 msgr2=0x7fcc28106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 --2- 172.21.15.79:0/2164702677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc28106b30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0098a0 tx=0x7fcc1c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2164702677 shutdown_connections 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 --2- 172.21.15.79:0/2164702677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc28106b30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2164702677 >> 172.21.15.79:0/2164702677 conn(0x7fcc28102110 msgr2=0x7fcc28104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2164702677 shutdown_connections 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.809+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2164702677 wait complete. 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2f11f700 1 Processor -- start 2024-04-22T06:34:53.642 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2f11f700 1 -- start start 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2f11f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc2819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2f11f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2810a570 con 0x7fcc28106710 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2cebb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc2819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2cebb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc2819c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36966/0 (socket says 172.21.15.79:36966) 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2cebb700 1 -- 172.21.15.79:0/2480306509 learned_addr learned my addr 172.21.15.79:0/2480306509 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.810+0000 7fcc2cebb700 1 -- 172.21.15.79:0/2480306509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2819c880 con 0x7fcc28106710 2024-04-22T06:34:53.643 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.811+0000 7fcc2cebb700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc2819c340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c003f40 tx=0x7fcc1c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.811+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc1c0038b0 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.811+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc1c003a50 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.811+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2819cb70 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.811+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc1c01a5c0 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.811+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2819f800 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.812+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fcc1c024430 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.812+0000 7fcc25ffb700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fcc180394a0 0x7fcc1803b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.812+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc280640a0 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.812+0000 7fcc27fff700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fcc180394a0 0x7fcc1803b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.812+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcc1c04d850 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.812+0000 7fcc27fff700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fcc180394a0 0x7fcc1803b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc14006fd0 tx=0x7fcc14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.817+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc1c02d910 con 0x7fcc28106710 2024-04-22T06:34:53.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.855+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fcc1c02d3f0 con 0x7fcc28106710 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.033+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fcc2819f9d0 con 0x7fcc180394a0 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.038+0000 7fcc25ffb700 1 -- 172.21.15.79:0/2480306509 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcc2819f9d0 con 0x7fcc180394a0 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.040+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fcc180394a0 msgr2=0x7fcc1803b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.040+0000 7fcc2f11f700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fcc180394a0 0x7fcc1803b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc14006fd0 tx=0x7fcc14006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 msgr2=0x7fcc2819c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc2819c340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c003f40 tx=0x7fcc1c011890 comp rx=0 tx=0).stop 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 shutdown_connections 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fcc180394a0 0x7fcc1803b960 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 --2- 172.21.15.79:0/2480306509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc28106710 0x7fcc2819c340 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 >> 172.21.15.79:0/2480306509 conn(0x7fcc28102110 msgr2=0x7fcc28102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 shutdown_connections 2024-04-22T06:34:53.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.041+0000 7fcc2f11f700 1 -- 172.21.15.79:0/2480306509 wait complete. 2024-04-22T06:34:53.963 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:53 smithi079 ceph-mon[27946]: mgrmap e8: x(active, since 2s) 2024-04-22T06:34:53.963 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:53 smithi079 ceph-mon[27946]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:53.963 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:53 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:53.963 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:53 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:54.891 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f5818fee700 1 Processor -- start 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f5818fee700 1 -- start start 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f5818fee700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f5814108d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f5818fee700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f58141092d0 con 0x7f58141088e0 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f581259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f5814108d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f581259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f5814108d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36970/0 (socket says 172.21.15.79:36970) 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f581259c700 1 -- 172.21.15.79:0/3194868451 learned_addr learned my addr 172.21.15.79:0/3194868451 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.064+0000 7f581259c700 1 -- 172.21.15.79:0/3194868451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5814107540 con 0x7f58141088e0 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f581259c700 1 --2- 172.21.15.79:0/3194868451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f5814108d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58040098a0 tx=0x7f5804009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbba95f5beb00f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f581159a700 1 -- 172.21.15.79:0/3194868451 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5804004160 con 0x7f58141088e0 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f581159a700 1 -- 172.21.15.79:0/3194868451 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5804004300 con 0x7f58141088e0 2024-04-22T06:34:54.892 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f581159a700 1 -- 172.21.15.79:0/3194868451 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5804011af0 con 0x7f58141088e0 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 -- 172.21.15.79:0/3194868451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 msgr2=0x7f5814108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 --2- 172.21.15.79:0/3194868451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f5814108d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58040098a0 tx=0x7f5804009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 -- 172.21.15.79:0/3194868451 shutdown_connections 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 --2- 172.21.15.79:0/3194868451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f5814108d00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 -- 172.21.15.79:0/3194868451 >> 172.21.15.79:0/3194868451 conn(0x7f5814102110 msgr2=0x7f5814104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 -- 172.21.15.79:0/3194868451 shutdown_connections 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.065+0000 7f5818fee700 1 -- 172.21.15.79:0/3194868451 wait complete. 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f5818fee700 1 Processor -- start 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f5818fee700 1 -- start start 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f5818fee700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f581419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f5818fee700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f581410b900 con 0x7f58141088e0 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f581259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f581419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.893 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f581259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f581419b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36974/0 (socket says 172.21.15.79:36974) 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.066+0000 7f581259c700 1 -- 172.21.15.79:0/1524986594 learned_addr learned my addr 172.21.15.79:0/1524986594 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f581259c700 1 -- 172.21.15.79:0/1524986594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581419b7b0 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f581259c700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f581419b270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5804003f40 tx=0x7f5804011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58040037d0 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5804003970 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f580401b5e0 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581419ba40 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.067+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5814106430 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.068+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f580401b8e0 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.068+0000 7f58037fe700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f57fc0394c0 0x7f57fc03b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.068+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58140640a0 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.068+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58040501c0 con 0x7f58141088e0 2024-04-22T06:34:54.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.068+0000 7f5811d9b700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f57fc0394c0 0x7f57fc03b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.069+0000 7f5811d9b700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f57fc0394c0 0x7f57fc03b980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5808006fd0 tx=0x7f5808006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.073+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f580400f2e0 con 0x7f58141088e0 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.272+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f5814103310 con 0x7f57fc0394c0 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.272+0000 7f58037fe700 1 -- 172.21.15.79:0/1524986594 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f5814103310 con 0x7f57fc0394c0 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f57fc0394c0 msgr2=0x7f57fc03b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f57fc0394c0 0x7f57fc03b980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5808006fd0 tx=0x7f5808006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 msgr2=0x7f581419b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f581419b270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5804003f40 tx=0x7f5804011b70 comp rx=0 tx=0).stop 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 shutdown_connections 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f57fc0394c0 0x7f57fc03b980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 --2- 172.21.15.79:0/1524986594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58141088e0 0x7f581419b270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.274+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 >> 172.21.15.79:0/1524986594 conn(0x7f5814102110 msgr2=0x7f5814102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:54.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.275+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 shutdown_connections 2024-04-22T06:34:54.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:54.275+0000 7f5818fee700 1 -- 172.21.15.79:0/1524986594 wait complete. 2024-04-22T06:34:54.896 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-04-22T06:34:55.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-mon[27946]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:55.741 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: Generating public/private rsa key pair. 2024-04-22T06:34:55.741 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: Your identification has been saved in /tmp/tmpb9y2b_46/key. 2024-04-22T06:34:55.741 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: Your public key has been saved in /tmp/tmpb9y2b_46/key.pub. 2024-04-22T06:34:55.741 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: The key fingerprint is: 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: SHA256:GgsOa5r5VOTHIgVx/wqAgRVF2gbxUXYXUIlI64sS6Ug ceph-146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: The key's randomart image is: 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: +---[RSA 3072]----+ 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: |o.B*=o+.o++o | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: |.o B +.o... | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: |. o * o | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: | B o . | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: | E= * = S | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: |.o B * * | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: |. * o = | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: | * . | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: |+.. | 2024-04-22T06:34:55.742 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:34:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: +----[SHA256]-----+ 2024-04-22T06:34:56.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f9b91700 1 Processor -- start 2024-04-22T06:34:56.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f9b91700 1 -- start start 2024-04-22T06:34:56.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f9b91700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f4108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.273 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f9b91700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f41f4108e30 con 0x7f41f4108440 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f4108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f4108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36988/0 (socket says 172.21.15.79:36988) 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f37fe700 1 -- 172.21.15.79:0/2956660040 learned_addr learned my addr 172.21.15.79:0/2956660040 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f37fe700 1 -- 172.21.15.79:0/2956660040 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f41095f0 con 0x7f41f4108440 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f37fe700 1 --2- 172.21.15.79:0/2956660040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f4108860 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41e40098a0 tx=0x7f41e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafadeba6c81a4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.284+0000 7f41f2ffd700 1 -- 172.21.15.79:0/2956660040 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e4004160 con 0x7f41f4108440 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f2ffd700 1 -- 172.21.15.79:0/2956660040 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f41e4004300 con 0x7f41f4108440 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f2ffd700 1 -- 172.21.15.79:0/2956660040 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e4011af0 con 0x7f41f4108440 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 -- 172.21.15.79:0/2956660040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 msgr2=0x7f41f4108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 --2- 172.21.15.79:0/2956660040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f4108860 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41e40098a0 tx=0x7f41e4009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 -- 172.21.15.79:0/2956660040 shutdown_connections 2024-04-22T06:34:56.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 --2- 172.21.15.79:0/2956660040 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f4108860 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 -- 172.21.15.79:0/2956660040 >> 172.21.15.79:0/2956660040 conn(0x7f41f4103a00 msgr2=0x7f41f4105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 -- 172.21.15.79:0/2956660040 shutdown_connections 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.285+0000 7f41f9b91700 1 -- 172.21.15.79:0/2956660040 wait complete. 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f9b91700 1 Processor -- start 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f9b91700 1 -- start start 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f9b91700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f407db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f9b91700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f41f410c9a0 con 0x7f41f4108440 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f407db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f407db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36998/0 (socket says 172.21.15.79:36998) 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f37fe700 1 -- 172.21.15.79:0/1886701539 learned_addr learned my addr 172.21.15.79:0/1886701539 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f37fe700 1 -- 172.21.15.79:0/1886701539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f407e0c0 con 0x7f41f4108440 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.286+0000 7f41f37fe700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f407db80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f41e4000c00 tx=0x7f41e4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e4003b20 con 0x7f41f4108440 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f41e401a8c0 con 0x7f41f4108440 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f407c2b0 con 0x7f41f4108440 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e4024940 con 0x7f41f4108440 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f407c7f0 con 0x7f41f4108440 2024-04-22T06:34:56.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f41e4022020 con 0x7f41f4108440 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.287+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f40640a0 con 0x7f41f4108440 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.288+0000 7f41f17fa700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f41e0039600 0x7f41e003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.288+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f41e404d910 con 0x7f41f4108440 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.288+0000 7f41ebfff700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f41e0039600 0x7f41e003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.288+0000 7f41ebfff700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f41e0039600 0x7f41e003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006fd0 tx=0x7f41dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.293+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41e401a610 con 0x7f41f4108440 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.491+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f41f4104c00 con 0x7f41e0039600 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.729+0000 7f41f17fa700 1 -- 172.21.15.79:0/1886701539 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f41f4104c00 con 0x7f41e0039600 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f41e0039600 msgr2=0x7f41e003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f41e0039600 0x7f41e003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006fd0 tx=0x7f41dc006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 msgr2=0x7f41f407db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f407db80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f41e4000c00 tx=0x7f41e4011b70 comp rx=0 tx=0).stop 2024-04-22T06:34:56.279 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 shutdown_connections 2024-04-22T06:34:56.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f41e0039600 0x7f41e003bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 --2- 172.21.15.79:0/1886701539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f41f4108440 0x7f41f407db80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 >> 172.21.15.79:0/1886701539 conn(0x7f41f4103a00 msgr2=0x7f41f41044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:56.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 shutdown_connections 2024-04-22T06:34:56.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.732+0000 7f41f9b91700 1 -- 172.21.15.79:0/1886701539 wait complete. 2024-04-22T06:34:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:56 smithi079 ceph-mon[27946]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:56 smithi079 ceph-mon[27946]: Generating ssh key... 2024-04-22T06:34:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:56 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:56 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:34:57.549 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:57.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afee6a700 1 Processor -- start 2024-04-22T06:34:57.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afee6a700 1 -- start start 2024-04-22T06:34:57.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afee6a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8108d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:57.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afee6a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8af81092f0 con 0x7f8af8108900 2024-04-22T06:34:57.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afcc06700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8108d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:57.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afcc06700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8108d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37000/0 (socket says 172.21.15.79:37000) 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.684+0000 7f8afcc06700 1 -- 172.21.15.79:0/3222211049 learned_addr learned my addr 172.21.15.79:0/3222211049 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8afcc06700 1 -- 172.21.15.79:0/3222211049 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af8107560 con 0x7f8af8108900 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8afcc06700 1 --2- 172.21.15.79:0/3222211049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8108d20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009e80 tx=0x7f8af000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e19bae65e145aa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8af77fe700 1 -- 172.21.15.79:0/3222211049 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af0017070 con 0x7f8af8108900 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8af77fe700 1 -- 172.21.15.79:0/3222211049 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8af000b8c0 con 0x7f8af8108900 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8af77fe700 1 -- 172.21.15.79:0/3222211049 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af0004490 con 0x7f8af8108900 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8afee6a700 1 -- 172.21.15.79:0/3222211049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 msgr2=0x7f8af8108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.685+0000 7f8afee6a700 1 --2- 172.21.15.79:0/3222211049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8108d20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009e80 tx=0x7f8af000b330 comp rx=0 tx=0).stop 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 -- 172.21.15.79:0/3222211049 shutdown_connections 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 --2- 172.21.15.79:0/3222211049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8108d20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 -- 172.21.15.79:0/3222211049 >> 172.21.15.79:0/3222211049 conn(0x7f8af8102110 msgr2=0x7f8af8104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 -- 172.21.15.79:0/3222211049 shutdown_connections 2024-04-22T06:34:57.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 -- 172.21.15.79:0/3222211049 wait complete. 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 Processor -- start 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 -- start start 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afee6a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8af810b920 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afcc06700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afcc06700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37006/0 (socket says 172.21.15.79:37006) 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.686+0000 7f8afcc06700 1 -- 172.21.15.79:0/1798774638 learned_addr learned my addr 172.21.15.79:0/1798774638 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8afcc06700 1 -- 172.21.15.79:0/1798774638 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af819cee0 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8afcc06700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8106e80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8af000b7b0 tx=0x7f8af000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af0020020 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8af0012430 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8af0017040 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af81a14f0 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.687+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af81a1890 con 0x7f8af8108900 2024-04-22T06:34:57.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.688+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8af0023020 con 0x7f8af8108900 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.688+0000 7f8af5ffb700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8ae0039560 0x7f8ae003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.688+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af80640a0 con 0x7f8af8108900 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.688+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8af000f050 con 0x7f8af8108900 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.688+0000 7f8af7fff700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8ae0039560 0x7f8ae003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.689+0000 7f8af7fff700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8ae0039560 0x7f8ae003ba20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8aec006fd0 tx=0x7f8aec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.693+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8af002a020 con 0x7f8af8108900 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.892+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8af81a1b20 con 0x7f8ae0039560 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.893+0000 7f8af5ffb700 1 -- 172.21.15.79:0/1798774638 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f8af81a1b20 con 0x7f8ae0039560 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8ae0039560 msgr2=0x7f8ae003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8ae0039560 0x7f8ae003ba20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8aec006fd0 tx=0x7f8aec006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 msgr2=0x7f8af8106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8106e80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8af000b7b0 tx=0x7f8af000be20 comp rx=0 tx=0).stop 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 shutdown_connections 2024-04-22T06:34:57.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8ae0039560 0x7f8ae003ba20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 --2- 172.21.15.79:0/1798774638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8af8108900 0x7f8af8106e80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.895+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 >> 172.21.15.79:0/1798774638 conn(0x7f8af8102110 msgr2=0x7f8af8102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.896+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 shutdown_connections 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.896+0000 7f8afee6a700 1 -- 172.21.15.79:0/1798774638 wait complete. 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T06:34:57.554 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-04-22T06:34:57.985 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:57 smithi079 ceph-mon[27946]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:59 smithi079 ceph-mon[27946]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:34:59 smithi079 ceph-mon[27946]: Deploying cephadm binary to smithi079 2024-04-22T06:35:02.807 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.013+0000 7f4f1bc37700 1 Processor -- start 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f1bc37700 1 -- start start 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f1bc37700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1410b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f1bc37700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1407bec0 con 0x7f4f14108ec0 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f199d3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1410b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f199d3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1410b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37012/0 (socket says 172.21.15.79:37012) 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f199d3700 1 -- 172.21.15.79:0/4259613591 learned_addr learned my addr 172.21.15.79:0/4259613591 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f199d3700 1 -- 172.21.15.79:0/4259613591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1410b7f0 con 0x7f4f14108ec0 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.014+0000 7f4f199d3700 1 --2- 172.21.15.79:0/4259613591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1410b2b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f040098a0 tx=0x7f4f04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38de9f745bf372cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f189d1700 1 -- 172.21.15.79:0/4259613591 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f04004160 con 0x7f4f14108ec0 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f189d1700 1 -- 172.21.15.79:0/4259613591 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4f04004300 con 0x7f4f14108ec0 2024-04-22T06:35:02.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f189d1700 1 -- 172.21.15.79:0/4259613591 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f04011af0 con 0x7f4f14108ec0 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 -- 172.21.15.79:0/4259613591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 msgr2=0x7f4f1410b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 --2- 172.21.15.79:0/4259613591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1410b2b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f040098a0 tx=0x7f4f04009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 -- 172.21.15.79:0/4259613591 shutdown_connections 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 --2- 172.21.15.79:0/4259613591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1410b2b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 -- 172.21.15.79:0/4259613591 >> 172.21.15.79:0/4259613591 conn(0x7f4f14102950 msgr2=0x7f4f14104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 -- 172.21.15.79:0/4259613591 shutdown_connections 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.015+0000 7f4f1bc37700 1 -- 172.21.15.79:0/4259613591 wait complete. 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f1bc37700 1 Processor -- start 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f1bc37700 1 -- start start 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f1bc37700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f1bc37700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1410c7c0 con 0x7f4f14108ec0 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f199d3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f199d3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1419c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37018/0 (socket says 172.21.15.79:37018) 2024-04-22T06:35:02.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f199d3700 1 -- 172.21.15.79:0/3311932914 learned_addr learned my addr 172.21.15.79:0/3311932914 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.016+0000 7f4f199d3700 1 -- 172.21.15.79:0/3311932914 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1419c910 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.017+0000 7f4f199d3700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1419c3d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009fb0 tx=0x7f4f04011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.017+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f040036a0 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.017+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4f0401a430 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.017+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1419cba0 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.017+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f04019550 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.017+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1419f890 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.018+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4f04003840 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.018+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f140640a0 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.018+0000 7f4f0affd700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4f000395b0 0x7f4f0003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.018+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4f0404d8a0 con 0x7f4f14108ec0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.018+0000 7f4f191d2700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4f000395b0 0x7f4f0003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.018+0000 7f4f191d2700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4f000395b0 0x7f4f0003ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f10006fd0 tx=0x7f4f10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:02.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.023+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f040172b0 con 0x7f4f14108ec0 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.223+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}) -- 0x7f4f14103c10 con 0x7f4f000395b0 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.224+0000 7f4f0affd700 1 -- 172.21.15.79:0/3311932914 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4f14103c10 con 0x7f4f000395b0 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4f000395b0 msgr2=0x7f4f0003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4f000395b0 0x7f4f0003ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f10006fd0 tx=0x7f4f10006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 msgr2=0x7f4f1419c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1419c3d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009fb0 tx=0x7f4f04011940 comp rx=0 tx=0).stop 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 shutdown_connections 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4f000395b0 0x7f4f0003ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 --2- 172.21.15.79:0/3311932914 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f14108ec0 0x7f4f1419c3d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 >> 172.21.15.79:0/3311932914 conn(0x7f4f14102950 msgr2=0x7f4f14103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 shutdown_connections 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.227+0000 7f4f1bc37700 1 -- 172.21.15.79:0/3311932914 wait complete. 2024-04-22T06:35:02.811 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mon service... 2024-04-22T06:35:03.349 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:03 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:03.349 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:03 smithi079 ceph-mon[27946]: Added host smithi079 2024-04-22T06:35:03.349 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:03 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T06:35:04.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97de91c700 1 Processor -- start 2024-04-22T06:35:04.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97de91c700 1 -- start start 2024-04-22T06:35:04.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97de91c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d810b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97de91c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f97d807be40 con 0x7f97d8108f40 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97d7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d810b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97d7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d810b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54672/0 (socket says 172.21.15.79:54672) 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97d7fff700 1 -- 172.21.15.79:0/2361154368 learned_addr learned my addr 172.21.15.79:0/2361154368 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97d7fff700 1 -- 172.21.15.79:0/2361154368 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d810b870 con 0x7f97d8108f40 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.348+0000 7f97d7fff700 1 --2- 172.21.15.79:0/2361154368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d810b330 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97c80098a0 tx=0x7f97c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611f52cf9a6fca59 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97d6ffd700 1 -- 172.21.15.79:0/2361154368 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97c8004160 con 0x7f97d8108f40 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97d6ffd700 1 -- 172.21.15.79:0/2361154368 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f97c8004300 con 0x7f97d8108f40 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97d6ffd700 1 -- 172.21.15.79:0/2361154368 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97c8011af0 con 0x7f97d8108f40 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 -- 172.21.15.79:0/2361154368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 msgr2=0x7f97d810b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 --2- 172.21.15.79:0/2361154368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d810b330 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97c80098a0 tx=0x7f97c8009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 -- 172.21.15.79:0/2361154368 shutdown_connections 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 --2- 172.21.15.79:0/2361154368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d810b330 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 -- 172.21.15.79:0/2361154368 >> 172.21.15.79:0/2361154368 conn(0x7f97d8102950 msgr2=0x7f97d8104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 -- 172.21.15.79:0/2361154368 shutdown_connections 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.349+0000 7f97de91c700 1 -- 172.21.15.79:0/2361154368 wait complete. 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97de91c700 1 Processor -- start 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97de91c700 1 -- start start 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97de91c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97de91c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f97d810c7d0 con 0x7f97d8108f40 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97d7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97d7fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54680/0 (socket says 172.21.15.79:54680) 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.350+0000 7f97d7fff700 1 -- 172.21.15.79:0/872684837 learned_addr learned my addr 172.21.15.79:0/872684837 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97d7fff700 1 -- 172.21.15.79:0/872684837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d819c940 con 0x7f97d8108f40 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97d7fff700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d819c400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009fb0 tx=0x7f97c8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97c80036a0 con 0x7f97d8108f40 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f97c801a430 con 0x7f97d8108f40 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97c8019550 con 0x7f97d8108f40 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d819cbd0 con 0x7f97d8108f40 2024-04-22T06:35:04.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.351+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d819f8c0 con 0x7f97d8108f40 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.352+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f97c8019800 con 0x7f97d8108f40 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.352+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d80640a0 con 0x7f97d8108f40 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.352+0000 7f97d57fa700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f97c00395b0 0x7f97c003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.352+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f97c804da10 con 0x7f97d8108f40 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.352+0000 7f97d77fe700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f97c00395b0 0x7f97c003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.353+0000 7f97d77fe700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f97c00395b0 0x7f97c003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f97cc006fd0 tx=0x7f97cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.357+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97c801a5e0 con 0x7f97d8108f40 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.577+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f97d8103bf0 con 0x7f97c00395b0 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.582+0000 7f97d57fa700 1 -- 172.21.15.79:0/872684837 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f97d8103bf0 con 0x7f97c00395b0 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f97c00395b0 msgr2=0x7f97c003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f97c00395b0 0x7f97c003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f97cc006fd0 tx=0x7f97cc006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 msgr2=0x7f97d819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d819c400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009fb0 tx=0x7f97c8011940 comp rx=0 tx=0).stop 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 shutdown_connections 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f97c00395b0 0x7f97c003ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 --2- 172.21.15.79:0/872684837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97d8108f40 0x7f97d819c400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 >> 172.21.15.79:0/872684837 conn(0x7f97d8102950 msgr2=0x7f97d81034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 shutdown_connections 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.584+0000 7f97de91c700 1 -- 172.21.15.79:0/872684837 wait complete. 2024-04-22T06:35:04.307 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mgr service... 2024-04-22T06:35:04.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:04 smithi079 ceph-mon[27946]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:04.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:04 smithi079 ceph-mon[27946]: Saving service mon spec with placement count:5 2024-04-22T06:35:04.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:04 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.727+0000 7f4e5263a700 1 Processor -- start 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.727+0000 7f4e5263a700 1 -- start start 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.727+0000 7f4e5263a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c10b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.727+0000 7f4e5263a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c07bde0 con 0x7f4e4c108f40 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.727+0000 7f4e4bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c10b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:05.629 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.727+0000 7f4e4bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c10b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54686/0 (socket says 172.21.15.79:54686) 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.728+0000 7f4e4bfff700 1 -- 172.21.15.79:0/3481544402 learned_addr learned my addr 172.21.15.79:0/3481544402 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.728+0000 7f4e4bfff700 1 -- 172.21.15.79:0/3481544402 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4c10b870 con 0x7f4e4c108f40 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.728+0000 7f4e4bfff700 1 --2- 172.21.15.79:0/3481544402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c10b330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e340098a0 tx=0x7f4e34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9231224569dd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.728+0000 7f4e4affd700 1 -- 172.21.15.79:0/3481544402 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e34004160 con 0x7f4e4c108f40 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.728+0000 7f4e4affd700 1 -- 172.21.15.79:0/3481544402 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4e34004300 con 0x7f4e4c108f40 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.728+0000 7f4e4affd700 1 -- 172.21.15.79:0/3481544402 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e34011af0 con 0x7f4e4c108f40 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 -- 172.21.15.79:0/3481544402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 msgr2=0x7f4e4c10b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 --2- 172.21.15.79:0/3481544402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c10b330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e340098a0 tx=0x7f4e34009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 -- 172.21.15.79:0/3481544402 shutdown_connections 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 --2- 172.21.15.79:0/3481544402 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c10b330 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 -- 172.21.15.79:0/3481544402 >> 172.21.15.79:0/3481544402 conn(0x7f4e4c102910 msgr2=0x7f4e4c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 -- 172.21.15.79:0/3481544402 shutdown_connections 2024-04-22T06:35:05.630 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.729+0000 7f4e5263a700 1 -- 172.21.15.79:0/3481544402 wait complete. 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e5263a700 1 Processor -- start 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e5263a700 1 -- start start 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e5263a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e5263a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c10c770 con 0x7f4e4c108f40 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e4bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e4bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c19c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54692/0 (socket says 172.21.15.79:54692) 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e4bfff700 1 -- 172.21.15.79:0/2738996112 learned_addr learned my addr 172.21.15.79:0/2738996112 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e4bfff700 1 -- 172.21.15.79:0/2738996112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4c19c900 con 0x7f4e4c108f40 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.730+0000 7f4e4bfff700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c19c3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e34011ac0 tx=0x7f4e34011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.731+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e34003970 con 0x7f4e4c108f40 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.731+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4c19cb90 con 0x7f4e4c108f40 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.731+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4e34003b10 con 0x7f4e4c108f40 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.731+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e34024610 con 0x7f4e4c108f40 2024-04-22T06:35:05.631 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.731+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e4c19f880 con 0x7f4e4c108f40 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.731+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4e34024890 con 0x7f4e4c108f40 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.732+0000 7f4e497fa700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4e380395b0 0x7f4e3803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.732+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4e3404d960 con 0x7f4e4c108f40 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.732+0000 7f4e4b7fe700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4e380395b0 0x7f4e3803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.732+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e4c0640a0 con 0x7f4e4c108f40 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.732+0000 7f4e4b7fe700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4e380395b0 0x7f4e3803ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c006fd0 tx=0x7f4e3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.737+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e340172b0 con 0x7f4e4c108f40 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.974+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4e4c103be0 con 0x7f4e380395b0 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.978+0000 7f4e497fa700 1 -- 172.21.15.79:0/2738996112 <== mgr.14120 v2:172.21.15.79:6800/3692925838 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4e4c103be0 con 0x7f4e380395b0 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.980+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4e380395b0 msgr2=0x7f4e3803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.980+0000 7f4e5263a700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4e380395b0 0x7f4e3803ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c006fd0 tx=0x7f4e3c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.980+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 msgr2=0x7f4e4c19c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.980+0000 7f4e5263a700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c19c3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e34011ac0 tx=0x7f4e34011ba0 comp rx=0 tx=0).stop 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.981+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 shutdown_connections 2024-04-22T06:35:05.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.981+0000 7f4e5263a700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f4e380395b0 0x7f4e3803ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:05.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.981+0000 7f4e5263a700 1 --2- 172.21.15.79:0/2738996112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e4c108f40 0x7f4e4c19c3c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:05.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.981+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 >> 172.21.15.79:0/2738996112 conn(0x7f4e4c102910 msgr2=0x7f4e4c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:05.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.981+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 shutdown_connections 2024-04-22T06:35:05.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:04.981+0000 7f4e5263a700 1 -- 172.21.15.79:0/2738996112 wait complete. 2024-04-22T06:35:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:05 smithi079 ceph-mon[27946]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:05 smithi079 ceph-mon[27946]: Saving service mgr spec with placement count:2 2024-04-22T06:35:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:05 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.049+0000 7f1b8f1b8700 1 Processor -- start 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.049+0000 7f1b8f1b8700 1 -- start start 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8f1b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b88108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8f1b8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1b88108e30 con 0x7f1b88108440 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8cf54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b88108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8cf54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b88108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54698/0 (socket says 172.21.15.79:54698) 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8cf54700 1 -- 172.21.15.79:0/544251048 learned_addr learned my addr 172.21.15.79:0/544251048 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8cf54700 1 -- 172.21.15.79:0/544251048 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b881095f0 con 0x7f1b88108440 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b8cf54700 1 --2- 172.21.15.79:0/544251048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b88108860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0098a0 tx=0x7f1b7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3b1bee266234aa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:07.032 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b877fe700 1 -- 172.21.15.79:0/544251048 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b7c004160 con 0x7f1b88108440 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b877fe700 1 -- 172.21.15.79:0/544251048 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b7c004300 con 0x7f1b88108440 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.050+0000 7f1b877fe700 1 -- 172.21.15.79:0/544251048 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b7c011af0 con 0x7f1b88108440 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/544251048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 msgr2=0x7f1b88108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 --2- 172.21.15.79:0/544251048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b88108860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0098a0 tx=0x7f1b7c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/544251048 shutdown_connections 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 --2- 172.21.15.79:0/544251048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b88108860 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/544251048 >> 172.21.15.79:0/544251048 conn(0x7f1b88103ae0 msgr2=0x7f1b88105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/544251048 shutdown_connections 2024-04-22T06:35:07.033 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.051+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/544251048 wait complete. 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8f1b8700 1 Processor -- start 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8f1b8700 1 -- start start 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8f1b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b8819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8f1b8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8810c9a0 con 0x7f1b88108440 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8cf54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b8819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8cf54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b8819c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54710/0 (socket says 172.21.15.79:54710) 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8cf54700 1 -- 172.21.15.79:0/1933916373 learned_addr learned my addr 172.21.15.79:0/1933916373 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8cf54700 1 -- 172.21.15.79:0/1933916373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8819c880 con 0x7f1b88108440 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8cf54700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b8819c340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c000c00 tx=0x7f1b7c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b7c003b20 con 0x7f1b88108440 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b7c01a8c0 con 0x7f1b88108440 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8819cb10 con 0x7f1b88108440 2024-04-22T06:35:07.034 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.052+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b7c024940 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.053+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8819f800 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.054+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1b7c022020 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.054+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b880640a0 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.054+0000 7f1b85ffb700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1b78039600 0x7f1b7803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.054+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1b7c04d920 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.054+0000 7f1b87fff700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1b78039600 0x7f1b7803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.054+0000 7f1b87fff700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1b78039600 0x7f1b7803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b74006fd0 tx=0x7f1b74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.059+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b7c01aae0 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.259+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1b8819fe50 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.264+0000 7f1b85ffb700 1 -- 172.21.15.79:0/1933916373 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f1b7c00b5a0 con 0x7f1b88108440 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.266+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1b78039600 msgr2=0x7f1b7803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.266+0000 7f1b8f1b8700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1b78039600 0x7f1b7803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b74006fd0 tx=0x7f1b74006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:07.035 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.266+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 msgr2=0x7f1b8819c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.266+0000 7f1b8f1b8700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b8819c340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c000c00 tx=0x7f1b7c011b70 comp rx=0 tx=0).stop 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.266+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 shutdown_connections 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.267+0000 7f1b8f1b8700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1b78039600 0x7f1b7803bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.267+0000 7f1b8f1b8700 1 --2- 172.21.15.79:0/1933916373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b88108440 0x7f1b8819c340 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.267+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 >> 172.21.15.79:0/1933916373 conn(0x7f1b88103ae0 msgr2=0x7f1b881045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.267+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 shutdown_connections 2024-04-22T06:35:07.036 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:06.267+0000 7f1b8f1b8700 1 -- 172.21.15.79:0/1933916373 wait complete. 2024-04-22T06:35:07.369 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:07 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1933916373' entity='client.admin' 2024-04-22T06:35:07.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:07 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:07.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:07 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:08.808 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.473+0000 7f8d96f61700 1 Processor -- start 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.473+0000 7f8d96f61700 1 -- start start 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.473+0000 7f8d96f61700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d90108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.473+0000 7f8d96f61700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8d90108e30 con 0x7f8d90108440 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d94cfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d90108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d94cfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d90108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54712/0 (socket says 172.21.15.79:54712) 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d94cfd700 1 -- 172.21.15.79:0/1904546146 learned_addr learned my addr 172.21.15.79:0/1904546146 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d94cfd700 1 -- 172.21.15.79:0/1904546146 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d901095f0 con 0x7f8d90108440 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d94cfd700 1 --2- 172.21.15.79:0/1904546146 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d90108860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d840098a0 tx=0x7f8d84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73963ea1cb7cbfa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d8f7fe700 1 -- 172.21.15.79:0/1904546146 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d84004160 con 0x7f8d90108440 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d8f7fe700 1 -- 172.21.15.79:0/1904546146 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d84004300 con 0x7f8d90108440 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.474+0000 7f8d8f7fe700 1 -- 172.21.15.79:0/1904546146 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d84011a20 con 0x7f8d90108440 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 -- 172.21.15.79:0/1904546146 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 msgr2=0x7f8d90108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 --2- 172.21.15.79:0/1904546146 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d90108860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d840098a0 tx=0x7f8d84009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.809 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 -- 172.21.15.79:0/1904546146 shutdown_connections 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 --2- 172.21.15.79:0/1904546146 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d90108860 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 -- 172.21.15.79:0/1904546146 >> 172.21.15.79:0/1904546146 conn(0x7f8d90103ae0 msgr2=0x7f8d90105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 -- 172.21.15.79:0/1904546146 shutdown_connections 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.475+0000 7f8d96f61700 1 -- 172.21.15.79:0/1904546146 wait complete. 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d96f61700 1 Processor -- start 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d96f61700 1 -- start start 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d96f61700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d9007db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d96f61700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010c9a0 con 0x7f8d90108440 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d94cfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d9007db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d94cfd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d9007db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54714/0 (socket says 172.21.15.79:54714) 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d94cfd700 1 -- 172.21.15.79:0/2039162354 learned_addr learned my addr 172.21.15.79:0/2039162354 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.476+0000 7f8d94cfd700 1 -- 172.21.15.79:0/2039162354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9007e0c0 con 0x7f8d90108440 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.477+0000 7f8d94cfd700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d9007db80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d840116e0 tx=0x7f8d84011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.810 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.477+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d840039e0 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.477+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8401b8f0 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.477+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9007c2b0 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.477+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d84023760 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.477+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9007c7f0 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.478+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8d8401b440 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.478+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d900640a0 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.478+0000 7f8d8dffb700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8d800395b0 0x7f8d8003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.478+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8d8404dea0 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.478+0000 7f8d8ffff700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8d800395b0 0x7f8d8003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.479+0000 7f8d8ffff700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8d800395b0 0x7f8d8003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006fd0 tx=0x7f8d7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.483+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d8401f030 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.708+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f8d9007cb30 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.711+0000 7f8d8dffb700 1 -- 172.21.15.79:0/2039162354 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f8d9007cb30 con 0x7f8d90108440 2024-04-22T06:35:08.811 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.713+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8d800395b0 msgr2=0x7f8d8003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.713+0000 7f8d96f61700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8d800395b0 0x7f8d8003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006fd0 tx=0x7f8d7c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.713+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 msgr2=0x7f8d9007db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.713+0000 7f8d96f61700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d9007db80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d840116e0 tx=0x7f8d84011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.714+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 shutdown_connections 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.714+0000 7f8d96f61700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f8d800395b0 0x7f8d8003ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.714+0000 7f8d96f61700 1 --2- 172.21.15.79:0/2039162354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8d90108440 0x7f8d9007db80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.714+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 >> 172.21.15.79:0/2039162354 conn(0x7f8d90103ae0 msgr2=0x7f8d90104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.714+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 shutdown_connections 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.714+0000 7f8d96f61700 1 -- 172.21.15.79:0/2039162354 wait complete. 2024-04-22T06:35:08.812 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-04-22T06:35:08.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:08 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2039162354' entity='client.admin' 2024-04-22T06:35:08.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:08 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:09.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:09 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/801506331' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T06:35:09.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:09 smithi079 ceph-mon[27946]: from='mgr.14120 172.21.15.79:0/2325847951' entity='mgr.x' 2024-04-22T06:35:09.890 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: ignoring --setuser ceph since I am not root 2024-04-22T06:35:09.890 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: ignoring --setgroup ceph since I am not root 2024-04-22T06:35:09.890 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:09.890+0000 7f5312b93200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:35:10.178 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:10.076+0000 7f5312b93200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:35:10.178 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:10.179+0000 7f5312b93200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:35:10.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca3b41700 1 Processor -- start 2024-04-22T06:35:10.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca3b41700 1 -- start start 2024-04-22T06:35:10.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca3b41700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c107740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca3b41700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c107d10 con 0x7f1c9c107320 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca18dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c107740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca18dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c107740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54722/0 (socket says 172.21.15.79:54722) 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.261+0000 7f1ca18dd700 1 -- 172.21.15.79:0/4197577122 learned_addr learned my addr 172.21.15.79:0/4197577122 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.262+0000 7f1ca18dd700 1 -- 172.21.15.79:0/4197577122 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c108520 con 0x7f1c9c107320 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.262+0000 7f1ca18dd700 1 --2- 172.21.15.79:0/4197577122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c107740 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c980098a0 tx=0x7f1c98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc73d0765d32be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.262+0000 7f1ca08db700 1 -- 172.21.15.79:0/4197577122 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c98004160 con 0x7f1c9c107320 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.262+0000 7f1ca08db700 1 -- 172.21.15.79:0/4197577122 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c98004300 con 0x7f1c9c107320 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.262+0000 7f1ca08db700 1 -- 172.21.15.79:0/4197577122 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c98011a20 con 0x7f1c9c107320 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 -- 172.21.15.79:0/4197577122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 msgr2=0x7f1c9c107740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 --2- 172.21.15.79:0/4197577122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c107740 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c980098a0 tx=0x7f1c98009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:10.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 -- 172.21.15.79:0/4197577122 shutdown_connections 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 --2- 172.21.15.79:0/4197577122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c107740 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 -- 172.21.15.79:0/4197577122 >> 172.21.15.79:0/4197577122 conn(0x7f1c9c1028a0 msgr2=0x7f1c9c104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 -- 172.21.15.79:0/4197577122 shutdown_connections 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.263+0000 7f1ca3b41700 1 -- 172.21.15.79:0/4197577122 wait complete. 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca3b41700 1 Processor -- start 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca3b41700 1 -- start start 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca3b41700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca3b41700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10b8d0 con 0x7f1c9c107320 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca18dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca18dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54730/0 (socket says 172.21.15.79:54730) 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca18dd700 1 -- 172.21.15.79:0/801506331 learned_addr learned my addr 172.21.15.79:0/801506331 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:10.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca18dd700 1 -- 172.21.15.79:0/801506331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c19c930 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.264+0000 7f1ca18dd700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c19c3f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c980119f0 tx=0x7f1c98011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.265+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c980039e0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.265+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c9801b8f0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.265+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c19cbc0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.265+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c19f8b0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.265+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c980237a0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.266+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1c9802c3f0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.266+0000 7f1c92ffd700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1c880395b0 0x7f1c8803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.266+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c9804edd0 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.266+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c80005330 con 0x7f1c9c107320 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.266+0000 7f1ca10dc700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1c880395b0 0x7f1c8803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.266+0000 7f1ca10dc700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1c880395b0 0x7f1c8803ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c006fd0 tx=0x7f1c8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:10.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.271+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c9801f030 con 0x7f1c9c107320 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.540+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f1c80005620 con 0x7f1c9c107320 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.715+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f1c980292a0 con 0x7f1c9c107320 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.717+0000 7f1c92ffd700 1 -- 172.21.15.79:0/801506331 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f1c980193d0 con 0x7f1c9c107320 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1c880395b0 msgr2=0x7f1c8803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1c880395b0 0x7f1c8803ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c006fd0 tx=0x7f1c8c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 msgr2=0x7f1c9c19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c19c3f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c980119f0 tx=0x7f1c98011ad0 comp rx=0 tx=0).stop 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 shutdown_connections 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f1c880395b0 0x7f1c8803ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 --2- 172.21.15.79:0/801506331 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c9c107320 0x7f1c9c19c3f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.718+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 >> 172.21.15.79:0/801506331 conn(0x7f1c9c1028a0 msgr2=0x7f1c9c1037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.719+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 shutdown_connections 2024-04-22T06:35:10.378 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.719+0000 7f1ca3b41700 1 -- 172.21.15.79:0/801506331 wait complete. 2024-04-22T06:35:10.819 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/801506331' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T06:35:10.819 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:10 smithi079 ceph-mon[27946]: mgrmap e9: x(active, since 18s) 2024-04-22T06:35:10.819 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:10.493+0000 7f5312b93200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:35:10.819 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:10.580+0000 7f5312b93200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.830+0000 7fa9f1099700 1 Processor -- start 2024-04-22T06:35:11.714 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.830+0000 7fa9f1099700 1 -- start start 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.830+0000 7fa9f1099700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.830+0000 7fa9f1099700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec108e30 con 0x7fa9ec108440 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9ead9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9ead9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54750/0 (socket says 172.21.15.79:54750) 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9ead9d700 1 -- 172.21.15.79:0/3609928258 learned_addr learned my addr 172.21.15.79:0/3609928258 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9ead9d700 1 -- 172.21.15.79:0/3609928258 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec1095f0 con 0x7fa9ec108440 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9ead9d700 1 --2- 172.21.15.79:0/3609928258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec108860 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0098a0 tx=0x7fa9dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88efe9b859f50f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9e9d9b700 1 -- 172.21.15.79:0/3609928258 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9dc004160 con 0x7fa9ec108440 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9e9d9b700 1 -- 172.21.15.79:0/3609928258 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9dc004300 con 0x7fa9ec108440 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.831+0000 7fa9e9d9b700 1 -- 172.21.15.79:0/3609928258 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9dc011a20 con 0x7fa9ec108440 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 -- 172.21.15.79:0/3609928258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 msgr2=0x7fa9ec108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 --2- 172.21.15.79:0/3609928258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec108860 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0098a0 tx=0x7fa9dc009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 -- 172.21.15.79:0/3609928258 shutdown_connections 2024-04-22T06:35:11.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 --2- 172.21.15.79:0/3609928258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec108860 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 -- 172.21.15.79:0/3609928258 >> 172.21.15.79:0/3609928258 conn(0x7fa9ec103ae0 msgr2=0x7fa9ec105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 -- 172.21.15.79:0/3609928258 shutdown_connections 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.832+0000 7fa9f1099700 1 -- 172.21.15.79:0/3609928258 wait complete. 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9f1099700 1 Processor -- start 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9f1099700 1 -- start start 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9f1099700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec07db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9f1099700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec10c9a0 con 0x7fa9ec108440 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9ead9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9ead9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec07db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54764/0 (socket says 172.21.15.79:54764) 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9ead9d700 1 -- 172.21.15.79:0/2620421975 learned_addr learned my addr 172.21.15.79:0/2620421975 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.833+0000 7fa9ead9d700 1 -- 172.21.15.79:0/2620421975 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec07e0c0 con 0x7fa9ec108440 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.834+0000 7fa9ead9d700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec07db80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0116e0 tx=0x7fa9dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.834+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9dc0039e0 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.834+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ec07c2b0 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.834+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9dc01b8f0 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.834+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9dc023760 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.834+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ec07c7f0 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.835+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ec0640a0 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.835+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa9dc0239f0 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.835+0000 7fa9d3fff700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 0x7fa9d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.835+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa9dc04df70 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.835+0000 7fa9ea59c700 1 -- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 msgr2=0x7fa9d403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.835+0000 7fa9ea59c700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 0x7fa9d403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.840+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9dc01f030 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.036+0000 7fa9ea59c700 1 -- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 msgr2=0x7fa9d403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.036+0000 7fa9ea59c700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 0x7fa9d403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.132+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa9ec07cb30 con 0x7fa9ec108440 2024-04-22T06:35:11.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.132+0000 7fa9d3fff700 1 -- 172.21.15.79:0/2620421975 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa9ec07cb30 con 0x7fa9ec108440 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 msgr2=0x7fa9d403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 0x7fa9d403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 msgr2=0x7fa9ec07db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec07db80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0116e0 tx=0x7fa9dc011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 shutdown_connections 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7fa9d40395b0 0x7fa9d403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 --2- 172.21.15.79:0/2620421975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa9ec108440 0x7fa9ec07db80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 >> 172.21.15.79:0/2620421975 conn(0x7fa9ec103ae0 msgr2=0x7fa9ec104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 shutdown_connections 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.135+0000 7fa9f1099700 1 -- 172.21.15.79:0/2620421975 wait complete. 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-04-22T06:35:11.718 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-04-22T06:35:11.786 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:11 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2620421975' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T06:35:12.131 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:11.787+0000 7f5312b93200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:35:12.131 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:11.958+0000 7f5312b93200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:35:12.132 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:12.132+0000 7f5312b93200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:35:12.707 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:12.418+0000 7f5312b93200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:35:12.707 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:12.509+0000 7f5312b93200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:35:12.707 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:12.708+0000 7f5312b93200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:35:12.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:12.899+0000 7f5312b93200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:35:13.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:13.484+0000 7f5312b93200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:35:14.212 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:13.912+0000 7f5312b93200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:35:14.212 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:14.010+0000 7f5312b93200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:35:14.212 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:14.213+0000 7f5312b93200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:35:15.096 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:14.755+0000 7f5312b93200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:35:15.096 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:14.850+0000 7f5312b93200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:35:15.437 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:15.097+0000 7f5312b93200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:35:15.437 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:15.231+0000 7f5312b93200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:35:15.437 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:15.326+0000 7f5312b93200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:35:15.437 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:15.438+0000 7f5312b93200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:35:15.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:15.734+0000 7f5312b93200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:35:16.611 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:16.338+0000 7f5312b93200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:35:16.611 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:35:16.559+0000 7f5312b93200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: Active manager daemon x restarted 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: Activating manager daemon x 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: mgrmap e10: x(active, starting, since 0.00351931s) 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: Manager daemon x is now available 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:35:17.911 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:35:17.911 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:17 smithi079 ceph-mon[27946]: [22/Apr/2024:06:35:16] ENGINE Bus STARTING 2024-04-22T06:35:17.912 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:17 smithi079 ceph-mon[27946]: mgrmap e11: x(active, since 1.00571s) 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.135+0000 7f7887ff1700 1 Processor -- start 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.135+0000 7f7887ff1700 1 -- start start 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.135+0000 7f7887ff1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788007c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.135+0000 7f7887ff1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f788007c900 con 0x7f788007de80 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7885d8d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788007c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7885d8d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788007c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54768/0 (socket says 172.21.15.79:54768) 2024-04-22T06:35:18.457 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7885d8d700 1 -- 172.21.15.79:0/353520263 learned_addr learned my addr 172.21.15.79:0/353520263 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7885d8d700 1 -- 172.21.15.79:0/353520263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788007ca80 con 0x7f788007de80 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7885d8d700 1 --2- 172.21.15.79:0/353520263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788007c330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f78700098a0 tx=0x7f7870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622dbc43c890a2f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7884d8b700 1 -- 172.21.15.79:0/353520263 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7870004160 con 0x7f788007de80 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7884d8b700 1 -- 172.21.15.79:0/353520263 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7870004300 con 0x7f788007de80 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.136+0000 7f7884d8b700 1 -- 172.21.15.79:0/353520263 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7870011a20 con 0x7f788007de80 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 -- 172.21.15.79:0/353520263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 msgr2=0x7f788007c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 --2- 172.21.15.79:0/353520263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788007c330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f78700098a0 tx=0x7f7870009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 -- 172.21.15.79:0/353520263 shutdown_connections 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 --2- 172.21.15.79:0/353520263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788007c330 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 -- 172.21.15.79:0/353520263 >> 172.21.15.79:0/353520263 conn(0x7f7880103ae0 msgr2=0x7f7880105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 -- 172.21.15.79:0/353520263 shutdown_connections 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 -- 172.21.15.79:0/353520263 wait complete. 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 Processor -- start 2024-04-22T06:35:18.458 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.137+0000 7f7887ff1700 1 -- start start 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7887ff1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7887ff1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f788010e9c0 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7885d8d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7885d8d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788019e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54784/0 (socket says 172.21.15.79:54784) 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7885d8d700 1 -- 172.21.15.79:0/915128656 learned_addr learned my addr 172.21.15.79:0/915128656 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7885d8d700 1 -- 172.21.15.79:0/915128656 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788019e9f0 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7885d8d700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788019e4b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7870000c00 tx=0x7f7870011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7870003aa0 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788019ec80 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787001b8f0 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7870023870 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.138+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78801a1970 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.139+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f7870023b70 con 0x7f788007de80 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.139+0000 7f7876ffd700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.459 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.139+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 --> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f786c03c1d0 con 0x7f786c039600 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.139+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f787004e030 con 0x7f788007de80 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.139+0000 7f788558c700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 msgr2=0x7f786c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.140+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.341+0000 7f788558c700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 msgr2=0x7f786c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.341+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.742+0000 7f788558c700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 msgr2=0x7f786c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:12.742+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.543+0000 7f788558c700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 msgr2=0x7f786c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.543+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.144+0000 7f788558c700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 msgr2=0x7f786c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3692925838 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.144+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.562+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f787002c3f0 con 0x7f788007de80 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.563+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 msgr2=0x7f786c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.563+0000 7f7876ffd700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.564+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f787004f6b0 con 0x7f788007de80 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.564+0000 7f7876ffd700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f786c03cfe0 0x7f786c03f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.564+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7870012ec0 con 0x7f786c03cfe0 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.564+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f786c03cfe0 0x7f786c03f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.565+0000 7f788558c700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f786c03cfe0 0x7f786c03f3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f787c003a10 tx=0x7f787c0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.565+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f7870012ec0 con 0x7f786c03cfe0 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.572+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f78800640a0 con 0x7f786c03cfe0 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.572+0000 7f7876ffd700 1 -- 172.21.15.79:0/915128656 <== mgr.14152 v2:172.21.15.79:6800/3007546422 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f78800640a0 con 0x7f786c03cfe0 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f786c03cfe0 msgr2=0x7f786c03f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f786c03cfe0 0x7f786c03f3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f787c003a10 tx=0x7f787c0092b0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 msgr2=0x7f788019e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788019e4b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7870000c00 tx=0x7f7870011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 shutdown_connections 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f786c03cfe0 0x7f786c03f3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.461 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f788007de80 0x7f788019e4b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.462 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 --2- 172.21.15.79:0/915128656 >> [v2:172.21.15.79:6800/3692925838,v1:172.21.15.79:6801/3692925838] conn(0x7f786c039600 0x7f786c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.462 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 >> 172.21.15.79:0/915128656 conn(0x7f7880103ae0 msgr2=0x7f7880104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:18.462 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 shutdown_connections 2024-04-22T06:35:18.462 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.573+0000 7f7887ff1700 1 -- 172.21.15.79:0/915128656 wait complete. 2024-04-22T06:35:18.462 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-04-22T06:35:18.462 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-04-22T06:35:18.722 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:18 smithi079 ceph-mon[27946]: [22/Apr/2024:06:35:16] ENGINE Serving on https://172.21.15.79:7150 2024-04-22T06:35:18.722 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:18 smithi079 ceph-mon[27946]: [22/Apr/2024:06:35:16] ENGINE Serving on http://172.21.15.79:8765 2024-04-22T06:35:18.722 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:18 smithi079 ceph-mon[27946]: [22/Apr/2024:06:35:16] ENGINE Bus STARTED 2024-04-22T06:35:18.722 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:18 smithi079 ceph-mon[27946]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T06:35:18.722 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:18 smithi079 ceph-mon[27946]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T06:35:18.722 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T06:35:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.926+0000 7ff1911bf700 1 Processor -- start 2024-04-22T06:35:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.926+0000 7ff1911bf700 1 -- start start 2024-04-22T06:35:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.926+0000 7ff1911bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.926+0000 7ff1911bf700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c107ce0 con 0x7ff18c1072f0 2024-04-22T06:35:19.802 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff18ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff18ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52730/0 (socket says 172.21.15.79:52730) 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff18ad9d700 1 -- 172.21.15.79:0/827848148 learned_addr learned my addr 172.21.15.79:0/827848148 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff18ad9d700 1 -- 172.21.15.79:0/827848148 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c1084f0 con 0x7ff18c1072f0 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff18ad9d700 1 --2- 172.21.15.79:0/827848148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c107710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1800098a0 tx=0x7ff180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8407be1c20b3f84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff189d9b700 1 -- 172.21.15.79:0/827848148 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff180004160 con 0x7ff18c1072f0 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff189d9b700 1 -- 172.21.15.79:0/827848148 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff180004300 con 0x7ff18c1072f0 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.927+0000 7ff189d9b700 1 -- 172.21.15.79:0/827848148 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff180011a20 con 0x7ff18c1072f0 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 -- 172.21.15.79:0/827848148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 msgr2=0x7ff18c107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 --2- 172.21.15.79:0/827848148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c107710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1800098a0 tx=0x7ff180009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 -- 172.21.15.79:0/827848148 shutdown_connections 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 --2- 172.21.15.79:0/827848148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c107710 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:19.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 -- 172.21.15.79:0/827848148 >> 172.21.15.79:0/827848148 conn(0x7ff18c102910 msgr2=0x7ff18c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 -- 172.21.15.79:0/827848148 shutdown_connections 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.928+0000 7ff1911bf700 1 -- 172.21.15.79:0/827848148 wait complete. 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff1911bf700 1 Processor -- start 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff1911bf700 1 -- start start 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff1911bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff1911bf700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c10b8a0 con 0x7ff18c1072f0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff18ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff18ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52736/0 (socket says 172.21.15.79:52736) 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff18ad9d700 1 -- 172.21.15.79:0/711430378 learned_addr learned my addr 172.21.15.79:0/711430378 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.929+0000 7ff18ad9d700 1 -- 172.21.15.79:0/711430378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c19c930 con 0x7ff18c1072f0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.930+0000 7ff18ad9d700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c19c3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff180000c00 tx=0x7ff180011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.930+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff180003aa0 con 0x7ff18c1072f0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.930+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18001b8f0 con 0x7ff18c1072f0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.930+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18c19cbc0 con 0x7ff18c1072f0 2024-04-22T06:35:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.930+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff180023890 con 0x7ff18c1072f0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.930+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18c19f8b0 con 0x7ff18c1072f0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.931+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7ff18002c3f0 con 0x7ff18c1072f0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.931+0000 7ff173fff700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff1740394e0 0x7ff17403b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.931+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff18004edd0 con 0x7ff18c1072f0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.931+0000 7ff18a59c700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff1740394e0 0x7ff17403b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.931+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18c0640a0 con 0x7ff18c1072f0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.932+0000 7ff18a59c700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff1740394e0 0x7ff17403b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff17c006fd0 tx=0x7ff17c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.937+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff18001f030 con 0x7ff18c1072f0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.151+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff18c02e170 con 0x7ff1740394e0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.265+0000 7ff173fff700 1 -- 172.21.15.79:0/711430378 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff18c02e170 con 0x7ff1740394e0 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff1740394e0 msgr2=0x7ff17403b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff1740394e0 0x7ff17403b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff17c006fd0 tx=0x7ff17c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 msgr2=0x7ff18c19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c19c3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff180000c00 tx=0x7ff180011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:19.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 shutdown_connections 2024-04-22T06:35:19.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff1740394e0 0x7ff17403b9a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:19.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 --2- 172.21.15.79:0/711430378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff18c1072f0 0x7ff18c19c3f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:19.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 >> 172.21.15.79:0/711430378 conn(0x7ff18c102910 msgr2=0x7ff18c1037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:19.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 shutdown_connections 2024-04-22T06:35:19.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.267+0000 7ff1911bf700 1 -- 172.21.15.79:0/711430378 wait complete. 2024-04-22T06:35:19.806 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-04-22T06:35:20.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:20 smithi079 ceph-mon[27946]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:20.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:20.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:20.448 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:20 smithi079 ceph-mon[27946]: mgrmap e12: x(active, since 2s) 2024-04-22T06:35:21.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$DzKhXf8Agwy65JaIizqTn.7u5wxYslAZHinTjHuHvZZTkIK/fF/w.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713767720, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T06:35:21.317 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a53993700 1 Processor -- start 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a53993700 1 -- start start 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a53993700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c106b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a53993700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c1070e0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a5172f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c106b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a5172f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c106b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52742/0 (socket says 172.21.15.79:52742) 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.218+0000 7f9a5172f700 1 -- 172.21.15.79:0/1552039985 learned_addr learned my addr 172.21.15.79:0/1552039985 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.219+0000 7f9a5172f700 1 -- 172.21.15.79:0/1552039985 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4c1096d0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.219+0000 7f9a5172f700 1 --2- 172.21.15.79:0/1552039985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c106b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a480098a0 tx=0x7f9a48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778ba54883f3a353 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.219+0000 7f9a43fff700 1 -- 172.21.15.79:0/1552039985 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a48004160 con 0x7f9a4c1066f0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.219+0000 7f9a43fff700 1 -- 172.21.15.79:0/1552039985 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a48004300 con 0x7f9a4c1066f0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.219+0000 7f9a43fff700 1 -- 172.21.15.79:0/1552039985 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a48011a20 con 0x7f9a4c1066f0 2024-04-22T06:35:21.318 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- 172.21.15.79:0/1552039985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 msgr2=0x7f9a4c106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 --2- 172.21.15.79:0/1552039985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c106b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a480098a0 tx=0x7f9a48009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- 172.21.15.79:0/1552039985 shutdown_connections 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 --2- 172.21.15.79:0/1552039985 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c106b10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- 172.21.15.79:0/1552039985 >> 172.21.15.79:0/1552039985 conn(0x7f9a4c102110 msgr2=0x7f9a4c104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- 172.21.15.79:0/1552039985 shutdown_connections 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- 172.21.15.79:0/1552039985 wait complete. 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 Processor -- start 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- start start 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.220+0000 7f9a53993700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c10ca80 con 0x7f9a4c1066f0 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a5172f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a5172f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c19c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52748/0 (socket says 172.21.15.79:52748) 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a5172f700 1 -- 172.21.15.79:0/2512912345 learned_addr learned my addr 172.21.15.79:0/2512912345 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:21.319 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a5172f700 1 -- 172.21.15.79:0/2512912345 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4c19c870 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a5172f700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c19c330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9a480116e0 tx=0x7f9a48011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a480039e0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a4801b8f0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a4c19cb00 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a48023760 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.221+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a4c19f7f0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.222+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f9a480239f0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.222+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a4c0640a0 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.222+0000 7f9a427fc700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380395b0 0x7f9a3803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.222+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9a4804e180 con 0x7f9a4c1066f0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.222+0000 7f9a50f2e700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380395b0 0x7f9a3803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.223+0000 7f9a50f2e700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380395b0 0x7f9a3803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c006fd0 tx=0x7f9a3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:21.320 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.227+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a4801f030 con 0x7f9a4c1066f0 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.449+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f9a4c1032e0 con 0x7f9a380395b0 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.677+0000 7f9a427fc700 1 -- 172.21.15.79:0/2512912345 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f9a4c1032e0 con 0x7f9a380395b0 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380395b0 msgr2=0x7f9a3803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380395b0 0x7f9a3803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c006fd0 tx=0x7f9a3c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 msgr2=0x7f9a4c19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c19c330 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9a480116e0 tx=0x7f9a48011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 shutdown_connections 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380395b0 0x7f9a3803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 --2- 172.21.15.79:0/2512912345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a4c1066f0 0x7f9a4c19c330 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 >> 172.21.15.79:0/2512912345 conn(0x7f9a4c102110 msgr2=0x7f9a4c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 shutdown_connections 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.680+0000 7f9a53993700 1 -- 172.21.15.79:0/2512912345 wait complete. 2024-04-22T06:35:21.321 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-04-22T06:35:21.561 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:21.562 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:21 smithi079 ceph-mon[27946]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:21.562 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:22.678 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:22 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3828705171' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T06:35:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T06:35:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43e1b80700 1 Processor -- start 2024-04-22T06:35:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43e1b80700 1 -- start start 2024-04-22T06:35:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43e1b80700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43e1b80700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc108e30 con 0x7f43dc108440 2024-04-22T06:35:23.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43db7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43db7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52752/0 (socket says 172.21.15.79:52752) 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.764+0000 7f43db7fe700 1 -- 172.21.15.79:0/1271826904 learned_addr learned my addr 172.21.15.79:0/1271826904 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.765+0000 7f43db7fe700 1 -- 172.21.15.79:0/1271826904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc1095f0 con 0x7f43dc108440 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.765+0000 7f43db7fe700 1 --2- 172.21.15.79:0/1271826904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc108860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43d00098a0 tx=0x7f43d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51135a4c56cf5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.765+0000 7f43da7fc700 1 -- 172.21.15.79:0/1271826904 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d0004160 con 0x7f43dc108440 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.765+0000 7f43da7fc700 1 -- 172.21.15.79:0/1271826904 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d0004300 con 0x7f43dc108440 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.765+0000 7f43da7fc700 1 -- 172.21.15.79:0/1271826904 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d0011a20 con 0x7f43dc108440 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 -- 172.21.15.79:0/1271826904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 msgr2=0x7f43dc108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 --2- 172.21.15.79:0/1271826904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc108860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43d00098a0 tx=0x7f43d0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 -- 172.21.15.79:0/1271826904 shutdown_connections 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 --2- 172.21.15.79:0/1271826904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc108860 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 -- 172.21.15.79:0/1271826904 >> 172.21.15.79:0/1271826904 conn(0x7f43dc103a80 msgr2=0x7f43dc105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 -- 172.21.15.79:0/1271826904 shutdown_connections 2024-04-22T06:35:23.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 -- 172.21.15.79:0/1271826904 wait complete. 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.766+0000 7f43e1b80700 1 Processor -- start 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43e1b80700 1 -- start start 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43e1b80700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43e1b80700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc10c9a0 con 0x7f43dc108440 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43db7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43db7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc19c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52762/0 (socket says 172.21.15.79:52762) 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43db7fe700 1 -- 172.21.15.79:0/3828705171 learned_addr learned my addr 172.21.15.79:0/3828705171 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43db7fe700 1 -- 172.21.15.79:0/3828705171 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc19c800 con 0x7f43dc108440 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43db7fe700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc19c2c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f43d00116e0 tx=0x7f43d0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d00039e0 con 0x7f43dc108440 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43dc19ca90 con 0x7f43dc108440 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d001b8f0 con 0x7f43dc108440 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.767+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d0023760 con 0x7f43dc108440 2024-04-22T06:35:23.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.768+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43dc19f780 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.768+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f43d001b440 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.768+0000 7f43d8ff9700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f43c40395b0 0x7f43c403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.768+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43dc0640a0 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.769+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f43d004e020 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.769+0000 7f43daffd700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f43c40395b0 0x7f43c403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.769+0000 7f43daffd700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f43c40395b0 0x7f43c403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f43cc006fd0 tx=0x7f43cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.774+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43d001f030 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.987+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f43dc19f950 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.987+0000 7f43d8ff9700 1 -- 172.21.15.79:0/3828705171 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f43d001bd50 con 0x7f43dc108440 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f43c40395b0 msgr2=0x7f43c403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f43c40395b0 0x7f43c403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f43cc006fd0 tx=0x7f43cc006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 msgr2=0x7f43dc19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:23.350 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc19c2c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f43d00116e0 tx=0x7f43d0011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 shutdown_connections 2024-04-22T06:35:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f43c40395b0 0x7f43c403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 --2- 172.21.15.79:0/3828705171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43dc108440 0x7f43dc19c2c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 >> 172.21.15.79:0/3828705171 conn(0x7f43dc103a80 msgr2=0x7f43dc104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 shutdown_connections 2024-04-22T06:35:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:21.990+0000 7f43e1b80700 1 -- 172.21.15.79:0/3828705171 wait complete. 2024-04-22T06:35:23.355 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:35:23.355 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-04-22T06:35:23.363 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:35:23.363 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-04-22T06:35:23.364 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-04-22T06:35:23.364 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T06:35:23.365 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-04-22T06:35:23.365 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:23.365 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079:8443/ 2024-04-22T06:35:23.365 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-04-22T06:35:23.365 INFO:teuthology.orchestra.run.smithi079.stdout: Password: a4nfxvp4im 2024-04-22T06:35:23.365 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:23.366 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config directory 2024-04-22T06:35:23.366 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-04-22T06:35:24.586 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:24 smithi079 ceph-mon[27946]: mgrmap e13: x(active, since 6s) 2024-04-22T06:35:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.787+0000 7f6c8bd4c700 1 Processor -- start 2024-04-22T06:35:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.787+0000 7f6c8bd4c700 1 -- start start 2024-04-22T06:35:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.787+0000 7f6c8bd4c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8410b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.787+0000 7f6c8bd4c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8407bea0 con 0x7f6c84108e80 2024-04-22T06:35:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c89ae8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8410b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c89ae8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8410b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54450/0 (socket says 172.21.15.79:54450) 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c89ae8700 1 -- 172.21.15.79:0/256407338 learned_addr learned my addr 172.21.15.79:0/256407338 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c89ae8700 1 -- 172.21.15.79:0/256407338 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8410b7b0 con 0x7f6c84108e80 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c89ae8700 1 --2- 172.21.15.79:0/256407338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8410b270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6c740098a0 tx=0x7f6c74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905c4403f17c14c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c88ae6700 1 -- 172.21.15.79:0/256407338 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c74004160 con 0x7f6c84108e80 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c88ae6700 1 -- 172.21.15.79:0/256407338 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c74004300 con 0x7f6c84108e80 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.788+0000 7f6c88ae6700 1 -- 172.21.15.79:0/256407338 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c74011a20 con 0x7f6c84108e80 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/256407338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 msgr2=0x7f6c8410b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 --2- 172.21.15.79:0/256407338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8410b270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6c740098a0 tx=0x7f6c74009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/256407338 shutdown_connections 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 --2- 172.21.15.79:0/256407338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8410b270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/256407338 >> 172.21.15.79:0/256407338 conn(0x7f6c84102910 msgr2=0x7f6c84104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/256407338 shutdown_connections 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.789+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/256407338 wait complete. 2024-04-22T06:35:24.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c8bd4c700 1 Processor -- start 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c8bd4c700 1 -- start start 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c8bd4c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c8bd4c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8410c750 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c89ae8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c89ae8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8419c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54464/0 (socket says 172.21.15.79:54464) 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c89ae8700 1 -- 172.21.15.79:0/3718260868 learned_addr learned my addr 172.21.15.79:0/3718260868 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c89ae8700 1 -- 172.21.15.79:0/3718260868 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8419c810 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c89ae8700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8419c2d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c740116e0 tx=0x7f6c74011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c74003b20 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c7401b8f0 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8419cb00 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c740238c0 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.790+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8419f790 con 0x7f6c84108e80 2024-04-22T06:35:24.596 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.791+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6c7401b440 con 0x7f6c84108e80 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.791+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c840640a0 con 0x7f6c84108e80 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.791+0000 7f6c7affd700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6c700396b0 0x7f6c7003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.791+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6c7404e1c0 con 0x7f6c84108e80 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.792+0000 7f6c892e7700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6c700396b0 0x7f6c7003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.792+0000 7f6c892e7700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6c700396b0 0x7f6c7003bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c80006fd0 tx=0x7f6c80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.796+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c7401f030 con 0x7f6c84108e80 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.997+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f6c8419f960 con 0x7f6c84108e80 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.998+0000 7f6c7affd700 1 -- 172.21.15.79:0/3718260868 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f6c7402c3f0 con 0x7f6c84108e80 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6c700396b0 msgr2=0x7f6c7003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6c700396b0 0x7f6c7003bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c80006fd0 tx=0x7f6c80006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 msgr2=0x7f6c8419c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8419c2d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c740116e0 tx=0x7f6c74011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 shutdown_connections 2024-04-22T06:35:24.597 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6c700396b0 0x7f6c7003bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.598 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 --2- 172.21.15.79:0/3718260868 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c84108e80 0x7f6c8419c2d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.598 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 >> 172.21.15.79:0/3718260868 conn(0x7f6c84102910 msgr2=0x7f6c84103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:24.598 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 shutdown_connections 2024-04-22T06:35:24.598 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.000+0000 7f6c8bd4c700 1 -- 172.21.15.79:0/3718260868 wait complete. 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f393d4cb700 1 Processor -- start 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f393d4cb700 1 -- start start 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f393d4cb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f393d4cb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3938107ce0 con 0x7f39381072f0 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f3936ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f3936ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54478/0 (socket says 172.21.15.79:54478) 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f3936ffd700 1 -- 172.21.15.79:0/3672472740 learned_addr learned my addr 172.21.15.79:0/3672472740 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f3936ffd700 1 -- 172.21.15.79:0/3672472740 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39381084f0 con 0x7f39381072f0 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.002+0000 7f3936ffd700 1 --2- 172.21.15.79:0/3672472740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938107710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39280098a0 tx=0x7f3928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f672d4d401926c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f3935ffb700 1 -- 172.21.15.79:0/3672472740 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3928004160 con 0x7f39381072f0 2024-04-22T06:35:25.868 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f3935ffb700 1 -- 172.21.15.79:0/3672472740 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3928004300 con 0x7f39381072f0 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f3935ffb700 1 -- 172.21.15.79:0/3672472740 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3928011a20 con 0x7f39381072f0 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 -- 172.21.15.79:0/3672472740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 msgr2=0x7f3938107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 --2- 172.21.15.79:0/3672472740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938107710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39280098a0 tx=0x7f3928009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 -- 172.21.15.79:0/3672472740 shutdown_connections 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 --2- 172.21.15.79:0/3672472740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938107710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 -- 172.21.15.79:0/3672472740 >> 172.21.15.79:0/3672472740 conn(0x7f3938102910 msgr2=0x7f3938104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 -- 172.21.15.79:0/3672472740 shutdown_connections 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.003+0000 7f393d4cb700 1 -- 172.21.15.79:0/3672472740 wait complete. 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f393d4cb700 1 Processor -- start 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f393d4cb700 1 -- start start 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f393d4cb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938192460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f393d4cb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f393810b8a0 con 0x7f39381072f0 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f3936ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938192460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.869 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f3936ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938192460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54492/0 (socket says 172.21.15.79:54492) 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f3936ffd700 1 -- 172.21.15.79:0/364623507 learned_addr learned my addr 172.21.15.79:0/364623507 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.004+0000 7f3936ffd700 1 -- 172.21.15.79:0/364623507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39381929a0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.005+0000 7f3936ffd700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938192460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f39280119f0 tx=0x7f3928011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.005+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39280039e0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.005+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392801b8f0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.005+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39280237a0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.005+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3938190b90 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.005+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39381910d0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.006+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f39280193d0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.006+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39380640a0 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.006+0000 7f391ffff700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f39200396b0 0x7f392003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.006+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f392804e290 con 0x7f39381072f0 2024-04-22T06:35:25.870 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.006+0000 7f39367fc700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f39200396b0 0x7f392003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.006+0000 7f39367fc700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f39200396b0 0x7f392003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f392c006fd0 tx=0x7f392c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.011+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f392801f030 con 0x7f39381072f0 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.315+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f3938196150 con 0x7f39381072f0 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.318+0000 7f391ffff700 1 -- 172.21.15.79:0/364623507 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f392801bb10 con 0x7f39381072f0 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f39200396b0 msgr2=0x7f392003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f39200396b0 0x7f392003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f392c006fd0 tx=0x7f392c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 msgr2=0x7f3938192460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938192460 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f39280119f0 tx=0x7f3928011ad0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 shutdown_connections 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f39200396b0 0x7f392003bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 --2- 172.21.15.79:0/364623507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39381072f0 0x7f3938192460 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 >> 172.21.15.79:0/364623507 conn(0x7f3938102910 msgr2=0x7f39381037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 shutdown_connections 2024-04-22T06:35:25.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.320+0000 7f393d4cb700 1 -- 172.21.15.79:0/364623507 wait complete. 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.872 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T06:35:25.873 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:25.873 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-04-22T06:35:25.905 INFO:tasks.cephadm:Fetching config... 2024-04-22T06:35:25.905 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:35:25.905 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T06:35:25.920 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T06:35:25.920 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:35:25.920 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T06:35:25.975 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T06:35:25.975 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:35:25.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T06:35:26.039 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T06:35:26.039 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:35:26.039 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T06:35:26.094 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T06:35:26.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:35:26.165 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:26.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:35:26.216 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:26.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:35:26.265 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmv8IaYpd8ykJuyofvAVgLg89BVbYVHHh2z6lrYIvdP/eWprLfynrL2RRsIrnyGadui9nCvD+qwl5fDl90lSM/q4jksgVTkkDVtVpoeArfxmvutoQF8HNT416W92SjnGfru/HL7h3O0t0FJwtoJgU+WO5j4gmAA2uc2ePS7iT9EQXNsIk+nbiddOZwGHfD/EdCNLnFHEQOpaRb+K3Qs43WQJeQ21BVYicvJ0uVtt+EZG53BYNKeXW6tMn5PpC6DhzrkWnusjSnJfXGmQnzEUXUvU2s7aP8wgbzYojjsAS0cJJtuNC+e6KQh2gYPft1lazTpoWKNlvaVDH0oHn+I+mHm8hGzswZVUR9+GeWSZ4A5beT1JRvKDZIN6q20sNFuO74/1RoxJoac+X2yyfIuyXiorKNQEEFoEzcsX+FX+AXovKAnH0oI4xktF2etOuQphMafBg51388jYqy3B24BdhzhdADDnfT0Fswmz9/mdpv0NhmTHlG556ArenglCyy1k= ceph-146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:26.277 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T06:35:26.517 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:26.639 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:26 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/364623507' entity='client.admin' 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 -- 172.21.15.79:0/2925258325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 msgr2=0x7f23501028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 --2- 172.21.15.79:0/2925258325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f23501028e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2340009910 tx=0x7f2340009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 -- 172.21.15.79:0/2925258325 shutdown_connections 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 --2- 172.21.15.79:0/2925258325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f23501028e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 -- 172.21.15.79:0/2925258325 >> 172.21.15.79:0/2925258325 conn(0x7f23500fda60 msgr2=0x7f23500ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 -- 172.21.15.79:0/2925258325 shutdown_connections 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.691+0000 7f23559ee700 1 -- 172.21.15.79:0/2925258325 wait complete. 2024-04-22T06:35:28.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f23559ee700 1 Processor -- start 2024-04-22T06:35:28.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f23559ee700 1 -- start start 2024-04-22T06:35:28.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f23559ee700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f2350197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:28.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f23559ee700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2350108400 con 0x7f23501024c0 2024-04-22T06:35:28.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f234effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f2350197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:28.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f234effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f2350197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54506/0 (socket says 172.21.15.79:54506) 2024-04-22T06:35:28.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.692+0000 7f234effd700 1 -- 172.21.15.79:0/3504041107 learned_addr learned my addr 172.21.15.79:0/3504041107 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:28.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f234effd700 1 -- 172.21.15.79:0/3504041107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2350198050 con 0x7f23501024c0 2024-04-22T06:35:28.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f234effd700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f2350197b10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2340005800 tx=0x7f2340009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:28.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2340027070 con 0x7f23501024c0 2024-04-22T06:35:28.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f234000be10 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23501982e0 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f234000f460 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.693+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235019af80 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.694+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f234002a070 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.694+0000 7f23549ec700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2338039650 0x7f233803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.694+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2350108b80 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.694+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f234001e030 con 0x7f23501024c0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.694+0000 7f234e7fc700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2338039650 0x7f233803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:28.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.695+0000 7f234e7fc700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2338039650 0x7f233803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2344006fd0 tx=0x7f2344006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:28.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.699+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2340013070 con 0x7f23501024c0 2024-04-22T06:35:28.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.897+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f235002e020 con 0x7f23501024c0 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.900+0000 7f23549ec700 1 -- 172.21.15.79:0/3504041107 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2340031030 con 0x7f23501024c0 2024-04-22T06:35:28.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.902+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2338039650 msgr2=0x7f233803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:28.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.902+0000 7f23559ee700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2338039650 0x7f233803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2344006fd0 tx=0x7f2344006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.902+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 msgr2=0x7f2350197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.902+0000 7f23559ee700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f2350197b10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2340005800 tx=0x7f2340009f60 comp rx=0 tx=0).stop 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.903+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 shutdown_connections 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.903+0000 7f23559ee700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2338039650 0x7f233803bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.903+0000 7f23559ee700 1 --2- 172.21.15.79:0/3504041107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23501024c0 0x7f2350197b10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.903+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 >> 172.21.15.79:0/3504041107 conn(0x7f23500fda60 msgr2=0x7f23500fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.903+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 shutdown_connections 2024-04-22T06:35:28.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:28.903+0000 7f23559ee700 1 -- 172.21.15.79:0/3504041107 wait complete. 2024-04-22T06:35:29.638 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:29.638 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:29.638 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:35:29.638 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:29.638 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:29.639 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:29.639 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3504041107' entity='client.admin' 2024-04-22T06:35:29.639 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:29.639 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:29.667 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T06:35:29.667 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T06:35:29.900 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:31.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 -- 172.21.15.79:0/2350966897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 msgr2=0x7f0a5c1017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:31.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 --2- 172.21.15.79:0/2350966897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c1017e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009910 tx=0x7f0a58009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 -- 172.21.15.79:0/2350966897 shutdown_connections 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 --2- 172.21.15.79:0/2350966897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c1017e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 -- 172.21.15.79:0/2350966897 >> 172.21.15.79:0/2350966897 conn(0x7f0a5c0fc980 msgr2=0x7f0a5c0feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 -- 172.21.15.79:0/2350966897 shutdown_connections 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 -- 172.21.15.79:0/2350966897 wait complete. 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 Processor -- start 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.452+0000 7f0a62d5c700 1 -- start start 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a62d5c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c18e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a62d5c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c107300 con 0x7f0a5c1013c0 2024-04-22T06:35:31.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a60af8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c18e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a60af8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c18e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54522/0 (socket says 172.21.15.79:54522) 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a60af8700 1 -- 172.21.15.79:0/309045591 learned_addr learned my addr 172.21.15.79:0/309045591 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a60af8700 1 -- 172.21.15.79:0/309045591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5c18e970 con 0x7f0a5c1013c0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a60af8700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c18e430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0a58005800 tx=0x7f0a58009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a58027070 con 0x7f0a5c1013c0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5c18cb60 con 0x7f0a5c1013c0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a5800be10 con 0x7f0a5c1013c0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.453+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a5800f460 con 0x7f0a5c1013c0 2024-04-22T06:35:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.454+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5c18d0a0 con 0x7f0a5c1013c0 2024-04-22T06:35:31.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.454+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0a5802a070 con 0x7f0a5c1013c0 2024-04-22T06:35:31.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.455+0000 7f0a51ffb700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a44039650 0x7f0a4403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:31.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.455+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0a5801e030 con 0x7f0a5c1013c0 2024-04-22T06:35:31.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.455+0000 7f0a53fff700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a44039650 0x7f0a4403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:31.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.455+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a5c107a80 con 0x7f0a5c1013c0 2024-04-22T06:35:31.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.455+0000 7f0a53fff700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a44039650 0x7f0a4403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c006fd0 tx=0x7f0a4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:31.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.460+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a58031070 con 0x7f0a5c1013c0 2024-04-22T06:35:31.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.663+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f0a5c18d760 con 0x7f0a44039650 2024-04-22T06:35:31.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.666+0000 7f0a51ffb700 1 -- 172.21.15.79:0/309045591 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0a5c18d760 con 0x7f0a44039650 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a44039650 msgr2=0x7f0a4403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a44039650 0x7f0a4403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c006fd0 tx=0x7f0a4c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 msgr2=0x7f0a5c18e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c18e430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0a58005800 tx=0x7f0a58009f60 comp rx=0 tx=0).stop 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 shutdown_connections 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a44039650 0x7f0a4403bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 --2- 172.21.15.79:0/309045591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a5c1013c0 0x7f0a5c18e430 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.668+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 >> 172.21.15.79:0/309045591 conn(0x7f0a5c0fc980 msgr2=0x7f0a5c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:31.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.669+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 shutdown_connections 2024-04-22T06:35:31.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:31.669+0000 7f0a62d5c700 1 -- 172.21.15.79:0/309045591 wait complete. 2024-04-22T06:35:32.449 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-04-22T06:35:32.449 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:35:32.449 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-04-22T06:35:32.468 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:35:32.468 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:32.525 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-04-22T06:35:32.525 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch host add smithi190 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:32.758 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:33.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:33 smithi079 ceph-mon[27946]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:33.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:33 smithi079 ceph-mon[27946]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:33.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:33 smithi079 ceph-mon[27946]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:35:34.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 -- 172.21.15.79:0/972192724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 msgr2=0x7fa11c108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 --2- 172.21.15.79:0/972192724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c108930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa104009910 tx=0x7fa104009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 -- 172.21.15.79:0/972192724 shutdown_connections 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 --2- 172.21.15.79:0/972192724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c108930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 -- 172.21.15.79:0/972192724 >> 172.21.15.79:0/972192724 conn(0x7fa11c0fdf70 msgr2=0x7fa11c100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 -- 172.21.15.79:0/972192724 shutdown_connections 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.256+0000 7fa121e90700 1 -- 172.21.15.79:0/972192724 wait complete. 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa121e90700 1 Processor -- start 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa121e90700 1 -- start start 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa121e90700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa121e90700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10c9c0 con 0x7fa11c108550 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55772/0 (socket says 172.21.15.79:55772) 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.257+0000 7fa11b7fe700 1 -- 172.21.15.79:0/1402820327 learned_addr learned my addr 172.21.15.79:0/1402820327 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa11b7fe700 1 -- 172.21.15.79:0/1402820327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c1983a0 con 0x7fa11c108550 2024-04-22T06:35:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa11b7fe700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c197e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa1040056d0 tx=0x7fa1040057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa104026070 con 0x7fa11c108550 2024-04-22T06:35:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa10400bc00 con 0x7fa11c108550 2024-04-22T06:35:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c198630 con 0x7fa11c108550 2024-04-22T06:35:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa104003920 con 0x7fa11c108550 2024-04-22T06:35:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.258+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c19c730 con 0x7fa11c108550 2024-04-22T06:35:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.259+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa104029070 con 0x7fa11c108550 2024-04-22T06:35:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.259+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11c103bc0 con 0x7fa11c108550 2024-04-22T06:35:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.259+0000 7fa118ff9700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa108039650 0x7fa10803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.259+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa10401e070 con 0x7fa11c108550 2024-04-22T06:35:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.259+0000 7fa11affd700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa108039650 0x7fa10803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.260+0000 7fa11affd700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa108039650 0x7fa10803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa10c006fd0 tx=0x7fa10c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:34.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.264+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa104013070 con 0x7fa11c108550 2024-04-22T06:35:34.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:34.463+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) -- 0x7fa11c1991c0 con 0x7fa108039650 2024-04-22T06:35:35.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:34 smithi079 ceph-mon[27946]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:35 smithi079 ceph-mon[27946]: Deploying cephadm binary to smithi190 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:37 smithi079 ceph-mon[27946]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:38.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.607+0000 7fa118ff9700 1 -- 172.21.15.79:0/1402820327 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa11c1991c0 con 0x7fa108039650 2024-04-22T06:35:38.606 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa108039650 msgr2=0x7fa10803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa108039650 0x7fa10803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa10c006fd0 tx=0x7fa10c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 msgr2=0x7fa11c197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c197e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa1040056d0 tx=0x7fa1040057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 shutdown_connections 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa108039650 0x7fa10803bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 --2- 172.21.15.79:0/1402820327 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa11c108550 0x7fa11c197e60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 >> 172.21.15.79:0/1402820327 conn(0x7fa11c0fdf70 msgr2=0x7fa11c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 shutdown_connections 2024-04-22T06:35:38.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:38.609+0000 7fa121e90700 1 -- 172.21.15.79:0/1402820327 wait complete. 2024-04-22T06:35:39.208 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T06:35:39.438 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:39.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:39 smithi079 ceph-mon[27946]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:39.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:39.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:39 smithi079 ceph-mon[27946]: Added host smithi190 2024-04-22T06:35:39.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:39.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:40.904 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:41.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 -- 172.21.15.79:0/3062106808 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 msgr2=0x7faab41039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 --2- 172.21.15.79:0/3062106808 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab41039d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faab0009910 tx=0x7faab0009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 -- 172.21.15.79:0/3062106808 shutdown_connections 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 --2- 172.21.15.79:0/3062106808 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab41039d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 -- 172.21.15.79:0/3062106808 >> 172.21.15.79:0/3062106808 conn(0x7faab40fce40 msgr2=0x7faab40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 -- 172.21.15.79:0/3062106808 shutdown_connections 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.002+0000 7faabb4c6700 1 -- 172.21.15.79:0/3062106808 wait complete. 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faabb4c6700 1 Processor -- start 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faabb4c6700 1 -- start start 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faabb4c6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab4101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faabb4c6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faab410c990 con 0x7faab41035f0 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faab9262700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab4101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faab9262700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab4101250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55796/0 (socket says 172.21.15.79:55796) 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.003+0000 7faab9262700 1 -- 172.21.15.79:0/353818538 learned_addr learned my addr 172.21.15.79:0/353818538 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faab9262700 1 -- 172.21.15.79:0/353818538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab4103140 con 0x7faab41035f0 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faab9262700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab4101250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faab00056d0 tx=0x7faab00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faab0026070 con 0x7faab41035f0 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab0004370 con 0x7faab41035f0 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab4101790 con 0x7faab41035f0 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faab00039b0 con 0x7faab41035f0 2024-04-22T06:35:41.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.004+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab4101c50 con 0x7faab41035f0 2024-04-22T06:35:41.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.005+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7faab0029070 con 0x7faab41035f0 2024-04-22T06:35:41.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.005+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab406afd0 con 0x7faab41035f0 2024-04-22T06:35:41.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.005+0000 7faaaa7fc700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faaa00396b0 0x7faaa003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.005+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faab001e070 con 0x7faab41035f0 2024-04-22T06:35:41.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.005+0000 7faab8a61700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faaa00396b0 0x7faaa003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.005+0000 7faab8a61700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faaa00396b0 0x7faaa003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:41.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.010+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faab0013070 con 0x7faab41035f0 2024-04-22T06:35:41.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.213+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faab4101ee0 con 0x7faaa00396b0 2024-04-22T06:35:41.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.214+0000 7faaaa7fc700 1 -- 172.21.15.79:0/353818538 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7faab4101ee0 con 0x7faaa00396b0 2024-04-22T06:35:41.212 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:41.213 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-04-22T06:35:41.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faaa00396b0 msgr2=0x7faaa003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faaa00396b0 0x7faaa003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 msgr2=0x7faab4101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab4101250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faab00056d0 tx=0x7faab00057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 shutdown_connections 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faaa00396b0 0x7faaa003bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 --2- 172.21.15.79:0/353818538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faab41035f0 0x7faab4101250 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 >> 172.21.15.79:0/353818538 conn(0x7faab40fce40 msgr2=0x7faab41063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 shutdown_connections 2024-04-22T06:35:41.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:41.216+0000 7faabb4c6700 1 -- 172.21.15.79:0/353818538 wait complete. 2024-04-22T06:35:41.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:41 smithi079 ceph-mon[27946]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:41.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:41.915 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-04-22T06:35:41.915 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:35:41.915 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-04-22T06:35:41.933 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:35:41.933 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:41.991 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-04-22T06:35:41.991 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch host add smithi193 2024-04-22T06:35:42.225 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:42.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:42 smithi079 ceph-mon[27946]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.706+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/3054931328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 msgr2=0x7fb7ac101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.706+0000 7fb7b0d0b700 1 --2- 172.21.15.79:0/3054931328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac101990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009910 tx=0x7fb79c009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/3054931328 shutdown_connections 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 --2- 172.21.15.79:0/3054931328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac101990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/3054931328 >> 172.21.15.79:0/3054931328 conn(0x7fb7ac0fce60 msgr2=0x7fb7ac0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/3054931328 shutdown_connections 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/3054931328 wait complete. 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 Processor -- start 2024-04-22T06:35:43.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.707+0000 7fb7b0d0b700 1 -- start start 2024-04-22T06:35:43.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7b0d0b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:43.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7b0d0b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac10b830 con 0x7fb7ac1015b0 2024-04-22T06:35:43.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7aa59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:43.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7aa59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41104/0 (socket says 172.21.15.79:41104) 2024-04-22T06:35:43.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7aa59c700 1 -- 172.21.15.79:0/2239956233 learned_addr learned my addr 172.21.15.79:0/2239956233 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:43.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7aa59c700 1 -- 172.21.15.79:0/2239956233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ac1983d0 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7aa59c700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac197e90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0056d0 tx=0x7fb79c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb79c026070 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb79c00bba0 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ac1986c0 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb79c003920 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.708+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ac19c760 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.709+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb79c029070 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.709+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ac102a90 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.710+0000 7fb79b7fe700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7940396b0 0x7fb79403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.710+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb79c01e070 con 0x7fb7ac1015b0 2024-04-22T06:35:43.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.710+0000 7fb7a9d9b700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7940396b0 0x7fb79403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:43.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.710+0000 7fb7a9d9b700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7940396b0 0x7fb79403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0006fd0 tx=0x7fb7a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:43.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.714+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb79c013070 con 0x7fb7ac1015b0 2024-04-22T06:35:43.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:43.916+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) -- 0x7fb7ac1991f0 con 0x7fb7940396b0 2024-04-22T06:35:43.916 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:43 smithi079 ceph-mon[27946]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:35:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: Deploying cephadm binary to smithi193 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:45 smithi079 ceph-mon[27946]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:46 smithi079 ceph-mon[27946]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:35:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:47 smithi079 ceph-mon[27946]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:48.040 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-04-22T06:35:48.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.041+0000 7fb79b7fe700 1 -- 172.21.15.79:0/2239956233 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb7ac1991f0 con 0x7fb7940396b0 2024-04-22T06:35:48.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7940396b0 msgr2=0x7fb79403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:48.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7940396b0 0x7fb79403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0006fd0 tx=0x7fb7a0006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:48.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 msgr2=0x7fb7ac197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:48.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac197e90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0056d0 tx=0x7fb79c0057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 shutdown_connections 2024-04-22T06:35:48.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7940396b0 0x7fb79403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 --2- 172.21.15.79:0/2239956233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7ac1015b0 0x7fb7ac197e90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.043+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 >> 172.21.15.79:0/2239956233 conn(0x7fb7ac0fce60 msgr2=0x7fb7ac06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:48.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.044+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 shutdown_connections 2024-04-22T06:35:48.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:48.044+0000 7fb7b0d0b700 1 -- 172.21.15.79:0/2239956233 wait complete. 2024-04-22T06:35:48.724 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:48.724 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:48.725 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:48.725 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:48.725 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.752 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T06:35:48.982 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:49.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:49 smithi079 ceph-mon[27946]: Added host smithi193 2024-04-22T06:35:49.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:49 smithi079 ceph-mon[27946]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:49.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:49.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 -- 172.21.15.79:0/3123239648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 msgr2=0x7f94dc101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 --2- 172.21.15.79:0/3123239648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc101990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009910 tx=0x7f94cc009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 -- 172.21.15.79:0/3123239648 shutdown_connections 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 --2- 172.21.15.79:0/3123239648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc101990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 -- 172.21.15.79:0/3123239648 >> 172.21.15.79:0/3123239648 conn(0x7f94dc0fce40 msgr2=0x7f94dc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 -- 172.21.15.79:0/3123239648 shutdown_connections 2024-04-22T06:35:50.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.644+0000 7f94e1d70700 1 -- 172.21.15.79:0/3123239648 wait complete. 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.645+0000 7f94e1d70700 1 Processor -- start 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.645+0000 7f94e1d70700 1 -- start start 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.645+0000 7f94e1d70700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.645+0000 7f94e1d70700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc10b830 con 0x7f94dc1015b0 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.645+0000 7f94db7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94db7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41134/0 (socket says 172.21.15.79:41134) 2024-04-22T06:35:50.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94db7fe700 1 -- 172.21.15.79:0/397847839 learned_addr learned my addr 172.21.15.79:0/397847839 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94db7fe700 1 -- 172.21.15.79:0/397847839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc1983c0 con 0x7f94dc1015b0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94db7fe700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc197e80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0056d0 tx=0x7f94cc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94cc026070 con 0x7f94dc1015b0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94cc00bba0 con 0x7f94dc1015b0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94cc003920 con 0x7f94dc1015b0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94dc1986b0 con 0x7f94dc1015b0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.646+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94dc19c750 con 0x7f94dc1015b0 2024-04-22T06:35:50.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.647+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f94cc029070 con 0x7f94dc1015b0 2024-04-22T06:35:50.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.647+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94dc102a90 con 0x7f94dc1015b0 2024-04-22T06:35:50.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.647+0000 7f94d8ff9700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f94c4039650 0x7f94c403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:50.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.647+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f94cc01e070 con 0x7f94dc1015b0 2024-04-22T06:35:50.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.647+0000 7f94daffd700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f94c4039650 0x7f94c403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:50.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.648+0000 7f94daffd700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f94c4039650 0x7f94c403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94d0006fd0 tx=0x7f94d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:50.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.652+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94cc013070 con 0x7f94dc1015b0 2024-04-22T06:35:50.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.854+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f94dc1991e0 con 0x7f94c4039650 2024-04-22T06:35:50.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.855+0000 7f94d8ff9700 1 -- 172.21.15.79:0/397847839 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f94dc1991e0 con 0x7f94c4039650 2024-04-22T06:35:50.854 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:35:50.854 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-04-22T06:35:50.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f94c4039650 msgr2=0x7f94c403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:50.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f94c4039650 0x7f94c403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94d0006fd0 tx=0x7f94d0006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:50.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 msgr2=0x7f94dc197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc197e80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0056d0 tx=0x7f94cc0057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 shutdown_connections 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f94c4039650 0x7f94c403bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 --2- 172.21.15.79:0/397847839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94dc1015b0 0x7f94dc197e80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.857+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 >> 172.21.15.79:0/397847839 conn(0x7f94dc0fce40 msgr2=0x7f94dc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.858+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 shutdown_connections 2024-04-22T06:35:50.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:50.858+0000 7f94e1d70700 1 -- 172.21.15.79:0/397847839 wait complete. 2024-04-22T06:35:51.466 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T06:35:51.467 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T06:35:51.701 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:51.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:51 smithi079 ceph-mon[27946]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:51.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:51 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:52.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:52 smithi079 ceph-mon[27946]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:53.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 -- 172.21.15.79:0/596830721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 msgr2=0x7f3bf41019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 --2- 172.21.15.79:0/596830721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf41019d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009910 tx=0x7f3be4009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:53.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 -- 172.21.15.79:0/596830721 shutdown_connections 2024-04-22T06:35:53.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 --2- 172.21.15.79:0/596830721 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf41019d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 -- 172.21.15.79:0/596830721 >> 172.21.15.79:0/596830721 conn(0x7f3bf4076ea0 msgr2=0x7f3bf40772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 -- 172.21.15.79:0/596830721 shutdown_connections 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.199+0000 7f3bf89ca700 1 -- 172.21.15.79:0/596830721 wait complete. 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf89ca700 1 Processor -- start 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf89ca700 1 -- start start 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf89ca700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf89ca700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf410b820 con 0x7f3bf41015f0 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf419a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44232/0 (socket says 172.21.15.79:44232) 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf259c700 1 -- 172.21.15.79:0/4183001216 learned_addr learned my addr 172.21.15.79:0/4183001216 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf259c700 1 -- 172.21.15.79:0/4183001216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf419a5f0 con 0x7f3bf41015f0 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3bf259c700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf419a0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3be4006010 tx=0x7f3be4005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:53.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.200+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3be4026070 con 0x7f3bf41015f0 2024-04-22T06:35:53.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.201+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf419a8e0 con 0x7f3bf41015f0 2024-04-22T06:35:53.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.201+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3be400bba0 con 0x7f3bf41015f0 2024-04-22T06:35:53.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.201+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3be4003910 con 0x7f3bf41015f0 2024-04-22T06:35:53.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.201+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf419e980 con 0x7f3bf41015f0 2024-04-22T06:35:53.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.201+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3be4029070 con 0x7f3bf41015f0 2024-04-22T06:35:53.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.201+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf4102a80 con 0x7f3bf41015f0 2024-04-22T06:35:53.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.202+0000 7f3be37fe700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3bdc039650 0x7f3bdc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:53.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.202+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3be401e070 con 0x7f3bf41015f0 2024-04-22T06:35:53.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.202+0000 7f3bf1d9b700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3bdc039650 0x7f3bdc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:53.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.202+0000 7f3bf1d9b700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3bdc039650 0x7f3bdc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3be8006fd0 tx=0x7f3be8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:53.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.207+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3be4013070 con 0x7f3bf41015f0 2024-04-22T06:35:53.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.411+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3bf419b410 con 0x7f3bf41015f0 2024-04-22T06:35:53.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.662+0000 7f3be37fe700 1 -- 172.21.15.79:0/4183001216 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f3be4030090 con 0x7f3bf41015f0 2024-04-22T06:35:53.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:53 smithi079 ceph-mon[27946]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:53.661 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:53 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4183001216' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T06:35:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.664+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3bdc039650 msgr2=0x7f3bdc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.664+0000 7f3bf89ca700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3bdc039650 0x7f3bdc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3be8006fd0 tx=0x7f3be8006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.664+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 msgr2=0x7f3bf419a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.664+0000 7f3bf89ca700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf419a0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3be4006010 tx=0x7f3be4005780 comp rx=0 tx=0).stop 2024-04-22T06:35:53.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.665+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 shutdown_connections 2024-04-22T06:35:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.665+0000 7f3bf89ca700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3bdc039650 0x7f3bdc03bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.665+0000 7f3bf89ca700 1 --2- 172.21.15.79:0/4183001216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3bf41015f0 0x7f3bf419a0b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.665+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 >> 172.21.15.79:0/4183001216 conn(0x7f3bf4076ea0 msgr2=0x7f3bf4069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.665+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 shutdown_connections 2024-04-22T06:35:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:35:53.665+0000 7f3bf89ca700 1 -- 172.21.15.79:0/4183001216 wait complete. 2024-04-22T06:35:53.664 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-04-22T06:35:54.284 INFO:tasks.cephadm:Adding mon.a on smithi079 2024-04-22T06:35:54.284 INFO:tasks.cephadm:Adding mon.b on smithi190 2024-04-22T06:35:54.284 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-04-22T06:35:54.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi079:172.21.15.79=a;smithi190:172.21.15.190=b;smithi193:172.21.15.193=c' 2024-04-22T06:35:54.514 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4183001216' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T06:35:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:35:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:54.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:35:54.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:54.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 -- 172.21.15.193:0/3834006115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 msgr2=0x7ff630101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 --2- 172.21.15.193:0/3834006115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630101990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff624009910 tx=0x7ff624009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 -- 172.21.15.193:0/3834006115 shutdown_connections 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 --2- 172.21.15.193:0/3834006115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630101990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 -- 172.21.15.193:0/3834006115 >> 172.21.15.193:0/3834006115 conn(0x7ff6300fce60 msgr2=0x7ff6300ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 -- 172.21.15.193:0/3834006115 shutdown_connections 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.937+0000 7ff636a9f700 1 -- 172.21.15.193:0/3834006115 wait complete. 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff636a9f700 1 Processor -- start 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff636a9f700 1 -- start start 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff636a9f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.937 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff636a9f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff63010c930 con 0x7ff6301015b0 2024-04-22T06:35:55.938 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff63483b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.938 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff63483b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.193:40290/0 (socket says 172.21.15.193:40290) 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.938+0000 7ff63483b700 1 -- 172.21.15.193:0/3244083444 learned_addr learned my addr 172.21.15.193:0/3244083444 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff63483b700 1 -- 172.21.15.193:0/3244083444 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6301983d0 con 0x7ff6301015b0 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff63483b700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630197e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6240056d0 tx=0x7ff6240057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff624026070 con 0x7ff6301015b0 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6301986c0 con 0x7ff6301015b0 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff62400bba0 con 0x7ff6301015b0 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff624003920 con 0x7ff6301015b0 2024-04-22T06:35:55.939 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.939+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63019c760 con 0x7ff6301015b0 2024-04-22T06:35:55.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.940+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff624029070 con 0x7ff6301015b0 2024-04-22T06:35:55.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.940+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff630102a90 con 0x7ff6301015b0 2024-04-22T06:35:55.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.940+0000 7ff62dffb700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff620039650 0x7ff62003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.940+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff62401e070 con 0x7ff6301015b0 2024-04-22T06:35:55.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.941+0000 7ff62ffff700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff620039650 0x7ff62003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.941+0000 7ff62ffff700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff620039650 0x7ff62003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff61c006fd0 tx=0x7ff61c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:55.945 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:55.946+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff624013070 con 0x7ff6301015b0 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:56.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.154+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi190:172.21.15.190=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) -- 0x7ff6301991f0 con 0x7ff620039650 2024-04-22T06:35:56.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.161+0000 7ff62dffb700 1 -- 172.21.15.193:0/3244083444 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff6301991f0 con 0x7ff620039650 2024-04-22T06:35:56.160 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-04-22T06:35:56.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff620039650 msgr2=0x7ff62003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:56.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff620039650 0x7ff62003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff61c006fd0 tx=0x7ff61c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:56.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 msgr2=0x7ff630197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:56.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630197e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6240056d0 tx=0x7ff6240057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:56.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 shutdown_connections 2024-04-22T06:35:56.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff620039650 0x7ff62003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:56.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 --2- 172.21.15.193:0/3244083444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff6301015b0 0x7ff630197e90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:56.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 >> 172.21.15.193:0/3244083444 conn(0x7ff6300fce60 msgr2=0x7ff630102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:56.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 shutdown_connections 2024-04-22T06:35:56.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:35:56.163+0000 7ff636a9f700 1 -- 172.21.15.193:0/3244083444 wait complete. 2024-04-22T06:35:56.744 DEBUG:teuthology.orchestra.run.smithi190:mon.b> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T06:35:56.747 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T06:35:56.749 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:35:56.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:35:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi190:172.21.15.190=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi190:172.21.15.190=b;smithi193:172.21.15.193=c;count:3 2024-04-22T06:35:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: Deploying daemon mon.c on smithi193 2024-04-22T06:35:57.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:57 smithi079 ceph-mon[27946]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:58.866 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:35:59.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:35:59 smithi079 ceph-mon[27946]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:00.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 -- 172.21.15.193:0/2788414684 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 msgr2=0x7fa8101078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:00.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 --2- 172.21.15.193:0/2788414684 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa8101078c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa800009910 tx=0x7fa800009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:00.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 -- 172.21.15.193:0/2788414684 shutdown_connections 2024-04-22T06:36:00.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 --2- 172.21.15.193:0/2788414684 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa8101078c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:00.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 -- 172.21.15.193:0/2788414684 >> 172.21.15.193:0/2788414684 conn(0x7fa8100fcf40 msgr2=0x7fa8100ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:00.573 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 -- 172.21.15.193:0/2788414684 shutdown_connections 2024-04-22T06:36:00.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.574+0000 7fa815067700 1 -- 172.21.15.193:0/2788414684 wait complete. 2024-04-22T06:36:00.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.575+0000 7fa815067700 1 Processor -- start 2024-04-22T06:36:00.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.575+0000 7fa815067700 1 -- start start 2024-04-22T06:36:00.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.575+0000 7fa815067700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa810198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:00.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.575+0000 7fa815067700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa81010b940 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80ed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa810198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80ed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa810198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.193:40312/0 (socket says 172.21.15.193:40312) 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80ed9d700 1 -- 172.21.15.193:0/2541653911 learned_addr learned my addr 172.21.15.193:0/2541653911 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80ed9d700 1 -- 172.21.15.193:0/2541653911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa810199300 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80ed9d700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa810198dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa8000056d0 tx=0x7fa8000057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800026070 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8101995f0 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa80000bba0 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800003920 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.576+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa810199ab0 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.577+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa810102b90 con 0x7fa8101074e0 2024-04-22T06:36:00.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.577+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa800029070 con 0x7fa8101074e0 2024-04-22T06:36:00.577 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.577+0000 7fa80cd99700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7fc039650 0x7fa7fc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:00.577 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.578+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa80001e070 con 0x7fa8101074e0 2024-04-22T06:36:00.577 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.578+0000 7fa80659c700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7fc039650 0x7fa7fc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:00.577 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.578+0000 7fa80659c700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7fc039650 0x7fa7fc03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8006fd0 tx=0x7fa7f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:00.581 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.583+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa800013070 con 0x7fa8101074e0 2024-04-22T06:36:00.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.873+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa8100640a0 con 0x7fa8101074e0 2024-04-22T06:36:00.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.874+0000 7fa80cd99700 1 -- 172.21.15.193:0/2541653911 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fa800003bb0 con 0x7fa8101074e0 2024-04-22T06:36:00.872 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:36:00.872 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","modified":"2024-04-22T06:34:18.390130Z","created":"2024-04-22T06:34:18.390130Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T06:36:00.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7fc039650 msgr2=0x7fa7fc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:00.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7fc039650 0x7fa7fc03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8006fd0 tx=0x7fa7f8006e40 comp rx=0 tx=0).stop 2024-04-22T06:36:00.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 msgr2=0x7fa810198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:00.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa810198dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa8000056d0 tx=0x7fa8000057b0 comp rx=0 tx=0).stop 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 shutdown_connections 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7fc039650 0x7fa7fc03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 --2- 172.21.15.193:0/2541653911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa8101074e0 0x7fa810198dc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 >> 172.21.15.193:0/2541653911 conn(0x7fa8100fcf40 msgr2=0x7fa810109060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 shutdown_connections 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:00.876+0000 7fa815067700 1 -- 172.21.15.193:0/2541653911 wait complete. 2024-04-22T06:36:00.875 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-04-22T06:36:02.595 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:36:02.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:36:02.843 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:36:06.296 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:06.296 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: mon.a calling monitor election 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: mon.c calling monitor election 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: monmap epoch 2 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: last_changed 2024-04-22T06:36:00.930289+0000 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:06.297 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: min_mon_release 19 (squid) 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: election_strategy: 1 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: fsmap 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: mgrmap e13: x(active, since 49s) 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: overall HEALTH_OK 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:06.298 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:06.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:07.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:07.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:07.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:07 smithi193 ceph-mon[31815]: Deploying daemon mon.b on smithi190 2024-04-22T06:36:07.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:07 smithi193 ceph-mon[31815]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:07.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.298+0000 7f6eda0f6700 1 -- 172.21.15.193:0/3546800550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ebc0035c0 msgr2=0x7f6ebc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:07.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.298+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/3546800550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ebc0035c0 0x7f6ebc005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4102490 tx=0x7f6ec800feb0 comp rx=0 tx=0).stop 2024-04-22T06:36:07.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.298+0000 7f6eda0f6700 1 -- 172.21.15.193:0/3546800550 shutdown_connections 2024-04-22T06:36:07.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.298+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/3546800550 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ebc0035c0 0x7f6ebc005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:07.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.298+0000 7f6eda0f6700 1 -- 172.21.15.193:0/3546800550 >> 172.21.15.193:0/3546800550 conn(0x7f6ed40fe050 msgr2=0x7f6ed4100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:07.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.298+0000 7f6eda0f6700 1 -- 172.21.15.193:0/3546800550 shutdown_connections 2024-04-22T06:36:07.298 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.299+0000 7f6eda0f6700 1 -- 172.21.15.193:0/3546800550 wait complete. 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.299+0000 7f6eda0f6700 1 Processor -- start 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6eda0f6700 1 -- start start 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6eda0f6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6eda0f6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ed4075280 0x7f6ed4075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6eda0f6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed410a4b0 con 0x7f6ed4075280 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6eda0f6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed410a330 con 0x7f6ebc0035c0 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:51378/0 (socket says 172.21.15.193:51378) 2024-04-22T06:36:07.300 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 -- 172.21.15.193:0/70034844 learned_addr learned my addr 172.21.15.193:0/70034844 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed2ffd700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ed4075280 0x7f6ed4075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 -- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 msgr2=0x7f6ed4074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 -- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 msgr2=0x7f6ed4074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed37fe700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed2ffd700 1 -- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 msgr2=0x7f6ed4074d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.300+0000 7f6ed2ffd700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6ed2ffd700 1 -- 172.21.15.193:0/70034844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed4076bc0 con 0x7f6ed4075280 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6ed2ffd700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ed4075280 0x7f6ed4075700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b4d0 tx=0x7f6ec400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f6ec4011630 con 0x7f6ed4075280 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ec4011c80 con 0x7f6ed4075280 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed41a41f0 con 0x7f6ed4075280 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f6ec4010da0 con 0x7f6ed4075280 2024-04-22T06:36:07.301 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.301+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed41a4750 con 0x7f6ed4075280 2024-04-22T06:36:07.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.302+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6ec4018020 con 0x7f6ed4075280 2024-04-22T06:36:07.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.302+0000 7f6ed0ff9700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ec0039440 0x7f6ec003b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:07.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.302+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed4103ca0 con 0x7f6ed4075280 2024-04-22T06:36:07.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.302+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ec404c240 con 0x7f6ed4075280 2024-04-22T06:36:07.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.302+0000 7f6ed37fe700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ec0039440 0x7f6ec003b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:07.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.303+0000 7f6ed37fe700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ec0039440 0x7f6ec003b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800fa60 tx=0x7f6ec8005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:07.306 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.307+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ec401f020 con 0x7f6ed4075280 2024-04-22T06:36:07.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:07.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:07.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:07.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:07 smithi079 ceph-mon[27946]: Deploying daemon mon.b on smithi190 2024-04-22T06:36:07.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:07 smithi079 ceph-mon[27946]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:07.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.593+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6ed402df80 con 0x7f6ed4075280 2024-04-22T06:36:07.592 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.593+0000 7f6ed0ff9700 1 -- 172.21.15.193:0/70034844 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f6ec4018570 con 0x7f6ed4075280 2024-04-22T06:36:07.592 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:36:07.593 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","modified":"2024-04-22T06:36:00.930289Z","created":"2024-04-22T06:34:18.390130Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ec0039440 msgr2=0x7f6ec003b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ec0039440 0x7f6ec003b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800fa60 tx=0x7f6ec8005a90 comp rx=0 tx=0).stop 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ed4075280 msgr2=0x7f6ed4075700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ed4075280 0x7f6ed4075700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b4d0 tx=0x7f6ec400b890 comp rx=0 tx=0).stop 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 shutdown_connections 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ebc0035c0 0x7f6ed4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ec0039440 0x7f6ec003b900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 --2- 172.21.15.193:0/70034844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ed4075280 0x7f6ed4075700 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 >> 172.21.15.193:0/70034844 conn(0x7f6ed40fe050 msgr2=0x7f6ed40fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 shutdown_connections 2024-04-22T06:36:07.595 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:07.596+0000 7f6eda0f6700 1 -- 172.21.15.193:0/70034844 wait complete. 2024-04-22T06:36:07.596 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-04-22T06:36:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:08 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/70034844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:08 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/70034844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:09.212 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:36:09.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:36:09.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:09 smithi193 ceph-mon[31815]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:09.454 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:36:09.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:09 smithi079 ceph-mon[27946]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:09.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:09 smithi190 systemd[1]: Started Ceph mon.b for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:15.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:15.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: mon.c calling monitor election 2024-04-22T06:36:15.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: mon.a calling monitor election 2024-04-22T06:36:15.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:15.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: monmap epoch 3 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: last_changed 2024-04-22T06:36:09.841411+0000 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: min_mon_release 19 (squid) 2024-04-22T06:36:15.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: election_strategy: 1 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: fsmap 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: mgrmap e13: x(active, since 58s) 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: overall HEALTH_OK 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:15.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: mon.c calling monitor election 2024-04-22T06:36:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: mon.a calling monitor election 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: monmap epoch 3 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: last_changed 2024-04-22T06:36:09.841411+0000 2024-04-22T06:36:15.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: min_mon_release 19 (squid) 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: election_strategy: 1 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: fsmap 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: mgrmap e13: x(active, since 58s) 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: overall HEALTH_OK 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:15.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:15.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 -- 172.21.15.193:0/2384152788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 msgr2=0x7fa064005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2384152788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa064005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa074101360 tx=0x7fa06800fe60 comp rx=0 tx=0).stop 2024-04-22T06:36:15.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 -- 172.21.15.193:0/2384152788 shutdown_connections 2024-04-22T06:36:15.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2384152788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa064005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 -- 172.21.15.193:0/2384152788 >> 172.21.15.193:0/2384152788 conn(0x7fa0740fcf20 msgr2=0x7fa0740ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:15.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 -- 172.21.15.193:0/2384152788 shutdown_connections 2024-04-22T06:36:15.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.283+0000 7fa07b08a700 1 -- 172.21.15.193:0/2384152788 wait complete. 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 Processor -- start 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 -- start start 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa07419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 0x7fa07419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa074109340 con 0x7fa0640035c0 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa0741091c0 con 0x7fa07419a830 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa07b08a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0741094c0 con 0x7fa07419f100 2024-04-22T06:36:15.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa079627700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa079627700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:60272/0 (socket says 172.21.15.193:60272) 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.284+0000 7fa079627700 1 -- 172.21.15.193:0/2683687765 learned_addr learned my addr 172.21.15.193:0/2683687765 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa07419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa073fff700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 0x7fa07419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa079627700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 msgr2=0x7fa07419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa079627700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 msgr2=0x7fa07419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa079627700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa079627700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa073fff700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 msgr2=0x7fa07419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa073fff700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 msgr2=0x7fa07419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa073fff700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 0x7fa07419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa073fff700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 0x7fa07419ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:36:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 msgr2=0x7fa07419f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 msgr2=0x7fa07419ebc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 0x7fa07419ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 -- 172.21.15.193:0/2683687765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0741a6e70 con 0x7fa0640035c0 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa078e26700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa07419a2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa074101360 tx=0x7fa068003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06802b020 con 0x7fa0640035c0 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.285+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0741a7040 con 0x7fa0640035c0 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.286+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0741a75d0 con 0x7fa0640035c0 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.286+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa06801d440 con 0x7fa0640035c0 2024-04-22T06:36:15.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.286+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068022040 con 0x7fa0640035c0 2024-04-22T06:36:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.287+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0740640a0 con 0x7fa0640035c0 2024-04-22T06:36:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.287+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa06802e020 con 0x7fa0640035c0 2024-04-22T06:36:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.287+0000 7fa071ffb700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa058039ae0 0x7fa05803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.287+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa06804ebf0 con 0x7fa0640035c0 2024-04-22T06:36:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.287+0000 7fa073fff700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa058039ae0 0x7fa05803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.287+0000 7fa073fff700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa058039ae0 0x7fa05803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa060009850 tx=0x7fa060008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.292+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa068035030 con 0x7fa0640035c0 2024-04-22T06:36:15.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.586+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa0741a7880 con 0x7fa0640035c0 2024-04-22T06:36:15.585 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.586+0000 7fa071ffb700 1 -- 172.21.15.193:0/2683687765 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7fa06800e5b0 con 0x7fa0640035c0 2024-04-22T06:36:15.586 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-22T06:36:15.586 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","modified":"2024-04-22T06:36:09.841411Z","created":"2024-04-22T06:34:18.390130Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa058039ae0 msgr2=0x7fa05803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa058039ae0 0x7fa05803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa060009850 tx=0x7fa060008040 comp rx=0 tx=0).stop 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 msgr2=0x7fa07419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa07419a2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa074101360 tx=0x7fa068003cd0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 shutdown_connections 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa07419f100 0x7fa07419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa058039ae0 0x7fa05803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa0640035c0 0x7fa07419a2f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 --2- 172.21.15.193:0/2683687765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa07419a830 0x7fa07419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 >> 172.21.15.193:0/2683687765 conn(0x7fa0740fcf20 msgr2=0x7fa07406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 shutdown_connections 2024-04-22T06:36:15.589 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:15.590+0000 7fa07b08a700 1 -- 172.21.15.193:0/2683687765 wait complete. 2024-04-22T06:36:15.590 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-04-22T06:36:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:15 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/2683687765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:16.219 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T06:36:16.219 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T06:36:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:15 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/2683687765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:16.467 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: mon.c calling monitor election 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: mon.a calling monitor election 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:17.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: monmap epoch 3 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: last_changed 2024-04-22T06:36:09.841411+0000 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: min_mon_release 19 (squid) 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: election_strategy: 1 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: fsmap 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: mgrmap e13: x(active, since 58s) 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: overall HEALTH_OK 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:17.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:17.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:17.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:17.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:17.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/2683687765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:17.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: mon.b calling monitor election 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: mon.b calling monitor election 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: mon.c calling monitor election 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: mon.a calling monitor election 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: monmap epoch 3 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:18.116 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: last_changed 2024-04-22T06:36:09.841411+0000 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: min_mon_release 19 (squid) 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: election_strategy: 1 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: fsmap 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: mgrmap e13: x(active, since 60s) 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: overall HEALTH_OK 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.117 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:18.118 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:18.119 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:17 smithi079 ceph-mon[27946]: Reconfiguring daemon mon.a on smithi079 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: mon.b calling monitor election 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: mon.b calling monitor election 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: mon.c calling monitor election 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: mon.a calling monitor election 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:36:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: monmap epoch 3 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: last_changed 2024-04-22T06:36:09.841411+0000 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: min_mon_release 19 (squid) 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: election_strategy: 1 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: fsmap 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: mgrmap e13: x(active, since 60s) 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: overall HEALTH_OK 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:18.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:17 smithi193 ceph-mon[31815]: Reconfiguring daemon mon.a on smithi079 2024-04-22T06:36:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: mon.b calling monitor election 2024-04-22T06:36:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:36:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:36:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:36:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: mon.b calling monitor election 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: mon.c calling monitor election 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: mon.a calling monitor election 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: monmap epoch 3 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: last_changed 2024-04-22T06:36:09.841411+0000 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: created 2024-04-22T06:34:18.390130+0000 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: min_mon_release 19 (squid) 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: election_strategy: 1 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: 0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: fsmap 2024-04-22T06:36:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: mgrmap e13: x(active, since 60s) 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: overall HEALTH_OK 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:18.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:36:18.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:18.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:18.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:18.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:17 smithi190 ceph-mon[31421]: Reconfiguring daemon mon.a on smithi079 2024-04-22T06:36:18.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 -- 172.21.15.79:0/748413525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 msgr2=0x7fe83c104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 --2- 172.21.15.79:0/748413525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c104770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009910 tx=0x7fe82c009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 -- 172.21.15.79:0/748413525 shutdown_connections 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 --2- 172.21.15.79:0/748413525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c104770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 -- 172.21.15.79:0/748413525 >> 172.21.15.79:0/748413525 conn(0x7fe83c0fc5b0 msgr2=0x7fe83c0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 -- 172.21.15.79:0/748413525 shutdown_connections 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.216+0000 7fe8411c5700 1 -- 172.21.15.79:0/748413525 wait complete. 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.217+0000 7fe8411c5700 1 Processor -- start 2024-04-22T06:36:18.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.217+0000 7fe8411c5700 1 -- start start 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe83c1987b0 0x7fe83c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c19d080 0x7fe83c19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c10caa0 con 0x7fe83c104390 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c10c920 con 0x7fe83c19d080 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c10cc20 con 0x7fe83c1987b0 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42346/0 (socket says 172.21.15.79:42346) 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 -- 172.21.15.79:0/2537326910 learned_addr learned my addr 172.21.15.79:0/2537326910 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83a59c700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe83c1987b0 0x7fe83c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 -- 172.21.15.79:0/2537326910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe83c1987b0 msgr2=0x7fe83c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe83c1987b0 0x7fe83c19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 -- 172.21.15.79:0/2537326910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c19d080 msgr2=0x7fe83c19d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:18.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c19d080 0x7fe83c19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 -- 172.21.15.79:0/2537326910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c1a4df0 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83a59c700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe83c1987b0 0x7fe83c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe83ad9d700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c198270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005b40 tx=0x7fe82c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c026070 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83c1a4fc0 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe82c00b750 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83c1a5490 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.218+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0038e0 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.219+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe82c029070 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.219+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83c0640a0 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.219+0000 7fe823fff700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe824039ae0 0x7fe82403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.219+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe82c04d7c0 con 0x7fe83c104390 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.220+0000 7fe83a59c700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe824039ae0 0x7fe82403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:18.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.220+0000 7fe83a59c700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe824039ae0 0x7fe82403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe8300079b0 tx=0x7fe830008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:18.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.225+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe82c013070 con 0x7fe83c104390 2024-04-22T06:36:18.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.434+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe83c1a5750 con 0x7fe83c104390 2024-04-22T06:36:18.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.435+0000 7fe823fff700 1 -- 172.21.15.79:0/2537326910 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fe82c030090 con 0x7fe83c104390 2024-04-22T06:36:18.434 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:18.434 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-04-22T06:36:18.434 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:18.434 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-04-22T06:36:18.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe824039ae0 msgr2=0x7fe82403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:18.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe824039ae0 0x7fe82403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe8300079b0 tx=0x7fe830008040 comp rx=0 tx=0).stop 2024-04-22T06:36:18.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 msgr2=0x7fe83c198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:18.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c198270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005b40 tx=0x7fe82c005030 comp rx=0 tx=0).stop 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 shutdown_connections 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe83c19d080 0x7fe83c19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe824039ae0 0x7fe82403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe83c104390 0x7fe83c198270 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 --2- 172.21.15.79:0/2537326910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe83c1987b0 0x7fe83c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.437+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 >> 172.21.15.79:0/2537326910 conn(0x7fe83c0fc5b0 msgr2=0x7fe83c105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.438+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 shutdown_connections 2024-04-22T06:36:18.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:18.438+0000 7fe8411c5700 1 -- 172.21.15.79:0/2537326910 wait complete. 2024-04-22T06:36:18.938 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:18.938 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2537326910' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:18.938 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:18 smithi079 ceph-mon[27946]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2537326910' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:18 smithi193 ceph-mon[31815]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2537326910' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:18 smithi190 ceph-mon[31421]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:19.859 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T06:36:19.859 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:36:19.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:36:19.925 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:36:19.925 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:19.988 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:36:19.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:36:20.014 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:36:20.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:20.080 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:36:20.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:36:20.111 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:36:20.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:20.177 INFO:tasks.cephadm:Adding mgr.x on smithi079 2024-04-22T06:36:20.178 INFO:tasks.cephadm:Adding mgr.y on smithi190 2024-04-22T06:36:20.178 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-04-22T06:36:20.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi079=x;smithi190=y;smithi193=z' 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:20.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:20 smithi193 ceph-mon[31815]: Reconfiguring daemon mon.b on smithi190 2024-04-22T06:36:20.452 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:20.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:20 smithi190 ceph-mon[31421]: Reconfiguring daemon mon.b on smithi190 2024-04-22T06:36:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:36:20.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:20.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:20.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:20.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:20 smithi079 ceph-mon[27946]: Reconfiguring daemon mon.b on smithi190 2024-04-22T06:36:21.590 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:21 smithi193 ceph-mon[31815]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:21.590 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:21.590 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:21.590 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:21.590 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:21.590 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:21 smithi190 ceph-mon[31421]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:21.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:21 smithi079 ceph-mon[27946]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:21.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:21.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:21.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:21.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:21.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 -- 172.21.15.193:0/2310310491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128103050 msgr2=0x7f0128105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 --2- 172.21.15.193:0/2310310491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128103050 0x7f0128105440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f012400b0a0 tx=0x7f012400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 -- 172.21.15.193:0/2310310491 shutdown_connections 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 --2- 172.21.15.193:0/2310310491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128103050 0x7f0128105440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 -- 172.21.15.193:0/2310310491 >> 172.21.15.193:0/2310310491 conn(0x7f01280fca80 msgr2=0x7f01280feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 -- 172.21.15.193:0/2310310491 shutdown_connections 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.989+0000 7f012f15e700 1 -- 172.21.15.193:0/2310310491 wait complete. 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 Processor -- start 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 -- start start 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 0x7f0128197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01281981b0 0x7f012819d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128198630 0x7f012819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0128108450 con 0x7f0128198630 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f01281082d0 con 0x7f01281981b0 2024-04-22T06:36:21.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012f15e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01281085d0 con 0x7f0128103050 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012cefa700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 0x7f0128197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012cefa700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 0x7f0128197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:60278/0 (socket says 172.21.15.193:60278) 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012cefa700 1 -- 172.21.15.193:0/268546224 learned_addr learned my addr 172.21.15.193:0/268546224 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f012d6fb700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128198630 0x7f012819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.990+0000 7f011ffff700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01281981b0 0x7f012819d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012cefa700 1 -- 172.21.15.193:0/268546224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01281981b0 msgr2=0x7f012819d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012cefa700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01281981b0 0x7f012819d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012cefa700 1 -- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128198630 msgr2=0x7f012819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012cefa700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128198630 0x7f012819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012cefa700 1 -- 172.21.15.193:0/268546224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01281a0090 con 0x7f0128103050 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f011ffff700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01281981b0 0x7f012819d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012cefa700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 0x7f0128197c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f012400b830 tx=0x7f012400bd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124017040 con 0x7f0128103050 2024-04-22T06:36:21.992 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0124005780 con 0x7f0128103050 2024-04-22T06:36:21.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01281a0320 con 0x7f0128103050 2024-04-22T06:36:21.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124012a00 con 0x7f0128103050 2024-04-22T06:36:21.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.991+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01281a08e0 con 0x7f0128103050 2024-04-22T06:36:21.993 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.992+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0124012ba0 con 0x7f0128103050 2024-04-22T06:36:21.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.992+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01280640a0 con 0x7f0128103050 2024-04-22T06:36:21.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.992+0000 7f011dffb700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0110039ae0 0x7f011003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:21.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.992+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f012400f020 con 0x7f0128103050 2024-04-22T06:36:21.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.992+0000 7f011ffff700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0110039ae0 0x7f011003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:21.994 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.993+0000 7f011ffff700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0110039ae0 0x7f011003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f01180079b0 tx=0x7f0118008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:21.996 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:21.998+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0124029020 con 0x7f0128103050 2024-04-22T06:36:22.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.213+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=x;smithi190=y;smithi193=z", "target": ["mon-mgr", ""]}) -- 0x7f0128062e80 con 0x7f0110039ae0 2024-04-22T06:36:22.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.220+0000 7f011dffb700 1 -- 172.21.15.193:0/268546224 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0128062e80 con 0x7f0110039ae0 2024-04-22T06:36:22.218 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0110039ae0 msgr2=0x7f011003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0110039ae0 0x7f011003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f01180079b0 tx=0x7f0118008040 comp rx=0 tx=0).stop 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 msgr2=0x7f0128197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 0x7f0128197c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f012400b830 tx=0x7f012400bd10 comp rx=0 tx=0).stop 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 shutdown_connections 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0110039ae0 0x7f011003bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0128198630 0x7f012819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0128103050 0x7f0128197c70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 --2- 172.21.15.193:0/268546224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01281981b0 0x7f012819d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 >> 172.21.15.193:0/268546224 conn(0x7f01280fca80 msgr2=0x7f01280fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:22.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 shutdown_connections 2024-04-22T06:36:22.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:36:22.222+0000 7f012f15e700 1 -- 172.21.15.193:0/268546224 wait complete. 2024-04-22T06:36:22.661 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:22 smithi193 ceph-mon[31815]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:36:22.661 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:22 smithi193 ceph-mon[31815]: Reconfiguring daemon mon.c on smithi193 2024-04-22T06:36:22.661 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:22 smithi190 ceph-mon[31421]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:36:22.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:22 smithi190 ceph-mon[31421]: Reconfiguring daemon mon.c on smithi193 2024-04-22T06:36:22.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:22.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:22 smithi079 ceph-mon[27946]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:36:22.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:22 smithi079 ceph-mon[27946]: Reconfiguring daemon mon.c on smithi193 2024-04-22T06:36:22.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:23 smithi193 ceph-mon[31815]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=x;smithi190=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:23.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:23 smithi193 ceph-mon[31815]: Saving service mgr spec with placement smithi079=x;smithi190=y;smithi193=z;count:3 2024-04-22T06:36:23.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:23 smithi193 ceph-mon[31815]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:23.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.549 DEBUG:teuthology.orchestra.run.smithi190:mgr.y> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T06:36:23.552 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.z.service 2024-04-22T06:36:23.555 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T06:36:23.555 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:36:23.555 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:36:23.570 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:36:23.570 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-04-22T06:36:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:23 smithi190 ceph-mon[31421]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=x;smithi190=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:23 smithi190 ceph-mon[31421]: Saving service mgr spec with placement smithi079=x;smithi190=y;smithi193=z;count:3 2024-04-22T06:36:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:23 smithi190 ceph-mon[31421]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-22 06:35:26.821963881 +0000 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-22 06:29:18.394564610 +0000 2024-04-22T06:36:23.625 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-22 06:29:18.394564610 +0000 2024-04-22T06:36:23.626 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-22 06:29:18.394564610 +0000 2024-04-22T06:36:23.626 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:36:23.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:23 smithi079 ceph-mon[27946]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi079=x;smithi190=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:23.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:23 smithi079 ceph-mon[27946]: Saving service mgr spec with placement smithi079=x;smithi190=y;smithi193=z;count:3 2024-04-22T06:36:23.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:23 smithi079 ceph-mon[27946]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:23.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:23.687 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-22T06:36:23.687 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-22T06:36:23.687 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000193453 s, 2.6 MB/s 2024-04-22T06:36:23.688 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:36:23.744 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-22 06:35:26.846963309 +0000 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-22 06:29:18.748556502 +0000 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-22 06:29:18.748556502 +0000 2024-04-22T06:36:23.799 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-22 06:29:18.748556502 +0000 2024-04-22T06:36:23.799 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:36:23.862 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-22T06:36:23.862 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-22T06:36:23.862 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000218361 s, 2.3 MB/s 2024-04-22T06:36:23.863 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:36:23.919 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-22 06:35:26.871962738 +0000 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-22 06:29:19.082548852 +0000 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-22 06:29:19.082548852 +0000 2024-04-22T06:36:23.974 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-22 06:29:19.082548852 +0000 2024-04-22T06:36:23.974 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:36:24.036 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-22T06:36:24.036 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-22T06:36:24.036 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000174655 s, 2.9 MB/s 2024-04-22T06:36:24.037 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:36:24.093 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-22 06:35:26.896962166 +0000 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-22 06:29:19.435540766 +0000 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-22 06:29:19.435540766 +0000 2024-04-22T06:36:24.148 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-22 06:29:19.435540766 +0000 2024-04-22T06:36:24.149 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:36:24.210 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-04-22T06:36:24.210 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-04-22T06:36:24.210 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000197871 s, 2.6 MB/s 2024-04-22T06:36:24.211 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:36:24.266 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-22T06:36:24.266 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:36:24.285 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:36:24.285 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-04-22T06:36:24.343 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:36:24.343 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-22 06:36:09.318266097 +0000 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-22 06:29:39.068476517 +0000 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-22 06:29:39.068476517 +0000 2024-04-22T06:36:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-22 06:29:39.068476517 +0000 2024-04-22T06:36:24.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:36:24.409 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-22T06:36:24.409 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-22T06:36:24.409 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000286616 s, 1.8 MB/s 2024-04-22T06:36:24.410 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:36:24.468 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-22 06:36:09.319266070 +0000 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-22 06:29:39.407467098 +0000 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-22 06:29:39.407467098 +0000 2024-04-22T06:36:24.524 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-22 06:29:39.407467098 +0000 2024-04-22T06:36:24.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:36:24.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:24.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:24.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:24.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:24.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:24.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:24.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.588 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-22T06:36:24.588 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-22T06:36:24.588 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000196179 s, 2.6 MB/s 2024-04-22T06:36:24.589 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:24.621 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.645 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-04-22T06:36:24.701 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:36:24.701 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-22 06:36:09.319266070 +0000 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-22 06:29:39.749457595 +0000 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-22 06:29:39.749457595 +0000 2024-04-22T06:36:24.702 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-22 06:29:39.749457595 +0000 2024-04-22T06:36:24.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:36:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:24.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:24.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.764 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-22T06:36:24.764 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-22T06:36:24.765 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000185366 s, 2.8 MB/s 2024-04-22T06:36:24.766 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:36:24.822 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-04-22T06:36:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-22 06:36:09.319266070 +0000 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-22 06:29:40.081448370 +0000 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-22 06:29:40.081448370 +0000 2024-04-22T06:36:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-22 06:29:40.081448370 +0000 2024-04-22T06:36:24.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:36:24.940 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-22T06:36:24.940 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-22T06:36:24.940 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000261125 s, 2.0 MB/s 2024-04-22T06:36:24.941 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:36:24.997 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-22T06:36:24.997 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:36:25.013 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:36:25.013 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-04-22T06:36:25.068 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-22 06:35:59.844109989 +0000 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-22 06:29:48.125072001 +0000 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-22 06:29:48.125072001 +0000 2024-04-22T06:36:25.069 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-22 06:29:48.125072001 +0000 2024-04-22T06:36:25.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:36:25.132 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-22T06:36:25.132 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-22T06:36:25.132 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000320688 s, 1.6 MB/s 2024-04-22T06:36:25.133 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:36:25.189 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-04-22T06:36:25.244 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:36:25.244 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:25.244 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T06:36:25.244 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:25.244 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:25.244 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-22 06:35:59.844109989 +0000 2024-04-22T06:36:25.245 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-22 06:29:48.456066731 +0000 2024-04-22T06:36:25.245 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-22 06:29:48.456066731 +0000 2024-04-22T06:36:25.245 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-22 06:29:48.456066731 +0000 2024-04-22T06:36:25.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:36:25.306 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-22T06:36:25.306 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-22T06:36:25.306 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000185018 s, 2.8 MB/s 2024-04-22T06:36:25.307 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:36:25.362 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-22 06:35:59.845109974 +0000 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-22 06:29:48.786061478 +0000 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-22 06:29:48.786061478 +0000 2024-04-22T06:36:25.417 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-22 06:29:48.786061478 +0000 2024-04-22T06:36:25.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:36:25.479 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-22T06:36:25.480 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-22T06:36:25.480 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00022653 s, 2.3 MB/s 2024-04-22T06:36:25.481 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:36:25.537 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-04-22T06:36:25.585 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:25 smithi193 ceph-mon[31815]: Deploying daemon mgr.z on smithi193 2024-04-22T06:36:25.585 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:25 smithi193 ceph-mon[31815]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-22 06:35:59.845109974 +0000 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-22 06:29:49.110056320 +0000 2024-04-22T06:36:25.592 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-22 06:29:49.110056320 +0000 2024-04-22T06:36:25.593 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-22 06:29:49.110056320 +0000 2024-04-22T06:36:25.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:36:25.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:25 smithi190 ceph-mon[31421]: Deploying daemon mgr.z on smithi193 2024-04-22T06:36:25.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:25 smithi190 ceph-mon[31421]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:25.730 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-22T06:36:25.730 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-22T06:36:25.730 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000231126 s, 2.2 MB/s 2024-04-22T06:36:25.731 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:36:25.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:25 smithi079 ceph-mon[27946]: Deploying daemon mgr.z on smithi193 2024-04-22T06:36:25.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:25 smithi079 ceph-mon[27946]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:25.746 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/vg_nvme/lv_4... 2024-04-22T06:36:25.747 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:36:25.924 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:25 smithi193 systemd[1]: Starting Ceph mgr.z for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:36:25.988 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:26.190 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 podman[33667]: 2024-04-22 06:36:26.133841119 +0000 UTC m=+0.106715002 container create e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-22T06:36:26.190 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 podman[33667]: 2024-04-22 06:36:26.04476245 +0000 UTC m=+0.017636334 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:36:26.499 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 podman[33667]: 2024-04-22 06:36:26.344588488 +0000 UTC m=+0.317462367 container init e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T06:36:26.499 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 podman[33667]: 2024-04-22 06:36:26.347108222 +0000 UTC m=+0.319982110 container start e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:36:26.499 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 ceph-mgr[33686]: -- 172.21.15.193:0/62292027 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5652e6d0e5a0 con 0x5652e6ce3000 2024-04-22T06:36:26.499 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 bash[33667]: e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e 2024-04-22T06:36:26.499 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 systemd[1]: Started Ceph mgr.z for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:26.784 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:26.500+0000 7f92fa0ed200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:36:26.784 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:26.684+0000 7f92fa0ed200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:26.784 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:26.785+0000 7f92fa0ed200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:36:27.439 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:27.091+0000 7f92fa0ed200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:36:27.439 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:27.175+0000 7f92fa0ed200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:36:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:27.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:27.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: Deploying daemon mgr.y on smithi190 2024-04-22T06:36:27.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:27 smithi190 ceph-mon[31421]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: Deploying daemon mgr.y on smithi190 2024-04-22T06:36:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:27 smithi079 ceph-mon[27946]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:27.767 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.767 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.767 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:27.768 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:27.768 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:27.768 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:27.768 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:27.768 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: Deploying daemon mgr.y on smithi190 2024-04-22T06:36:27.768 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:27 smithi193 ceph-mon[31815]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:36:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:36:27.912 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-04-22T06:36:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-04-22T06:36:27.912 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112208 s, 934 MB/s 2024-04-22T06:36:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-04-22T06:36:28.515 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-04-22T06:36:28.574 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 systemd[1]: Starting Ceph mgr.y for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:36:28.574 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 podman[32786]: 2024-04-22 06:36:28.575659522 +0000 UTC m=+0.096352991 container create e4afe626c571633f8a69e222401385de5f602100183bb179a1cacadbe63cec29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:36:28.689 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:28.410+0000 7f92fa0ed200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:36:28.689 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:28.586+0000 7f92fa0ed200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:28.748 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:28.842 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 podman[32786]: 2024-04-22 06:36:28.496787368 +0000 UTC m=+0.017480808 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:36:28.842 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 podman[32786]: 2024-04-22 06:36:28.769751758 +0000 UTC m=+0.290445184 container init e4afe626c571633f8a69e222401385de5f602100183bb179a1cacadbe63cec29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:36:28.843 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 podman[32786]: 2024-04-22 06:36:28.772282406 +0000 UTC m=+0.292975835 container start e4afe626c571633f8a69e222401385de5f602100183bb179a1cacadbe63cec29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-22T06:36:28.843 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 ceph-mgr[32805]: -- 172.21.15.190:0/598512776 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561435ba65a0 con 0x561435b7b000 2024-04-22T06:36:29.061 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:28.773+0000 7f92fa0ed200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:36:29.115 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 bash[32786]: e4afe626c571633f8a69e222401385de5f602100183bb179a1cacadbe63cec29 2024-04-22T06:36:29.115 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 systemd[1]: Started Ceph mgr.y for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:29.115 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:28.924+0000 7f5a91bfe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:36:29.115 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:29.115+0000 7f5a91bfe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:29.371 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:29.062+0000 7f92fa0ed200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:36:29.371 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:29.159+0000 7f92fa0ed200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:36:29.371 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:29.372+0000 7f92fa0ed200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:36:29.453 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:29.226+0000 7f5a91bfe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:36:29.627 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:29.579+0000 7f92fa0ed200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:29.777 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:29 smithi079 ceph-mon[27946]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:29.777 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.777 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.777 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.777 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.777 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:29.864 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:29.566+0000 7f5a91bfe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:36:29.864 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:29.662+0000 7f5a91bfe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:36:29.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-mon[31421]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:29.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:29.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-mon[31815]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:30.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 -- 172.21.15.79:0/3284883561 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 msgr2=0x7f8df8102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/3284883561 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df8102ec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8de00099c0 tx=0x7f8de0009cd0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 -- 172.21.15.79:0/3284883561 shutdown_connections 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/3284883561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8df8104640 0x7f8df810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/3284883561 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df8102ec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/3284883561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8103ca0 0x7f8df8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 -- 172.21.15.79:0/3284883561 >> 172.21.15.79:0/3284883561 conn(0x7f8df80fe040 msgr2=0x7f8df8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 -- 172.21.15.79:0/3284883561 shutdown_connections 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.286+0000 7f8dfcac9700 1 -- 172.21.15.79:0/3284883561 wait complete. 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 Processor -- start 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 -- start start 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df80767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8df8103ca0 0x7f8df8074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8104640 0x7f8df8075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810e5b0 con 0x7f8df8102aa0 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810e430 con 0x7f8df8103ca0 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8dfcac9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810e730 con 0x7f8df8104640 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8df659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df80767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.287+0000 7f8df659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df80767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57404/0 (socket says 172.21.15.79:57404) 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 -- 172.21.15.79:0/2466370477 learned_addr learned my addr 172.21.15.79:0/2466370477 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df5d9b700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8df8103ca0 0x7f8df8074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df6d9d700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8104640 0x7f8df8075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 -- 172.21.15.79:0/2466370477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8104640 msgr2=0x7f8df8075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8104640 0x7f8df8075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 -- 172.21.15.79:0/2466370477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8df8103ca0 msgr2=0x7f8df8074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8df8103ca0 0x7f8df8074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 -- 172.21.15.79:0/2466370477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df8075c20 con 0x7f8df8102aa0 2024-04-22T06:36:30.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8df659c700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df80767e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8de00038c0 tx=0x7f8de00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:30.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0027070 con 0x7f8df8102aa0 2024-04-22T06:36:30.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8de00040e0 con 0x7f8df8102aa0 2024-04-22T06:36:30.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df81a2db0 con 0x7f8df8102aa0 2024-04-22T06:36:30.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0022df0 con 0x7f8df8102aa0 2024-04-22T06:36:30.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.288+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df81a31d0 con 0x7f8df8102aa0 2024-04-22T06:36:30.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.289+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8de002a070 con 0x7f8df8102aa0 2024-04-22T06:36:30.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.289+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df8069730 con 0x7f8df8102aa0 2024-04-22T06:36:30.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.289+0000 7f8def7fe700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8de4039890 0x7f8de403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.289+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8de004d7a0 con 0x7f8df8102aa0 2024-04-22T06:36:30.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.289+0000 7f8df5d9b700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8de4039890 0x7f8de403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.290+0000 7f8df5d9b700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8de4039890 0x7f8de403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8de8004630 tx=0x7f8de800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:30.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.294+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8de0013070 con 0x7f8df8102aa0 2024-04-22T06:36:30.439 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:30.215+0000 7f92fa0ed200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:36:30.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:30.509+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8df8062e80 con 0x7f8de4039890 2024-04-22T06:36:30.938 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:30.647+0000 7f92fa0ed200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:36:30.938 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:30.741+0000 7f92fa0ed200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:36:30.938 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:30.939+0000 7f92fa0ed200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:36:31.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:30.898+0000 7f5a91bfe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:36:31.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:31.066+0000 7f5a91bfe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:31.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.401 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:31 smithi079 ceph-mon[27946]: Reconfiguring daemon mgr.x on smithi079 2024-04-22T06:36:31.522 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:31.235+0000 7f5a91bfe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:31.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:36:31.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:31.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:31.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-mon[31421]: Reconfiguring daemon mgr.x on smithi079 2024-04-22T06:36:31.689 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:31.459+0000 7f92fa0ed200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:31.550+0000 7f92fa0ed200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:31.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-mon[31815]: Reconfiguring daemon mgr.x on smithi079 2024-04-22T06:36:31.817 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:31.523+0000 7f5a91bfe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:36:31.817 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:31.615+0000 7f5a91bfe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:36:31.817 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:31.817+0000 7f5a91bfe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:36:32.126 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:31.788+0000 7f92fa0ed200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:36:32.126 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:31.919+0000 7f92fa0ed200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:36:32.126 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:32.013+0000 7f92fa0ed200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:36:32.126 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:32.127+0000 7f92fa0ed200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:32.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:32.009+0000 7f5a91bfe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:32.690 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:32.428+0000 7f92fa0ed200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:36:32.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:32.953 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:32.600+0000 7f5a91bfe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:36:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:33.313 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:33.020+0000 7f5a91bfe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:36:33.313 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:33.115+0000 7f5a91bfe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:36:33.313 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:33.314+0000 7f5a91bfe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:36:33.439 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:33.022+0000 7f92fa0ed200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:36:33.439 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:36:33.237+0000 7f92fa0ed200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:36:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2718626773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6"}]: dispatch 2024-04-22T06:36:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2718626773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6"}]': finished 2024-04-22T06:36:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:36:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:33.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: Standby manager daemon z started 2024-04-22T06:36:33.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:36:33.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:33.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:36:33.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:33 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:33.846+0000 7f5a91bfe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2718626773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6"}]: dispatch 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2718626773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6"}]': finished 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: Standby manager daemon z started 2024-04-22T06:36:33.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:36:33.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:33.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:36:33.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:33.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:33.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2718626773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6"}]: dispatch 2024-04-22T06:36:33.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2718626773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6"}]': finished 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: Standby manager daemon z started 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:36:33.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:33 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/662815516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:34.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:34.174+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f8de000f7e0 con 0x7f8df8102aa0 2024-04-22T06:36:34.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:33.936+0000 7f5a91bfe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:36:34.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:34.176+0000 7f5a91bfe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:36:34.627 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:34.308+0000 7f5a91bfe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:36:34.628 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:34.401+0000 7f5a91bfe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:36:34.628 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:34.515+0000 7f5a91bfe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:34.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/342508326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:34.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:34 smithi193 ceph-mon[31815]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-22T06:36:34.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:34 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:36:34.953 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:34.817+0000 7f5a91bfe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:36:34.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/342508326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:34.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-mon[31421]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-22T06:36:34.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:34 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:36:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/342508326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:34 smithi079 ceph-mon[27946]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-22T06:36:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:34 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:36:35.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:35 smithi190 ceph-mon[31421]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:35.703 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:35.413+0000 7f5a91bfe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:36:35.703 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:36:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:36:35.627+0000 7f5a91bfe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:36:35.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:35 smithi193 ceph-mon[31815]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:35.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:35 smithi079 ceph-mon[27946]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:36.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:36.181+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f8de00163a0 con 0x7f8df8102aa0 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: Standby manager daemon y started 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-22T06:36:36.768 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:36 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:36:36.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: Standby manager daemon y started 2024-04-22T06:36:36.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:36:36.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:36.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:36:36.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:36.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-22T06:36:36.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:36 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: Standby manager daemon y started 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/61858337' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-22T06:36:36.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:36 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:36:37.897 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:37 smithi079 ceph-mon[27946]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:37.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:37 smithi193 ceph-mon[31815]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:37 smithi190 ceph-mon[31421]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:39.581 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:39.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:39.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:39.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:39 smithi079 ceph-mon[27946]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:39.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:39.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:39.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:39.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:39 smithi190 ceph-mon[31421]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:39.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:39.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:39.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:39.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:39.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:39 smithi193 ceph-mon[31815]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:41.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:41 smithi079 ceph-mon[27946]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:41.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:36:41.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:41.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:41 smithi193 ceph-mon[31815]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:36:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:41 smithi190 ceph-mon[31421]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:36:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:42.737 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:42 smithi079 ceph-mon[27946]: Deploying daemon osd.0 on smithi079 2024-04-22T06:36:42.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:42 smithi193 ceph-mon[31815]: Deploying daemon osd.0 on smithi079 2024-04-22T06:36:42.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:42 smithi190 ceph-mon[31421]: Deploying daemon osd.0 on smithi079 2024-04-22T06:36:43.792 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:43 smithi079 ceph-mon[27946]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:43 smithi193 ceph-mon[31815]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:43 smithi190 ceph-mon[31421]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:45.718 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:45 smithi079 ceph-mon[27946]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:45.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:45 smithi193 ceph-mon[31815]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:45.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:45 smithi190 ceph-mon[31421]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:47.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:47.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:47.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:47.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:47 smithi079 ceph-mon[27946]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:47.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:47.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:47.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:47.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:47 smithi193 ceph-mon[31815]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:47.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:47.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:47.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:47.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:47 smithi190 ceph-mon[31421]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:49.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:49 smithi079 ceph-mon[27946]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:49.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.768+0000 7f8def7fe700 1 -- 172.21.15.79:0/2466370477 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8df8062e80 con 0x7f8de4039890 2024-04-22T06:36:49.768 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-04-22T06:36:49.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8de4039890 msgr2=0x7f8de403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:49.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8de4039890 0x7f8de403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8de8004630 tx=0x7f8de800b3c0 comp rx=0 tx=0).stop 2024-04-22T06:36:49.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 msgr2=0x7f8df80767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:49.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df80767e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8de00038c0 tx=0x7f8de00047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:49.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 shutdown_connections 2024-04-22T06:36:49.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8df8104640 0x7f8df8075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:49.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8de4039890 0x7f8de403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:49.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8df8102aa0 0x7f8df80767e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:49.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 --2- 172.21.15.79:0/2466370477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8df8103ca0 0x7f8df8074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:49.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 >> 172.21.15.79:0/2466370477 conn(0x7f8df80fe040 msgr2=0x7f8df81070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:49.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 shutdown_connections 2024-04-22T06:36:49.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:36:49.770+0000 7f8dfcac9700 1 -- 172.21.15.79:0/2466370477 wait complete. 2024-04-22T06:36:49.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:49 smithi193 ceph-mon[31815]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:49 smithi190 ceph-mon[31421]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:50.383 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T06:36:50.385 INFO:tasks.cephadm:Deploying osd.1 on smithi190 with /dev/vg_nvme/lv_4... 2024-04-22T06:36:50.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:36:50.629 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:50 smithi079 ceph-mon[27946]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:50 smithi190 ceph-mon[31421]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:36:51.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:50 smithi193 ceph-mon[31815]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:36:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:51.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:51 smithi190 ceph-mon[31421]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:51.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:51 smithi190 ceph-mon[31421]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:36:51.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:51 smithi190 ceph-mon[31421]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:36:51.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:51 smithi190 ceph-mon[31421]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:36:51.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:51 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:51 smithi079 ceph-mon[27946]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:51 smithi079 ceph-mon[27946]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:51 smithi079 ceph-mon[27946]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:51 smithi079 ceph-mon[27946]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:51 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:51 smithi193 ceph-mon[31815]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:51 smithi193 ceph-mon[31815]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:36:52.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:51 smithi193 ceph-mon[31815]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:36:52.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:51 smithi193 ceph-mon[31815]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:36:52.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:51 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:52.345 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0[40118]: 2024-04-22T06:36:52.162+0000 7fd7ee902700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T06:36:52.346 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0[40118]: 2024-04-22T06:36:52.165+0000 7fd7e5eec700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:36:52.440 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:36:52.440 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:36:52.440 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-22T06:36:52.440 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-22T06:36:52.440 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126785 s, 827 MB/s 2024-04-22T06:36:52.440 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-22T06:36:53.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:52 smithi190 ceph-mon[31421]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:36:53.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:52 smithi190 ceph-mon[31421]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:36:53.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:52 smithi190 ceph-mon[31421]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' 2024-04-22T06:36:53.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-04-22T06:36:53.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-mon[27946]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:36:53.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-mon[27946]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:36:53.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:52 smithi079 ceph-mon[27946]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' 2024-04-22T06:36:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:52 smithi193 ceph-mon[31815]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:36:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:52 smithi193 ceph-mon[31815]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:36:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:52 smithi193 ceph-mon[31815]: from='osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155]' entity='osd.0' 2024-04-22T06:36:53.315 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:36:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:36:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:36:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] boot 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:53.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:36:54.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:36:54.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:54.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:54.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] boot 2024-04-22T06:36:54.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:54.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: osd.0 [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] boot 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:54.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:54.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3107643056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4104660 msgr2=0x7f15f410af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3107643056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4104660 0x7f15f410af10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f15e40099c0 tx=0x7f15e4009cd0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3107643056 shutdown_connections 2024-04-22T06:36:54.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3107643056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4104660 0x7f15f410af10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3107643056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 0x7f15f4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3107643056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4103cc0 0x7f15f4104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3107643056 >> 172.21.15.190:0/3107643056 conn(0x7f15f40fe060 msgr2=0x7f15f41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3107643056 shutdown_connections 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.910+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3107643056 wait complete. 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 Processor -- start 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 -- start start 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 0x7f15f40767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 0x7f15f4074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4104660 0x7f15f4075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f15f410e540 con 0x7f15f4104660 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f15f410e3c0 con 0x7f15f4103cc0 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fc7a2700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15f410e6c0 con 0x7f15f4102ac0 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15f9d3d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 0x7f15f4074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15f9d3d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 0x7f15f4074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:53188/0 (socket says 172.21.15.190:53188) 2024-04-22T06:36:54.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15f9d3d700 1 -- 172.21.15.190:0/3038741426 learned_addr learned my addr 172.21.15.190:0/3038741426 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.911+0000 7f15fa53e700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 0x7f15f40767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15fad3f700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4104660 0x7f15f4075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f9d3d700 1 -- 172.21.15.190:0/3038741426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 msgr2=0x7f15f40767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f9d3d700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 0x7f15f40767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f9d3d700 1 -- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4104660 msgr2=0x7f15f4075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f9d3d700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4104660 0x7f15f4075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f9d3d700 1 -- 172.21.15.190:0/3038741426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f4075c20 con 0x7f15f4103cc0 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15fad3f700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4104660 0x7f15f4075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15fa53e700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 0x7f15f40767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f9d3d700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 0x7f15f4074e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15e800d950 tx=0x7f15e800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8016070 con 0x7f15f4103cc0 2024-04-22T06:36:54.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f41a2d10 con 0x7f15f4103cc0 2024-04-22T06:36:54.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15e8009df0 con 0x7f15f4103cc0 2024-04-22T06:36:54.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e800f3c0 con 0x7f15f4103cc0 2024-04-22T06:36:54.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.912+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f41a31c0 con 0x7f15f4103cc0 2024-04-22T06:36:54.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.913+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f4069730 con 0x7f15f4103cc0 2024-04-22T06:36:54.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.914+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f15e8019070 con 0x7f15f4103cc0 2024-04-22T06:36:54.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.914+0000 7f15f37fe700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f15dc0a3bb0 0x7f15dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.915+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f15e80ce4a0 con 0x7f15f4103cc0 2024-04-22T06:36:54.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.915+0000 7f15fa53e700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f15dc0a3bb0 0x7f15dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.915+0000 7f15fa53e700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f15dc0a3bb0 0x7f15dc0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f15f4103b20 tx=0x7f15ec0091c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:54.917 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:54.919+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15e809c8f0 con 0x7f15f4103cc0 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi079 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi079 to 17088M 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:55.122 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:36:55.124+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f15f4062e80 con 0x7f15dc0a3bb0 2024-04-22T06:36:55.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi079 2024-04-22T06:36:55.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi079 to 17088M 2024-04-22T06:36:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:36:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:55.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi079 2024-04-22T06:36:55.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi079 to 17088M 2024-04-22T06:36:55.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:55.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:36:55.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:55.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:55.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:36:55.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:55 smithi190 ceph-mon[31421]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:55.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:55.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:55.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:56.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:55 smithi193 ceph-mon[31815]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:56.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:56.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:56.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:55 smithi079 ceph-mon[27946]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:56.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:56 smithi190 ceph-mon[31421]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:56 smithi193 ceph-mon[31815]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:57.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:56 smithi079 ceph-mon[27946]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:58.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:57 smithi190 ceph-mon[31421]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:58.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:57 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/2870145498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]: dispatch 2024-04-22T06:36:58.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:57 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]: dispatch 2024-04-22T06:36:58.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:57 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]': finished 2024-04-22T06:36:58.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:57 smithi190 ceph-mon[31421]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:36:58.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:57 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:36:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:57 smithi193 ceph-mon[31815]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:57 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/2870145498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]: dispatch 2024-04-22T06:36:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:57 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]: dispatch 2024-04-22T06:36:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:57 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]': finished 2024-04-22T06:36:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:57 smithi193 ceph-mon[31815]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:36:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:57 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:36:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:57 smithi079 ceph-mon[27946]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:57 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/2870145498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]: dispatch 2024-04-22T06:36:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:57 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]: dispatch 2024-04-22T06:36:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:57 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "072857b1-5aa7-4510-b9e9-5b6cc192458c"}]': finished 2024-04-22T06:36:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:57 smithi079 ceph-mon[27946]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:36:58.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:36:59.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:58 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/908662026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:59.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:58 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/908662026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:59.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:58 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/908662026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:00.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:36:59 smithi193 ceph-mon[31815]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:00.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:36:59 smithi190 ceph-mon[31421]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:00.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:36:59 smithi079 ceph-mon[27946]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:01 smithi190 ceph-mon[31421]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:01 smithi193 ceph-mon[31815]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:01 smithi079 ceph-mon[27946]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:03.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:03 smithi190 ceph-mon[31421]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:04.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:03 smithi193 ceph-mon[31815]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:03 smithi079 ceph-mon[27946]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:06.042 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:05 smithi079 ceph-mon[27946]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:06.043 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:37:06.043 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:06.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:05 smithi190 ceph-mon[31421]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:06.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:37:06.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:05 smithi193 ceph-mon[31815]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:37:06.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:07.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:06 smithi190 ceph-mon[31421]: Deploying daemon osd.1 on smithi190 2024-04-22T06:37:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:06 smithi193 ceph-mon[31815]: Deploying daemon osd.1 on smithi190 2024-04-22T06:37:07.238 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:06 smithi079 ceph-mon[27946]: Deploying daemon osd.1 on smithi190 2024-04-22T06:37:08.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:07 smithi190 ceph-mon[31421]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:08.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:07 smithi193 ceph-mon[31815]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:08.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:07 smithi079 ceph-mon[27946]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:09.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:09 smithi190 ceph-mon[31421]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:09 smithi193 ceph-mon[31815]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:10.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:09 smithi079 ceph-mon[27946]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:11 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:11 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:11 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:11 smithi190 ceph-mon[31421]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:11.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:11 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:11 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:11 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:11 smithi193 ceph-mon[31815]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:11 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:11 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:11 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:11 smithi079 ceph-mon[27946]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:13 smithi190 ceph-mon[31421]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.835 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 1 on host 'smithi190' 2024-04-22T06:37:13.835 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.835+0000 7f15f37fe700 1 -- 172.21.15.190:0/3038741426 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f15f4062e80 con 0x7f15dc0a3bb0 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f15dc0a3bb0 msgr2=0x7f15dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f15dc0a3bb0 0x7f15dc0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f15f4103b20 tx=0x7f15ec0091c0 comp rx=0 tx=0).stop 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 msgr2=0x7f15f4074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 0x7f15f4074e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15e800d950 tx=0x7f15e800dc60 comp rx=0 tx=0).stop 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 shutdown_connections 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f15dc0a3bb0 0x7f15dc0a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15f4104660 0x7f15f4075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102ac0 0x7f15f40767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 --2- 172.21.15.190:0/3038741426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15f4103cc0 0x7f15f4074e30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:13.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 >> 172.21.15.190:0/3038741426 conn(0x7f15f40fe060 msgr2=0x7f15f4106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:13.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 shutdown_connections 2024-04-22T06:37:13.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:37:13.838+0000 7f15fc7a2700 1 -- 172.21.15.190:0/3038741426 wait complete. 2024-04-22T06:37:13.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:13 smithi193 ceph-mon[31815]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:13.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:13.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:13.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:13 smithi079 ceph-mon[27946]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:14.480 DEBUG:teuthology.orchestra.run.smithi190:osd.1> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T06:37:14.482 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-04-22T06:37:14.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:37:14.728 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:37:15.035 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:15.035 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:15.035 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:15.035 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:14 smithi193 ceph-mon[31815]: from='osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:15.035 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:14 smithi193 ceph-mon[31815]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:14 smithi190 ceph-mon[31421]: from='osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:14 smithi190 ceph-mon[31421]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:14 smithi079 ceph-mon[27946]: from='osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:14 smithi079 ceph-mon[27946]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:15.928 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:15 smithi190 ceph-mon[31421]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:15.928 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:15 smithi190 ceph-mon[31421]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:37:15.928 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:15 smithi190 ceph-mon[31421]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:37:15.928 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:15 smithi190 ceph-mon[31421]: from='osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:37:15.929 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:15.929 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:15 smithi190 ceph-mon[31421]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:37:16.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:15 smithi193 ceph-mon[31815]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:16.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:15 smithi193 ceph-mon[31815]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:37:16.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:15 smithi193 ceph-mon[31815]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:37:16.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:15 smithi193 ceph-mon[31815]: from='osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:37:16.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:16.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:15 smithi193 ceph-mon[31815]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:37:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:15 smithi079 ceph-mon[27946]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:15 smithi079 ceph-mon[27946]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:37:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:15 smithi079 ceph-mon[27946]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:37:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:15 smithi079 ceph-mon[27946]: from='osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:37:16.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:16.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:15 smithi079 ceph-mon[27946]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:37:16.453 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1[36018]: 2024-04-22T06:37:16.248+0000 7f0d66132700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T06:37:16.453 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1[36018]: 2024-04-22T06:37:16.251+0000 7f0d5d71c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:37:16.605 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:37:16.605 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:37:16.605 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-22T06:37:16.605 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-22T06:37:16.605 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113357 s, 925 MB/s 2024-04-22T06:37:16.605 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-22T06:37:17.048 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-mon[31421]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:37:17.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-mon[31421]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:37:17.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-mon[31421]: from='osd.1 ' entity='osd.1' 2024-04-22T06:37:17.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:16 smithi190 ceph-mon[31421]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:16 smithi193 ceph-mon[31815]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:37:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:16 smithi193 ceph-mon[31815]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:37:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:16 smithi193 ceph-mon[31815]: from='osd.1 ' entity='osd.1' 2024-04-22T06:37:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:16 smithi193 ceph-mon[31815]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:17.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:16 smithi079 ceph-mon[27946]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:16 smithi079 ceph-mon[27946]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:16 smithi079 ceph-mon[27946]: from='osd.1 ' entity='osd.1' 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:16 smithi079 ceph-mon[27946]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:17.461 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:37:17.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:37:17.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:37:17.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] boot 2024-04-22T06:37:17.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:37:17.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi190 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi190 to 17088M 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:17.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:37:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:37:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] boot 2024-04-22T06:37:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi190 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi190 to 17088M 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:18.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: osd.1 [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] boot 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:18.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi190 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi190 to 17088M 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:18.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.001+0000 7f66e7c9a700 1 -- 172.21.15.193:0/1111047179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 msgr2=0x7f66e0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.001+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/1111047179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e0106b00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66d800b0a0 tx=0x7f66d800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.001+0000 7f66e7c9a700 1 -- 172.21.15.193:0/1111047179 shutdown_connections 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.001+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/1111047179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e0106b00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.001+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/1111047179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 0x7f66e0103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.001+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/1111047179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 0x7f66e0106140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- 172.21.15.193:0/1111047179 >> 172.21.15.193:0/1111047179 conn(0x7f66e00fcfa0 msgr2=0x7f66e00ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- 172.21.15.193:0/1111047179 shutdown_connections 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- 172.21.15.193:0/1111047179 wait complete. 2024-04-22T06:37:19.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 Processor -- start 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- start start 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 0x7f66e019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 0x7f66e019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e01a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f66e0114b50 con 0x7f66e0103cb0 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f66e01149d0 con 0x7f66e0101340 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e7c9a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66e0114cd0 con 0x7f66e0106680 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.002+0000 7f66e6237700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e01a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e01a6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50764/0 (socket says 172.21.15.193:50764) 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 -- 172.21.15.193:0/567880118 learned_addr learned my addr 172.21.15.193:0/567880118 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e5235700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 0x7f66e019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 -- 172.21.15.193:0/567880118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 msgr2=0x7f66e019f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e5a36700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 0x7f66e019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 0x7f66e019f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:19.004 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 -- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 msgr2=0x7f66e019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 0x7f66e019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 -- 172.21.15.193:0/567880118 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e01a6dd0 con 0x7f66e0106680 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e5235700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 0x7f66e019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e5a36700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 0x7f66e019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e6237700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e01a6730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66d800cf60 tx=0x7f66d8007770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8021070 con 0x7f66e0106680 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66d8004020 con 0x7f66e0106680 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d801d690 con 0x7f66e0106680 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e01a7060 con 0x7f66e0106680 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.003+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e01a7670 con 0x7f66e0106680 2024-04-22T06:37:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.004+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e0069730 con 0x7f66e0106680 2024-04-22T06:37:19.006 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.005+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f66d80041d0 con 0x7f66e0106680 2024-04-22T06:37:19.006 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.006+0000 7f66d6ffd700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f66cc0a3c30 0x7f66cc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:19.006 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.006+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f66d800f020 con 0x7f66e0106680 2024-04-22T06:37:19.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.006+0000 7f66e5a36700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f66cc0a3c30 0x7f66cc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:19.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.007+0000 7f66e5a36700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f66cc0a3c30 0x7f66cc0a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f66dc005fd0 tx=0x7f66dc005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:19.009 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.010+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f66d802a020 con 0x7f66e0106680 2024-04-22T06:37:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:18 smithi193 ceph-mon[31815]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:18 smithi190 ceph-mon[31421]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:19.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:19.217+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f66e0062e80 con 0x7f66cc0a3c30 2024-04-22T06:37:19.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:18 smithi079 ceph-mon[27946]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:20.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:19 smithi193 ceph-mon[31815]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:37:20.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:19 smithi193 ceph-mon[31815]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:20.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:19 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:20.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:19 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:20.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:19 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:19 smithi190 ceph-mon[31421]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:37:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:19 smithi190 ceph-mon[31421]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:19 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:19 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:19 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:19 smithi079 ceph-mon[27946]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:19 smithi079 ceph-mon[27946]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:19 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:19 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:19 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:20.907 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:20 smithi193 ceph-mon[31815]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:21.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:20 smithi190 ceph-mon[31421]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:21.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:20 smithi079 ceph-mon[27946]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:22.065 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:21 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/3546322855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]: dispatch 2024-04-22T06:37:22.065 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:21 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]: dispatch 2024-04-22T06:37:22.065 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:21 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]': finished 2024-04-22T06:37:22.065 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:21 smithi193 ceph-mon[31815]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:37:22.065 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:21 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/3546322855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]: dispatch 2024-04-22T06:37:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:21 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]: dispatch 2024-04-22T06:37:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:21 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]': finished 2024-04-22T06:37:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:21 smithi190 ceph-mon[31421]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:37:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:21 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/3546322855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]: dispatch 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:21 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]: dispatch 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:21 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e"}]': finished 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:21 smithi079 ceph-mon[27946]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:23.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:22 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/1632776220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:23.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:22 smithi193 ceph-mon[31815]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:22 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/1632776220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:22 smithi190 ceph-mon[31421]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:22 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/1632776220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:22 smithi079 ceph-mon[27946]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:25 smithi193 ceph-mon[31815]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:25 smithi190 ceph-mon[31421]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:25 smithi079 ceph-mon[27946]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:27.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:27 smithi193 ceph-mon[31815]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:27 smithi190 ceph-mon[31421]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:27 smithi079 ceph-mon[27946]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:29 smithi193 ceph-mon[31815]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:37:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:29.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:29 smithi193 ceph-mon[31815]: Deploying daemon osd.2 on smithi193 2024-04-22T06:37:29.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:29 smithi190 ceph-mon[31421]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:29.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:37:29.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:29.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:29 smithi190 ceph-mon[31421]: Deploying daemon osd.2 on smithi193 2024-04-22T06:37:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:29 smithi079 ceph-mon[27946]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:37:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:29 smithi079 ceph-mon[27946]: Deploying daemon osd.2 on smithi193 2024-04-22T06:37:31.820 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:31 smithi193 ceph-mon[31815]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:31.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:31 smithi190 ceph-mon[31421]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:31 smithi079 ceph-mon[27946]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:33.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:33 smithi193 ceph-mon[31815]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:33.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:33 smithi190 ceph-mon[31421]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:33.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:33 smithi079 ceph-mon[27946]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:35.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:35 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:35.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:35 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:35 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:35 smithi193 ceph-mon[31815]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:35 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:35 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:35 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:35 smithi190 ceph-mon[31421]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:35 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:35 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:35 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:35.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:35 smithi079 ceph-mon[27946]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:37.281 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-04-22T06:37:37.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.282+0000 7f66d6ffd700 1 -- 172.21.15.193:0/567880118 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f66e0062e80 con 0x7f66cc0a3c30 2024-04-22T06:37:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f66cc0a3c30 msgr2=0x7f66cc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f66cc0a3c30 0x7f66cc0a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f66dc005fd0 tx=0x7f66dc005dc0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 msgr2=0x7f66e01a6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e01a6730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66d800cf60 tx=0x7f66d8007770 comp rx=0 tx=0).stop 2024-04-22T06:37:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 shutdown_connections 2024-04-22T06:37:37.283 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66e0106680 0x7f66e01a6730 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66e0101340 0x7f66e019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f66cc0a3c30 0x7f66cc0a60f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 --2- 172.21.15.193:0/567880118 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f66e0103cb0 0x7f66e019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 >> 172.21.15.193:0/567880118 conn(0x7f66e00fcfa0 msgr2=0x7f66e0101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:37.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 shutdown_connections 2024-04-22T06:37:37.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:37:37.284+0000 7f66e7c9a700 1 -- 172.21.15.193:0/567880118 wait complete. 2024-04-22T06:37:37.846 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:37 smithi193 ceph-mon[31815]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:37.846 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:37 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:37.846 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:37 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:37.873 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T06:37:37.875 INFO:tasks.cephadm:Deploying osd.3 on smithi079 with /dev/vg_nvme/lv_3... 2024-04-22T06:37:37.875 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:37:37.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:37 smithi079 ceph-mon[27946]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:37.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:37 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:37.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:37 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:37 smithi190 ceph-mon[31421]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:37 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:37 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:38.132 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:38.189 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2[36978]: 2024-04-22T06:37:37.932+0000 7fe8d20457c0 -1 osd.2 0 log_to_monitors true 2024-04-22T06:37:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:38 smithi079 ceph-mon[27946]: from='osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:38 smithi079 ceph-mon[27946]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:38 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:38 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:38.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:38 smithi193 ceph-mon[31815]: from='osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:38.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:38 smithi193 ceph-mon[31815]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:38.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:38 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:38.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:38 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:38.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:38 smithi190 ceph-mon[31421]: from='osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:38.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:38 smithi190 ceph-mon[31421]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:38.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:38 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:38.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:38 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:39.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:39 smithi193 ceph-mon[31815]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:39.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:39 smithi193 ceph-mon[31815]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:37:39.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:39 smithi193 ceph-mon[31815]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:37:39.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:39.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:39 smithi193 ceph-mon[31815]: from='osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:37:39.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:39 smithi193 ceph-mon[31815]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:37:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:39 smithi190 ceph-mon[31421]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:39 smithi190 ceph-mon[31421]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:37:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:39 smithi190 ceph-mon[31421]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:37:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:39 smithi190 ceph-mon[31421]: from='osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:37:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:39 smithi190 ceph-mon[31421]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:37:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:39 smithi079 ceph-mon[27946]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:39 smithi079 ceph-mon[27946]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:37:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:39 smithi079 ceph-mon[27946]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:37:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:39 smithi079 ceph-mon[27946]: from='osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:37:39.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:39 smithi079 ceph-mon[27946]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:37:40.101 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:37:40.101 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:37:40.101 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-04-22T06:37:40.101 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-04-22T06:37:40.101 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111589 s, 940 MB/s 2024-04-22T06:37:40.101 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-04-22T06:37:40.189 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2[36978]: 2024-04-22T06:37:40.080+0000 7fe8c76aa700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T06:37:40.189 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2[36978]: 2024-04-22T06:37:40.085+0000 7fe8bf495700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:37:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:40 smithi079 ceph-mon[27946]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:37:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:40 smithi079 ceph-mon[27946]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:37:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:40 smithi079 ceph-mon[27946]: from='osd.2 ' entity='osd.2' 2024-04-22T06:37:40.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-04-22T06:37:40.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-mon[31815]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:37:40.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-mon[31815]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:37:40.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:40.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:40.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:40 smithi193 ceph-mon[31815]: from='osd.2 ' entity='osd.2' 2024-04-22T06:37:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:40 smithi190 ceph-mon[31421]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:37:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:40 smithi190 ceph-mon[31421]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:37:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:40 smithi190 ceph-mon[31421]: from='osd.2 ' entity='osd.2' 2024-04-22T06:37:41.058 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:41.870 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:37:41.870 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:37:41.870 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:41.870 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] boot 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:41.871 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] boot 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:41.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: osd.2 [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] boot 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:41.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:41.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:41.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:42.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 -- 172.21.15.79:0/1899447726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb01019b0 msgr2=0x7f9eb0101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1899447726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb01019b0 0x7f9eb0101dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0099c0 tx=0x7f9eac009cd0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 -- 172.21.15.79:0/1899447726 shutdown_connections 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1899447726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 0x7f9eb0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1899447726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb01019b0 0x7f9eb0101dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1899447726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eb0102bb0 0x7f9eb0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 -- 172.21.15.79:0/1899447726 >> 172.21.15.79:0/1899447726 conn(0x7f9eb00fcf70 msgr2=0x7f9eb00ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 -- 172.21.15.79:0/1899447726 shutdown_connections 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 -- 172.21.15.79:0/1899447726 wait complete. 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 Processor -- start 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.654+0000 7f9eb7513700 1 -- start start 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eb01019b0 0x7f9eb0198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 0x7f9eb01992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 0x7f9eb01a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb010d4a0 con 0x7f9eb0102bb0 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb010d320 con 0x7f9eb01019b0 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb010d620 con 0x7f9eb0103550 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 0x7f9eb01992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 0x7f9eb01992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49180/0 (socket says 172.21.15.79:49180) 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 -- 172.21.15.79:0/1471444310 learned_addr learned my addr 172.21.15.79:0/1471444310 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb52af700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eb01019b0 0x7f9eb0198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb5ab0700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 0x7f9eb01a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 -- 172.21.15.79:0/1471444310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 msgr2=0x7f9eb01a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 0x7f9eb01a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 -- 172.21.15.79:0/1471444310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eb01019b0 msgr2=0x7f9eb0198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eb01019b0 0x7f9eb0198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 -- 172.21.15.79:0/1471444310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb01a0990 con 0x7f9eb0102bb0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb5ab0700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 0x7f9eb01a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb4aae700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 0x7f9eb01992d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c920 tx=0x7f9ea000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0007b00 con 0x7f9eb0102bb0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea0004ce0 con 0x7f9eb0102bb0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.655+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb01a0c80 con 0x7f9eb0102bb0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.656+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0005550 con 0x7f9eb0102bb0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.656+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb01a1190 con 0x7f9eb0102bb0 2024-04-22T06:37:42.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.657+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb0069760 con 0x7f9eb0102bb0 2024-04-22T06:37:42.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.659+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f9ea0007ca0 con 0x7f9eb0102bb0 2024-04-22T06:37:42.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.659+0000 7f9ea67fc700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9e9c0a3bb0 0x7f9e9c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.659+0000 7f9eb52af700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9e9c0a3bb0 0x7f9e9c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.659+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2851+0+0 (secure 0 0 0) 0x7f9ea00cecd0 con 0x7f9eb0102bb0 2024-04-22T06:37:42.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.660+0000 7f9eb52af700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9e9c0a3bb0 0x7f9e9c0a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9eac006010 tx=0x7f9eac005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.666+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9eb0069760 con 0x7f9eb0102bb0 2024-04-22T06:37:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi193 2024-04-22T06:37:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi193 to 17088M 2024-04-22T06:37:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:42.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:42 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:42.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:42.876+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9eb006b8f0 con 0x7f9e9c0a3bb0 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi193 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi193 to 17088M 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:42.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:42 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:42.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi193 2024-04-22T06:37:42.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi193 to 17088M 2024-04-22T06:37:42.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:42.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:42 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:44.014 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41010]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:44.014 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41010]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.014 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41010]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.014 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41010]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:44.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:44.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:37:44.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:44.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:44.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:44.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:44.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:44.061 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44528]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:44.061 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44528]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44528]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44528]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44531]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:44.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44531]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44531]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:37:43 smithi079 sudo[44531]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:44.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:44.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:37:44.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:44.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:44.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:44.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:44.135 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40160]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:44.135 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40160]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.136 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40160]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.136 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40160]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.136 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40163]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:44.136 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40163]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.136 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40163]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.136 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:37:43 smithi190 sudo[40163]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.439 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41013]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:44.440 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.440 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41013]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.440 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41013]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41016]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41016]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41016]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41016]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41019]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41019]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41019]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 sudo[41019]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40166]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40166]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40166]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40166]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40169]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40169]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40169]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 sudo[40169]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44534]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44534]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44534]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44534]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44540]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44540]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44540]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:44.472 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 sudo[44540]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:45.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:37:45.065+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9ea0098c50 con 0x7f9eb0102bb0 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.065 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:44 smithi079 ceph-mon[27946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:44 smithi193 ceph-mon[31815]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:37:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:45.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:45.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:45.205 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:44 smithi190 ceph-mon[31421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:46.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:46.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:37:46.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:37:46.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2335338318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0398c3f2-3fd7-40da-bdd0-20c01352af3c"}]: dispatch 2024-04-22T06:37:46.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2335338318' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0398c3f2-3fd7-40da-bdd0-20c01352af3c"}]': finished 2024-04-22T06:37:46.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T06:37:46.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:45 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2335338318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0398c3f2-3fd7-40da-bdd0-20c01352af3c"}]: dispatch 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2335338318' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0398c3f2-3fd7-40da-bdd0-20c01352af3c"}]': finished 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T06:37:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:45 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:37:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:37:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2335338318' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0398c3f2-3fd7-40da-bdd0-20c01352af3c"}]: dispatch 2024-04-22T06:37:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2335338318' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0398c3f2-3fd7-40da-bdd0-20c01352af3c"}]': finished 2024-04-22T06:37:46.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T06:37:46.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:47.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:46 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1200611631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:47.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:46 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1200611631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:46 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1200611631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:47 smithi193 ceph-mon[31815]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:48.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:47 smithi190 ceph-mon[31421]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:47 smithi079 ceph-mon[27946]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:49.839 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:49 smithi079 ceph-mon[27946]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:50.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:49 smithi193 ceph-mon[31815]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:50.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:49 smithi190 ceph-mon[31421]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:51.987 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:51 smithi079 ceph-mon[27946]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:51 smithi193 ceph-mon[31815]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:52.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:51 smithi190 ceph-mon[31421]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:37:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:53.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:37:53.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:53.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:37:53.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:54.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:53 smithi079 ceph-mon[27946]: Deploying daemon osd.3 on smithi079 2024-04-22T06:37:54.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:53 smithi079 ceph-mon[27946]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:53 smithi193 ceph-mon[31815]: Deploying daemon osd.3 on smithi079 2024-04-22T06:37:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:53 smithi193 ceph-mon[31815]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:53 smithi190 ceph-mon[31421]: Deploying daemon osd.3 on smithi079 2024-04-22T06:37:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:53 smithi190 ceph-mon[31421]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:55 smithi079 ceph-mon[27946]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:56.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:55 smithi193 ceph-mon[31815]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:55 smithi190 ceph-mon[31421]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.079 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:57 smithi193 ceph-mon[31815]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:57 smithi190 ceph-mon[31421]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:57 smithi079 ceph-mon[27946]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:59.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:59.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:59.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:59.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:37:59 smithi079 ceph-mon[27946]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:59.291 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:59.291 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:59.291 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:59.291 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:37:59 smithi193 ceph-mon[31815]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:37:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:37:59 smithi190 ceph-mon[31421]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:01.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:01 smithi079 ceph-mon[27946]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:01.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:01.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:01.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:01 smithi193 ceph-mon[31815]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:01 smithi190 ceph-mon[31421]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.030 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 3 on host 'smithi079' 2024-04-22T06:38:02.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.031+0000 7f9ea67fc700 1 -- 172.21.15.79:0/1471444310 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9eb006b8f0 con 0x7f9e9c0a3bb0 2024-04-22T06:38:02.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9e9c0a3bb0 msgr2=0x7f9e9c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9e9c0a3bb0 0x7f9e9c0a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9eac006010 tx=0x7f9eac005be0 comp rx=0 tx=0).stop 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 msgr2=0x7f9eb01992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 0x7f9eb01992d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c920 tx=0x7f9ea000cc30 comp rx=0 tx=0).stop 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 shutdown_connections 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9eb0103550 0x7f9eb01a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eb01019b0 0x7f9eb0198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9e9c0a3bb0 0x7f9e9c0a6070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 --2- 172.21.15.79:0/1471444310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9eb0102bb0 0x7f9eb01992d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 >> 172.21.15.79:0/1471444310 conn(0x7f9eb00fcf70 msgr2=0x7f9eb006b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 shutdown_connections 2024-04-22T06:38:02.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:02.033+0000 7f9eb7513700 1 -- 172.21.15.79:0/1471444310 wait complete. 2024-04-22T06:38:02.650 DEBUG:teuthology.orchestra.run.smithi079:osd.3> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T06:38:02.653 INFO:tasks.cephadm:Deploying osd.4 on smithi190 with /dev/vg_nvme/lv_3... 2024-04-22T06:38:02.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:38:02.895 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:38:02.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:02 smithi079 ceph-mon[27946]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:38:02.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.928 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:02 smithi193 ceph-mon[31815]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:38:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:02.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:02 smithi190 ceph-mon[31421]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:38:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:03.930 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:03 smithi079 ceph-mon[27946]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:03.931 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:03 smithi079 ceph-mon[27946]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:38:03.931 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:03 smithi079 ceph-mon[27946]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:38:03.931 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:03 smithi079 ceph-mon[27946]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:38:03.931 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:03 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:03.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:03 smithi193 ceph-mon[31815]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:03.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:03 smithi193 ceph-mon[31815]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:38:03.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:03 smithi193 ceph-mon[31815]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:38:03.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:03 smithi193 ceph-mon[31815]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:38:03.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:03 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:03.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:03 smithi190 ceph-mon[31421]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:03.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:03 smithi190 ceph-mon[31421]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:38:03.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:03 smithi190 ceph-mon[31421]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:38:03.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:03 smithi190 ceph-mon[31421]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:38:03.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:03 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.240 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3[46645]: 2024-04-22T06:38:04.091+0000 7f80f2ba1700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T06:38:04.240 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3[46645]: 2024-04-22T06:38:04.096+0000 7f80ea98c700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:38:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:38:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:38:04.887 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-22T06:38:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-22T06:38:04.887 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113458 s, 924 MB/s 2024-04-22T06:38:04.887 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-22T06:38:04.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:38:04.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:38:04.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' 2024-04-22T06:38:04.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] boot 2024-04-22T06:38:04.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T06:38:04.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:38:04.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:38:04.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' 2024-04-22T06:38:04.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] boot 2024-04-22T06:38:04.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T06:38:04.920 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:38:04.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:38:04.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:04.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: from='osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201]' entity='osd.3' 2024-04-22T06:38:04.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: osd.3 [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] boot 2024-04-22T06:38:04.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T06:38:04.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:05.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-04-22T06:38:05.809 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:38:05.844 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:05.845 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:05.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:38:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:38:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:05.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:06.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:06 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi079 2024-04-22T06:38:06.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:06 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi079 to 8544M 2024-04-22T06:38:06.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:06 smithi190 ceph-mon[31421]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:38:06.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:06 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi079 2024-04-22T06:38:06.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:06 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi079 to 8544M 2024-04-22T06:38:06.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:06 smithi193 ceph-mon[31815]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:38:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:06 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi079 2024-04-22T06:38:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:06 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi079 to 8544M 2024-04-22T06:38:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:06 smithi079 ceph-mon[27946]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 -- 172.21.15.190:0/1722534159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8104640 msgr2=0x7f93c810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1722534159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8104640 0x7f93c810aef0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93c000b0d0 tx=0x7f93c000b3e0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 -- 172.21.15.190:0/1722534159 shutdown_connections 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1722534159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8104640 0x7f93c810aef0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1722534159 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8102aa0 0x7f93c8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1722534159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8103ca0 0x7f93c8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 -- 172.21.15.190:0/1722534159 >> 172.21.15.190:0/1722534159 conn(0x7f93c80fe040 msgr2=0x7f93c8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 -- 172.21.15.190:0/1722534159 shutdown_connections 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.384+0000 7f93cf59f700 1 -- 172.21.15.190:0/1722534159 wait complete. 2024-04-22T06:38:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 Processor -- start 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 -- start start 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 0x7f93c8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8103ca0 0x7f93c81990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8104640 0x7f93c81a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f93c810e5e0 con 0x7f93c8103ca0 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f93c810e460 con 0x7f93c8102aa0 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cf59f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f93c810e760 con 0x7f93c8104640 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cd33b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 0x7f93c8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cd33b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 0x7f93c8198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:54718/0 (socket says 172.21.15.190:54718) 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cd33b700 1 -- 172.21.15.190:0/1389620288 learned_addr learned my addr 172.21.15.190:0/1389620288 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93cdb3c700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8104640 0x7f93c81a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.385+0000 7f93ccb3a700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8103ca0 0x7f93c81990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cd33b700 1 -- 172.21.15.190:0/1389620288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8104640 msgr2=0x7f93c81a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cd33b700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8104640 0x7f93c81a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cd33b700 1 -- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8103ca0 msgr2=0x7f93c81990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cd33b700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8103ca0 0x7f93c81990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cd33b700 1 -- 172.21.15.190:0/1389620288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c81a0800 con 0x7f93c8102aa0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cdb3c700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8104640 0x7f93c81a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93ccb3a700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8103ca0 0x7f93c81990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cd33b700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 0x7f93c8198ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93c400c990 tx=0x7f93c400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4007b00 con 0x7f93c8102aa0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c81a0af0 con 0x7f93c8102aa0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c4007ca0 con 0x7f93c8102aa0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4018660 con 0x7f93c8102aa0 2024-04-22T06:38:07.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.386+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c81a1080 con 0x7f93c8102aa0 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.387+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c8069730 con 0x7f93c8102aa0 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.388+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f93c4004030 con 0x7f93c8102aa0 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.389+0000 7f93be7fc700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93b40a3c40 0x7f93b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.389+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f93c40cf280 con 0x7f93c8102aa0 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.389+0000 7f93ccb3a700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93b40a3c40 0x7f93b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.390+0000 7f93ccb3a700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93b40a3c40 0x7f93b40a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93b8007900 tx=0x7f93b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.393+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93c4014030 con 0x7f93c8102aa0 2024-04-22T06:38:07.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:07.602+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f93c8062e80 con 0x7f93b40a3c40 2024-04-22T06:38:07.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:07 smithi193 ceph-mon[31815]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 157 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:07.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:07 smithi193 ceph-mon[31815]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:38:07.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:07.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:07.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:07.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:07 smithi190 ceph-mon[31421]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 157 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:07.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:07 smithi190 ceph-mon[31421]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:38:07.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:07.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:07.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:07 smithi079 ceph-mon[27946]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 157 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:07 smithi079 ceph-mon[27946]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:38:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:07.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:08.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:08 smithi193 ceph-mon[31815]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:08.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:08 smithi193 ceph-mon[31815]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:38:08.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:08 smithi190 ceph-mon[31421]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:08.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:08 smithi190 ceph-mon[31421]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:38:08.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:08 smithi079 ceph-mon[27946]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:08.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:08 smithi079 ceph-mon[27946]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:38:09.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:09 smithi193 ceph-mon[31815]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 157 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:38:09.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:09 smithi190 ceph-mon[31421]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 157 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:38:09.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:09 smithi079 ceph-mon[27946]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 157 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:38:10.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:10 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/981936077' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]: dispatch 2024-04-22T06:38:10.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:10 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]: dispatch 2024-04-22T06:38:10.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:10 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]': finished 2024-04-22T06:38:10.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:10 smithi193 ceph-mon[31815]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T06:38:10.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:10.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:10 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/981936077' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]: dispatch 2024-04-22T06:38:10.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:10 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]: dispatch 2024-04-22T06:38:10.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:10 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]': finished 2024-04-22T06:38:10.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:10 smithi190 ceph-mon[31421]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T06:38:10.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/981936077' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]: dispatch 2024-04-22T06:38:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:10 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]: dispatch 2024-04-22T06:38:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:10 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b4fb272-de89-414a-b82c-95b8169e29a8"}]': finished 2024-04-22T06:38:10.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:10 smithi079 ceph-mon[27946]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T06:38:10.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:11.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:11 smithi193 ceph-mon[31815]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:38:11.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:11 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/1528807505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:11.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:11 smithi190 ceph-mon[31421]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:38:11.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:11 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/1528807505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:11.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:11 smithi079 ceph-mon[27946]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:38:11.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:11 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/1528807505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:13.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:13 smithi193 ceph-mon[31815]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:38:13.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:13 smithi190 ceph-mon[31421]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:38:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:13 smithi079 ceph-mon[27946]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:38:15.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:15 smithi193 ceph-mon[31815]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T06:38:15.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:15 smithi190 ceph-mon[31421]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T06:38:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:15 smithi079 ceph-mon[27946]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T06:38:17.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:17 smithi193 ceph-mon[31815]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:17.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:17 smithi190 ceph-mon[31421]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:17.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:17 smithi079 ceph-mon[27946]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:38:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:18 smithi193 ceph-mon[31815]: Deploying daemon osd.4 on smithi190 2024-04-22T06:38:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:38:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:18 smithi190 ceph-mon[31421]: Deploying daemon osd.4 on smithi190 2024-04-22T06:38:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:38:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:18 smithi079 ceph-mon[27946]: Deploying daemon osd.4 on smithi190 2024-04-22T06:38:19.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:19 smithi193 ceph-mon[31815]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:19.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:19 smithi190 ceph-mon[31421]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:19.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:19 smithi079 ceph-mon[27946]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:21.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:21 smithi190 ceph-mon[31421]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:21.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:21 smithi193 ceph-mon[31815]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:21.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:21 smithi079 ceph-mon[27946]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:23 smithi190 ceph-mon[31421]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:23 smithi079 ceph-mon[27946]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:24.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:23 smithi193 ceph-mon[31815]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:24.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:24.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:24.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:25 smithi190 ceph-mon[31421]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:25 smithi079 ceph-mon[27946]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:26.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:25 smithi193 ceph-mon[31815]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:27.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.165+0000 7f93be7fc700 1 -- 172.21.15.190:0/1389620288 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f93c8062e80 con 0x7f93b40a3c40 2024-04-22T06:38:27.165 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 4 on host 'smithi190' 2024-04-22T06:38:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93b40a3c40 msgr2=0x7f93b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93b40a3c40 0x7f93b40a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93b8007900 tx=0x7f93b8008040 comp rx=0 tx=0).stop 2024-04-22T06:38:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 msgr2=0x7f93c8198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 0x7f93c8198ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93c400c990 tx=0x7f93c400cd50 comp rx=0 tx=0).stop 2024-04-22T06:38:27.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 shutdown_connections 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c8104640 0x7f93c81a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f93c8102aa0 0x7f93c8198ba0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93b40a3c40 0x7f93b40a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 --2- 172.21.15.190:0/1389620288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93c8103ca0 0x7f93c81990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 >> 172.21.15.190:0/1389620288 conn(0x7f93c80fe040 msgr2=0x7f93c8106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 shutdown_connections 2024-04-22T06:38:27.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:38:27.168+0000 7f93cf59f700 1 -- 172.21.15.190:0/1389620288 wait complete. 2024-04-22T06:38:27.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:27.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:27.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:27.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:27.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:27.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:27 smithi193 ceph-mon[31815]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:27.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:27 smithi190 ceph-mon[31421]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:27.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:27.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:27 smithi079 ceph-mon[27946]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:27.868 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T06:38:27.870 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-04-22T06:38:27.871 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:38:28.116 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:38:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:29 smithi193 ceph-mon[31815]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:38:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:29 smithi193 ceph-mon[31815]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:38:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:29 smithi193 ceph-mon[31815]: from='osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:38:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:29 smithi193 ceph-mon[31815]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:38:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:29 smithi193 ceph-mon[31815]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-mon[31421]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:38:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-mon[31421]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:38:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-mon[31421]: from='osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:38:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-mon[31421]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:38:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-mon[31421]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:29 smithi079 ceph-mon[27946]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:38:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:29 smithi079 ceph-mon[27946]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:38:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:29 smithi079 ceph-mon[27946]: from='osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:38:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:29 smithi079 ceph-mon[27946]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:38:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:29 smithi079 ceph-mon[27946]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:29.826 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4[42955]: 2024-04-22T06:38:29.572+0000 7f089031d700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T06:38:29.827 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:38:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4[42955]: 2024-04-22T06:38:29.578+0000 7f0887907700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:38:30.136 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:38:30.136 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:38:30.136 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-22T06:38:30.136 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-22T06:38:30.136 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113601 s, 923 MB/s 2024-04-22T06:38:30.136 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-22T06:38:30.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:30 smithi193 ceph-mon[31815]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:38:30.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:30 smithi193 ceph-mon[31815]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:38:30.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:30 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:30.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:30 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:30.437 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:30 smithi193 ceph-mon[31815]: from='osd.4 ' entity='osd.4' 2024-04-22T06:38:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:30 smithi190 ceph-mon[31421]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:38:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:30 smithi190 ceph-mon[31421]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:38:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:30 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:30 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:30 smithi190 ceph-mon[31421]: from='osd.4 ' entity='osd.4' 2024-04-22T06:38:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:30 smithi079 ceph-mon[27946]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:38:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:30 smithi079 ceph-mon[27946]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:38:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:30 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:30 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:30 smithi079 ceph-mon[27946]: from='osd.4 ' entity='osd.4' 2024-04-22T06:38:30.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-04-22T06:38:31.023 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] boot 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:31.396 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:31 smithi193 ceph-mon[31815]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] boot 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:31 smithi190 ceph-mon[31421]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:31.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:38:31.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:38:31.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:31.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: osd.4 [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] boot 2024-04-22T06:38:31.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T06:38:31.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:31.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:31 smithi079 ceph-mon[27946]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:32.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/1422429499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 msgr2=0x7fc8f4109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:32.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/1422429499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f4109df0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00b0a0 tx=0x7fc8ec00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:38:32.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/1422429499 shutdown_connections 2024-04-22T06:38:32.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/1422429499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f4109df0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:32.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/1422429499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8f41019a0 0x7fc8f4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:32.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/1422429499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8f4102ba0 0x7fc8f4103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/1422429499 >> 172.21.15.193:0/1422429499 conn(0x7fc8f40fcf40 msgr2=0x7fc8f40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/1422429499 shutdown_connections 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.629+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/1422429499 wait complete. 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 Processor -- start 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 -- start start 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8f41019a0 0x7fc8f4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8f4102ba0 0x7fc8f41991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f41a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f410d4b0 con 0x7fc8f4102ba0 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f410d330 con 0x7fc8f41019a0 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8fa4b8700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f410d630 con 0x7fc8f4103540 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f8a55700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f41a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:32.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f8a55700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f41a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:37950/0 (socket says 172.21.15.193:37950) 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f8a55700 1 -- 172.21.15.193:0/2964395133 learned_addr learned my addr 172.21.15.193:0/2964395133 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f8a55700 1 -- 172.21.15.193:0/2964395133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8f41019a0 msgr2=0x7fc8f4198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f3fff700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8f41019a0 0x7fc8f4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f8a55700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8f41019a0 0x7fc8f4198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f8a55700 1 -- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8f4102ba0 msgr2=0x7fc8f41991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.630+0000 7fc8f37fe700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8f4102ba0 0x7fc8f41991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8f8a55700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8f4102ba0 0x7fc8f41991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8f8a55700 1 -- 172.21.15.193:0/2964395133 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f41a08a0 con 0x7fc8f4103540 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8f8a55700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f41a01d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0062a0 tx=0x7fc8ec004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec021070 con 0x7fc8f4103540 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f41a0b30 con 0x7fc8f4103540 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8ec007db0 con 0x7fc8f4103540 2024-04-22T06:38:32.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec01d450 con 0x7fc8f4103540 2024-04-22T06:38:32.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.631+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8f41a1030 con 0x7fc8f4103540 2024-04-22T06:38:32.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.632+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f4069730 con 0x7fc8f4103540 2024-04-22T06:38:32.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.633+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc8ec004000 con 0x7fc8f4103540 2024-04-22T06:38:32.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.633+0000 7fc8f17fa700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8dc0a3b20 0x7fc8dc0a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:32.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.633+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fc8ec00f020 con 0x7fc8f4103540 2024-04-22T06:38:32.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.634+0000 7fc8f3fff700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8dc0a3b20 0x7fc8dc0a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:32.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.634+0000 7fc8f3fff700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8dc0a3b20 0x7fc8dc0a5fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc8f419a130 tx=0x7fc8e8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:32.637 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.638+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8ec02a070 con 0x7fc8f4103540 2024-04-22T06:38:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi190 2024-04-22T06:38:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi190 to 8544M 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:32 smithi193 ceph-mon[31815]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi190 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi190 to 8544M 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:32 smithi190 ceph-mon[31421]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi190 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi190 to 8544M 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:32 smithi079 ceph-mon[27946]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:38:32.842 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:32.843+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc8f40ff150 con 0x7fc8dc0a3b20 2024-04-22T06:38:33.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:33 smithi193 ceph-mon[31815]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:33.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:33 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:33.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:33 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:33.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:33 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:33.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:33 smithi190 ceph-mon[31421]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:33.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:33 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:33.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:33 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:33.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:33 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:33 smithi079 ceph-mon[27946]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:33 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:33 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:33 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:34.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:34 smithi193 ceph-mon[31815]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:34.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:34 smithi190 ceph-mon[31421]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:34.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:34 smithi079 ceph-mon[27946]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:35.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:35 smithi193 ceph-mon[31815]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:35.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:35 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/1315801262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]: dispatch 2024-04-22T06:38:35.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:35 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]: dispatch 2024-04-22T06:38:35.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:35 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]': finished 2024-04-22T06:38:35.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:35 smithi193 ceph-mon[31815]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T06:38:35.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:35 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:35.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:35 smithi190 ceph-mon[31421]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:35.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:35 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/1315801262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]: dispatch 2024-04-22T06:38:35.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:35 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]: dispatch 2024-04-22T06:38:35.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:35 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]': finished 2024-04-22T06:38:35.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:35 smithi190 ceph-mon[31421]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T06:38:35.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:35 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:35.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:35 smithi079 ceph-mon[27946]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:35.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/1315801262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]: dispatch 2024-04-22T06:38:35.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:35 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]: dispatch 2024-04-22T06:38:35.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:35 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6cbf2d1-2a90-41c3-a524-243e7ed3e105"}]': finished 2024-04-22T06:38:35.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:35 smithi079 ceph-mon[27946]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T06:38:35.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:35 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:36 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/1770493721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:36.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:36 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/1770493721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:36.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:36 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/1770493721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:37.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:37 smithi193 ceph-mon[31815]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:37 smithi190 ceph-mon[31421]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:37 smithi079 ceph-mon[27946]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:39.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:39 smithi193 ceph-mon[31815]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:39.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:39 smithi190 ceph-mon[31421]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:39 smithi079 ceph-mon[27946]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:41.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:41 smithi193 ceph-mon[31815]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:41.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:41 smithi190 ceph-mon[31421]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:41.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:41 smithi079 ceph-mon[27946]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:43 smithi193 ceph-mon[31815]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:38:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:43 smithi193 ceph-mon[31815]: Deploying daemon osd.5 on smithi193 2024-04-22T06:38:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:43 smithi190 ceph-mon[31421]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:38:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:43 smithi190 ceph-mon[31421]: Deploying daemon osd.5 on smithi193 2024-04-22T06:38:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:43 smithi079 ceph-mon[27946]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:38:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:43 smithi079 ceph-mon[27946]: Deploying daemon osd.5 on smithi193 2024-04-22T06:38:45.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:45 smithi193 ceph-mon[31815]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:45.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:45 smithi190 ceph-mon[31421]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:45 smithi079 ceph-mon[27946]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:47.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:47 smithi193 ceph-mon[31815]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:47.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:47 smithi190 ceph-mon[31421]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:47 smithi079 ceph-mon[27946]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:49 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:49 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:49 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:49 smithi193 ceph-mon[31815]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:49.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:49.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:49.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:49.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:49 smithi079 ceph-mon[27946]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:49 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:49.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:49 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:49.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:49 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:49.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:49 smithi190 ceph-mon[31421]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:51.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.657+0000 7fc8f17fa700 1 -- 172.21.15.193:0/2964395133 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc8f40ff150 con 0x7fc8dc0a3b20 2024-04-22T06:38:51.657 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8dc0a3b20 msgr2=0x7fc8dc0a5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8dc0a3b20 0x7fc8dc0a5fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc8f419a130 tx=0x7fc8e8008040 comp rx=0 tx=0).stop 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 msgr2=0x7fc8f41a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f41a01d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0062a0 tx=0x7fc8ec004ef0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 shutdown_connections 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8f4103540 0x7fc8f41a01d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8f41019a0 0x7fc8f4198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8dc0a3b20 0x7fc8dc0a5fe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 --2- 172.21.15.193:0/2964395133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8f4102ba0 0x7fc8f41991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 >> 172.21.15.193:0/2964395133 conn(0x7fc8f40fcf40 msgr2=0x7fc8f40fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 shutdown_connections 2024-04-22T06:38:51.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:38:51.660+0000 7fc8fa4b8700 1 -- 172.21.15.193:0/2964395133 wait complete. 2024-04-22T06:38:51.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:51 smithi193 ceph-mon[31815]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:51.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:51 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:51.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:51 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:51.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:51 smithi190 ceph-mon[31421]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:51.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:51 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:51.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:51 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:51 smithi079 ceph-mon[27946]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:51 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:51 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:52.452 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T06:38:52.454 INFO:tasks.cephadm:Deploying osd.6 on smithi079 with /dev/vg_nvme/lv_2... 2024-04-22T06:38:52.455 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:38:52.700 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:38:52.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:52.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:52.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:52 smithi079 ceph-mon[27946]: from='osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:52.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:52 smithi079 ceph-mon[27946]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:52 smithi193 ceph-mon[31815]: from='osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:52.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:52 smithi193 ceph-mon[31815]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:53.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:53.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:53.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:52 smithi190 ceph-mon[31421]: from='osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:53.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:52 smithi190 ceph-mon[31421]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:53.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-mon[31815]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:53.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-mon[31815]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:38:53.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-mon[31815]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:38:53.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:53.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-mon[31815]: from='osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:38:53.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-mon[31815]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:38:53.940 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:38:53.734+0000 7fea2f8c9700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T06:38:53.940 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:38:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:38:53.743+0000 7fea2b6bc700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:38:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:53 smithi079 ceph-mon[27946]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:53 smithi079 ceph-mon[27946]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:38:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:53 smithi079 ceph-mon[27946]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:38:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:53 smithi079 ceph-mon[27946]: from='osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:38:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:53 smithi079 ceph-mon[27946]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:38:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:53 smithi190 ceph-mon[31421]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:53 smithi190 ceph-mon[31421]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:38:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:53 smithi190 ceph-mon[31421]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:38:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:53 smithi190 ceph-mon[31421]: from='osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:38:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:53 smithi190 ceph-mon[31421]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:38:54.651 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:38:54.651 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:38:54.651 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-04-22T06:38:54.651 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-04-22T06:38:54.651 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114243 s, 918 MB/s 2024-04-22T06:38:54.651 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-04-22T06:38:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:38:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:38:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] boot 2024-04-22T06:38:54.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T06:38:54.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:38:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:38:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] boot 2024-04-22T06:38:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T06:38:55.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: osd.5 [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] boot 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T06:38:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:55.353 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-04-22T06:38:55.591 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:38:55.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:55 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:38:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:38:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:55 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:56.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:56.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:56.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:56.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:56.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:55 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi193 2024-04-22T06:38:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi193 to 8544M 2024-04-22T06:38:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:56.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:56.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:56.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:56 smithi079 ceph-mon[27946]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:38:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi193 2024-04-22T06:38:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi193 to 8544M 2024-04-22T06:38:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:56 smithi193 ceph-mon[31815]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:38:57.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi193 2024-04-22T06:38:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi193 to 8544M 2024-04-22T06:38:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:38:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:56 smithi190 ceph-mon[31421]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:38:57.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 -- 172.21.15.79:0/1924580273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0102aa0 msgr2=0x7f51d0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:57.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 --2- 172.21.15.79:0/1924580273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0102aa0 0x7f51d0102ec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f51c00099c0 tx=0x7f51c0009cd0 comp rx=0 tx=0).stop 2024-04-22T06:38:57.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 -- 172.21.15.79:0/1924580273 shutdown_connections 2024-04-22T06:38:57.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 --2- 172.21.15.79:0/1924580273 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d0104640 0x7f51d010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 --2- 172.21.15.79:0/1924580273 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0102aa0 0x7f51d0102ec0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 --2- 172.21.15.79:0/1924580273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0103ca0 0x7f51d0104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 -- 172.21.15.79:0/1924580273 >> 172.21.15.79:0/1924580273 conn(0x7f51d00fe040 msgr2=0x7f51d0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 -- 172.21.15.79:0/1924580273 shutdown_connections 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.244+0000 7f51d6479700 1 -- 172.21.15.79:0/1924580273 wait complete. 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 Processor -- start 2024-04-22T06:38:57.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 -- start start 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0102aa0 0x7f51d0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d0103ca0 0x7f51d01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 0x7f51d01a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f51d010e590 con 0x7f51d0104640 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f51d010e410 con 0x7f51d0103ca0 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d6479700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f51d010e710 con 0x7f51d0102aa0 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d4a16700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 0x7f51d01a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d4a16700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 0x7f51d01a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47864/0 (socket says 172.21.15.79:47864) 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51d4a16700 1 -- 172.21.15.79:0/4158151497 learned_addr learned my addr 172.21.15.79:0/4158151497 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:38:57.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51cffff700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0102aa0 0x7f51d0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.245+0000 7f51cf7fe700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d0103ca0 0x7f51d01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d4a16700 1 -- 172.21.15.79:0/4158151497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0102aa0 msgr2=0x7f51d0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d4a16700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0102aa0 0x7f51d0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d4a16700 1 -- 172.21.15.79:0/4158151497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d0103ca0 msgr2=0x7f51d01991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d4a16700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d0103ca0 0x7f51d01991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d4a16700 1 -- 172.21.15.79:0/4158151497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d01a08e0 con 0x7f51d0104640 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d4a16700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 0x7f51d01a0210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51c8007ee0 tx=0x7f51c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c800dbd0 con 0x7f51d0104640 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d0074cd0 con 0x7f51d0104640 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c800dd70 con 0x7f51d0104640 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8005990 con 0x7f51d0104640 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.246+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d00751e0 con 0x7f51d0104640 2024-04-22T06:38:57.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.247+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d0069730 con 0x7f51d0104640 2024-04-22T06:38:57.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.248+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f51c801a020 con 0x7f51d0104640 2024-04-22T06:38:57.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.249+0000 7f51cd7fa700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51bc0a3cc0 0x7f51bc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:57.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.249+0000 7f51cffff700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51bc0a3cc0 0x7f51bc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:57.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.249+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f51c80cf360 con 0x7f51d0104640 2024-04-22T06:38:57.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.249+0000 7f51cffff700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51bc0a3cc0 0x7f51bc0a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f51c00038c0 tx=0x7f51c000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:57.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.253+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51c8010070 con 0x7f51d0104640 2024-04-22T06:38:57.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:38:57.461+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f51d0062e80 con 0x7f51bc0a3cc0 2024-04-22T06:38:57.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:57 smithi079 ceph-mon[27946]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:38:57.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:57.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:57.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:57 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:58.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:57 smithi193 ceph-mon[31815]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:38:58.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:57 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:58.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:57 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:58.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:57 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:58.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:57 smithi190 ceph-mon[31421]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:38:58.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:57 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:58.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:57 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:58.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:57 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:59.070 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:58 smithi079 ceph-mon[27946]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:59.071 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:58 smithi079 ceph-mon[27946]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:38:59.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:58 smithi193 ceph-mon[31815]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:59.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:58 smithi193 ceph-mon[31815]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:38:59.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:58 smithi190 ceph-mon[31421]: from='client.14331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:59.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:58 smithi190 ceph-mon[31421]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:39:00.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:59 smithi193 ceph-mon[31815]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:39:00.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:38:59 smithi193 ceph-mon[31815]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:39:00.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:59 smithi190 ceph-mon[31421]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:39:00.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:38:59 smithi190 ceph-mon[31421]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:39:00.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:59 smithi079 ceph-mon[27946]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:39:00.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:38:59 smithi079 ceph-mon[27946]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:39:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:00 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3862072893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e2a0de-8900-4bfc-afb5-ed608c8f174d"}]: dispatch 2024-04-22T06:39:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:00 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3862072893' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95e2a0de-8900-4bfc-afb5-ed608c8f174d"}]': finished 2024-04-22T06:39:00.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:00 smithi079 ceph-mon[27946]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T06:39:00.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:00 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:01.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:00 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3862072893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e2a0de-8900-4bfc-afb5-ed608c8f174d"}]: dispatch 2024-04-22T06:39:01.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:00 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3862072893' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95e2a0de-8900-4bfc-afb5-ed608c8f174d"}]': finished 2024-04-22T06:39:01.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:00 smithi193 ceph-mon[31815]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T06:39:01.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:00 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:01.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:00 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3862072893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95e2a0de-8900-4bfc-afb5-ed608c8f174d"}]: dispatch 2024-04-22T06:39:01.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:00 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3862072893' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95e2a0de-8900-4bfc-afb5-ed608c8f174d"}]': finished 2024-04-22T06:39:01.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:00 smithi190 ceph-mon[31421]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T06:39:01.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:00 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:01 smithi193 ceph-mon[31815]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-22T06:39:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:01 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/730459311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:02.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:01 smithi190 ceph-mon[31421]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-22T06:39:02.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:01 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/730459311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:01 smithi079 ceph-mon[27946]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-22T06:39:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:01 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/730459311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:04.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:03 smithi193 ceph-mon[31815]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:39:04.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:03 smithi190 ceph-mon[31421]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:39:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:03 smithi079 ceph-mon[27946]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:39:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:05 smithi079 ceph-mon[27946]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-22T06:39:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:05 smithi193 ceph-mon[31815]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-22T06:39:06.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:05 smithi190 ceph-mon[31421]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-22T06:39:07.979 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:07 smithi079 ceph-mon[27946]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:08.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:07 smithi193 ceph-mon[31815]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:08.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:07 smithi190 ceph-mon[31421]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:08.898 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:08 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:39:08.898 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:08 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:09.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:08 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:39:09.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:08 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:08 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:39:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:08 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:09.930 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:09 smithi079 ceph-mon[27946]: Deploying daemon osd.6 on smithi079 2024-04-22T06:39:09.931 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:09 smithi079 ceph-mon[27946]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:09 smithi193 ceph-mon[31815]: Deploying daemon osd.6 on smithi079 2024-04-22T06:39:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:09 smithi193 ceph-mon[31815]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:09 smithi190 ceph-mon[31421]: Deploying daemon osd.6 on smithi079 2024-04-22T06:39:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:09 smithi190 ceph-mon[31421]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:12.039 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:11 smithi079 ceph-mon[27946]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:12.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:11 smithi193 ceph-mon[31815]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:12.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:11 smithi190 ceph-mon[31421]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:13 smithi079 ceph-mon[27946]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:14.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:13 smithi193 ceph-mon[31815]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:14.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:13 smithi190 ceph-mon[31421]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:14 smithi079 ceph-mon[27946]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:15.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:15.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:15.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:15.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:14 smithi193 ceph-mon[31815]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:15.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:15.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:15.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:15.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:14 smithi190 ceph-mon[31421]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:17.858 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:17 smithi079 ceph-mon[27946]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:17.858 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.858 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.858 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:17.858 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:17.858 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.919 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 6 on host 'smithi079' 2024-04-22T06:39:17.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.920+0000 7f51cd7fa700 1 -- 172.21.15.79:0/4158151497 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f51d0062e80 con 0x7f51bc0a3cc0 2024-04-22T06:39:17.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.922+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51bc0a3cc0 msgr2=0x7f51bc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:17.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.922+0000 7f51d6479700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51bc0a3cc0 0x7f51bc0a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f51c00038c0 tx=0x7f51c000b540 comp rx=0 tx=0).stop 2024-04-22T06:39:17.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.922+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 msgr2=0x7f51d01a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:17.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.922+0000 7f51d6479700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 0x7f51d01a0210 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51c8007ee0 tx=0x7f51c800d3b0 comp rx=0 tx=0).stop 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 shutdown_connections 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51bc0a3cc0 0x7f51bc0a6180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f51d0104640 0x7f51d01a0210 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0102aa0 0x7f51d0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 --2- 172.21.15.79:0/4158151497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51d0103ca0 0x7f51d01991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 >> 172.21.15.79:0/4158151497 conn(0x7f51d00fe040 msgr2=0x7f51d0107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 shutdown_connections 2024-04-22T06:39:17.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:39:17.923+0000 7f51d6479700 1 -- 172.21.15.79:0/4158151497 wait complete. 2024-04-22T06:39:17.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:17 smithi193 ceph-mon[31815]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:17.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:17.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:17.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:17 smithi190 ceph-mon[31421]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.647 DEBUG:teuthology.orchestra.run.smithi079:osd.6> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T06:39:18.649 INFO:tasks.cephadm:Deploying osd.7 on smithi190 with /dev/vg_nvme/lv_2... 2024-04-22T06:39:18.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:39:18.893 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:39:18.897 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:18 smithi079 ceph-mon[27946]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:39:18.897 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.898 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.898 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:18 smithi193 ceph-mon[31815]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:39:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:18 smithi190 ceph-mon[31421]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:39:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:19.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:19 smithi193 ceph-mon[31815]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:19.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:19 smithi193 ceph-mon[31815]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:39:19.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:19 smithi193 ceph-mon[31815]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T06:39:19.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:19 smithi193 ceph-mon[31815]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:39:19.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:19 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:19.950 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:19 smithi079 ceph-mon[27946]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:19.950 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:19 smithi079 ceph-mon[27946]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:39:19.950 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:19 smithi079 ceph-mon[27946]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T06:39:19.950 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:19 smithi079 ceph-mon[27946]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:39:19.950 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:19 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:19.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:19 smithi190 ceph-mon[31421]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:19.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:19 smithi190 ceph-mon[31421]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:39:19.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:19 smithi190 ceph-mon[31421]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T06:39:19.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:19 smithi190 ceph-mon[31421]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:39:19.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:19 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:39:20.119+0000 7f363ee3f700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T06:39:20.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:39:20.128+0000 7f3636c2a700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:39:20.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:20 smithi193 ceph-mon[31815]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:39:20.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:20 smithi193 ceph-mon[31815]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:39:20.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:20 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:20 smithi193 ceph-mon[31815]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' 2024-04-22T06:39:20.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:20 smithi190 ceph-mon[31421]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:39:20.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:20 smithi190 ceph-mon[31421]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:39:20.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:20 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:20 smithi190 ceph-mon[31421]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-mon[27946]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-mon[27946]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:20 smithi079 ceph-mon[27946]: from='osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926]' entity='osd.6' 2024-04-22T06:39:21.012 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:39:21.012 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:39:21.012 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-22T06:39:21.012 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-22T06:39:21.013 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0152834 s, 686 MB/s 2024-04-22T06:39:21.013 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 2024-04-22T06:39:21.013 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-22T06:39:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:39:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:39:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:21.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] boot 2024-04-22T06:39:21.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T06:39:21.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:21.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-04-22T06:39:21.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:39:21.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:39:21.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:21.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:21.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] boot 2024-04-22T06:39:21.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T06:39:21.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:21.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:39:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:39:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: osd.6 [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] boot 2024-04-22T06:39:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T06:39:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:22.060 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:39:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi079 2024-04-22T06:39:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi079 to 5696M 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:23.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:22 smithi190 ceph-mon[31421]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T06:39:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi079 2024-04-22T06:39:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi079 to 5696M 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:22 smithi079 ceph-mon[27946]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T06:39:23.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi079 2024-04-22T06:39:23.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi079 to 5696M 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:23.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:22 smithi193 ceph-mon[31815]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 -- 172.21.15.190:0/2159908518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 msgr2=0x7f35481095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 --2- 172.21.15.190:0/2159908518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481095c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3540009fc0 tx=0x7f354000b330 comp rx=0 tx=0).stop 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 -- 172.21.15.190:0/2159908518 shutdown_connections 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 --2- 172.21.15.190:0/2159908518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481095c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 --2- 172.21.15.190:0/2159908518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3548101170 0x7f3548101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 --2- 172.21.15.190:0/2159908518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3548102370 0x7f35481027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 -- 172.21.15.190:0/2159908518 >> 172.21.15.190:0/2159908518 conn(0x7f35480fc710 msgr2=0x7f35480feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 -- 172.21.15.190:0/2159908518 shutdown_connections 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 -- 172.21.15.190:0/2159908518 wait complete. 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 Processor -- start 2024-04-22T06:39:23.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.682+0000 7f354efa3700 1 -- start start 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3548101170 0x7f3548198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3548102370 0x7f35481990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f354810d500 con 0x7f3548102370 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f354810d380 con 0x7f3548102d10 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f354810d680 con 0x7f3548101170 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:58012/0 (socket says 172.21.15.190:58012) 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 -- 172.21.15.190:0/3630401155 learned_addr learned my addr 172.21.15.190:0/3630401155 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f3547fff700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3548102370 0x7f35481990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354cd3f700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3548101170 0x7f3548198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 -- 172.21.15.190:0/3630401155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3548101170 msgr2=0x7f3548198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:23.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3548101170 0x7f3548198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 -- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3548102370 msgr2=0x7f35481990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3548102370 0x7f35481990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 -- 172.21.15.190:0/3630401155 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35481a07c0 con 0x7f3548102d10 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354d540700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481a0120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f354000cfa0 tx=0x7f354000cfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540021070 con 0x7f3548102d10 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35481a0a50 con 0x7f3548102d10 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3540007ac0 con 0x7f3548102d10 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.683+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35400052a0 con 0x7f3548102d10 2024-04-22T06:39:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.684+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35481a0f50 con 0x7f3548102d10 2024-04-22T06:39:23.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.685+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3548069730 con 0x7f3548102d10 2024-04-22T06:39:23.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.686+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3540024070 con 0x7f3548102d10 2024-04-22T06:39:23.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.686+0000 7f3545ffb700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f35380a3cc0 0x7f35380a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:23.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.687+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f354000f020 con 0x7f3548102d10 2024-04-22T06:39:23.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.687+0000 7f354cd3f700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f35380a3cc0 0x7f35380a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:23.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.687+0000 7f354cd3f700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f35380a3cc0 0x7f35380a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35481021d0 tx=0x7f353c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:23.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.690+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f354002b020 con 0x7f3548102d10 2024-04-22T06:39:23.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:23.894+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f35480fe840 con 0x7f35380a3cc0 2024-04-22T06:39:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:23 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:24.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:24.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:24.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:23 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:24.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:24.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:24.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:23 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:24 smithi190 ceph-mon[31421]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:24 smithi190 ceph-mon[31421]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:25.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:24 smithi079 ceph-mon[27946]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:25.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:24 smithi079 ceph-mon[27946]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:25.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:24 smithi193 ceph-mon[31815]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:25.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:24 smithi193 ceph-mon[31815]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/2248213244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]: dispatch 2024-04-22T06:39:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]: dispatch 2024-04-22T06:39:27.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]': finished 2024-04-22T06:39:27.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T06:39:27.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:27.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/4253754912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/2248213244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]: dispatch 2024-04-22T06:39:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]: dispatch 2024-04-22T06:39:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]': finished 2024-04-22T06:39:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T06:39:27.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:27.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/4253754912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/2248213244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]: dispatch 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]: dispatch 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57546c5-ce88-456a-b5c8-0a8ba8dcafae"}]': finished 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/4253754912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:29.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:29 smithi193 ceph-mon[31815]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:29.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:29 smithi190 ceph-mon[31421]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:29 smithi079 ceph-mon[27946]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:31.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:31 smithi193 ceph-mon[31815]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:31.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:31 smithi190 ceph-mon[31421]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:31 smithi079 ceph-mon[27946]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:33.933 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:33 smithi190 ceph-mon[31421]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:33.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:33 smithi193 ceph-mon[31815]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:33.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:33 smithi079 ceph-mon[27946]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:34.926 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:34 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:39:34.926 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:34 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:34.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:34 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:39:34.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:34 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:34 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:39:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:34 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:35.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:35 smithi190 ceph-mon[31421]: Deploying daemon osd.7 on smithi190 2024-04-22T06:39:35.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:35 smithi190 ceph-mon[31421]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:35.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:35 smithi193 ceph-mon[31815]: Deploying daemon osd.7 on smithi190 2024-04-22T06:39:35.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:35 smithi193 ceph-mon[31815]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:35.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:35 smithi079 ceph-mon[27946]: Deploying daemon osd.7 on smithi190 2024-04-22T06:39:35.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:35 smithi079 ceph-mon[27946]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:37.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:37 smithi193 ceph-mon[31815]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:37 smithi190 ceph-mon[31421]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:37.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:37 smithi079 ceph-mon[27946]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:39.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:39 smithi193 ceph-mon[31815]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:39.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:39 smithi190 ceph-mon[31421]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:39 smithi079 ceph-mon[27946]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:41.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:41.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:41.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:41.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:40 smithi190 ceph-mon[31421]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:40 smithi079 ceph-mon[27946]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:41.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:41.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:41.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:41.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:40 smithi193 ceph-mon[31815]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:43 smithi193 ceph-mon[31815]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:43.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:43 smithi190 ceph-mon[31421]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:43 smithi079 ceph-mon[27946]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:43.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:43.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:43.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:43.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.111 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.113+0000 7f3545ffb700 1 -- 172.21.15.190:0/3630401155 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f35480fe840 con 0x7f35380a3cc0 2024-04-22T06:39:44.112 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 7 on host 'smithi190' 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f35380a3cc0 msgr2=0x7f35380a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f35380a3cc0 0x7f35380a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35481021d0 tx=0x7f353c009260 comp rx=0 tx=0).stop 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 msgr2=0x7f35481a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481a0120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f354000cfa0 tx=0x7f354000cfd0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 shutdown_connections 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3548102d10 0x7f35481a0120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3548101170 0x7f3548198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f35380a3cc0 0x7f35380a6180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 --2- 172.21.15.190:0/3630401155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3548102370 0x7f35481990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 >> 172.21.15.190:0/3630401155 conn(0x7f35480fc710 msgr2=0x7f35480fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 shutdown_connections 2024-04-22T06:39:44.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:39:44.116+0000 7f354efa3700 1 -- 172.21.15.190:0/3630401155 wait complete. 2024-04-22T06:39:44.773 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:44 smithi190 ceph-mon[31421]: from='osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:44.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:44 smithi190 ceph-mon[31421]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:44.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:44.807 DEBUG:teuthology.orchestra.run.smithi190:osd.7> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T06:39:44.810 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-04-22T06:39:44.810 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:39:44.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:44 smithi193 ceph-mon[31815]: from='osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:44.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:44 smithi193 ceph-mon[31815]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:44.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:44 smithi079 ceph-mon[27946]: from='osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:44 smithi079 ceph-mon[27946]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:45.055 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:39:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:45 smithi193 ceph-mon[31815]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:45 smithi193 ceph-mon[31815]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:39:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:45 smithi193 ceph-mon[31815]: from='osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:39:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:45 smithi193 ceph-mon[31815]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T06:39:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:45 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:45 smithi193 ceph-mon[31815]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:39:45.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:45 smithi190 ceph-mon[31421]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:45.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:45 smithi190 ceph-mon[31421]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:39:45.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:45 smithi190 ceph-mon[31421]: from='osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:39:45.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:45 smithi190 ceph-mon[31421]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T06:39:45.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:45 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:45.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:45 smithi190 ceph-mon[31421]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:39:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:45 smithi079 ceph-mon[27946]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:45 smithi079 ceph-mon[27946]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:39:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:45 smithi079 ceph-mon[27946]: from='osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:39:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:45 smithi079 ceph-mon[27946]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T06:39:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:45 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:45.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:45 smithi079 ceph-mon[27946]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:39:46.276 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:39:46.131+0000 7fbd1df84700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T06:39:46.277 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:39:46.139+0000 7fbd15d6f700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:39:46.863 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:39:46.863 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:39:46.863 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-22T06:39:46.863 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-22T06:39:46.863 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131891 s, 795 MB/s 2024-04-22T06:39:46.864 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-22T06:39:46.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:46 smithi193 ceph-mon[31815]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:39:46.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:46 smithi193 ceph-mon[31815]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:39:46.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:46 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:46.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:46 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:46.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:46 smithi193 ceph-mon[31815]: from='osd.7 ' entity='osd.7' 2024-04-22T06:39:46.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-mon[31421]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:39:46.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-mon[31421]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:39:46.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:46.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:46.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:46 smithi190 ceph-mon[31421]: from='osd.7 ' entity='osd.7' 2024-04-22T06:39:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:46 smithi079 ceph-mon[27946]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:39:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:46 smithi079 ceph-mon[27946]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:39:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:46 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:46 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:46 smithi079 ceph-mon[27946]: from='osd.7 ' entity='osd.7' 2024-04-22T06:39:47.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-04-22T06:39:47.782 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:39:47.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:47 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:39:47.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:47 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:39:47.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:47 smithi193 ceph-mon[31815]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:47.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:47 smithi193 ceph-mon[31815]: osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] boot 2024-04-22T06:39:47.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:47 smithi193 ceph-mon[31815]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T06:39:47.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:47.863 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:47 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:39:47.863 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:47 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:39:47.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:47 smithi190 ceph-mon[31421]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:47.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:47 smithi190 ceph-mon[31421]: osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] boot 2024-04-22T06:39:47.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:47 smithi190 ceph-mon[31421]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T06:39:47.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:47 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:39:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:47 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:39:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:47 smithi079 ceph-mon[27946]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:47 smithi079 ceph-mon[27946]: osd.7 [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] boot 2024-04-22T06:39:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:47 smithi079 ceph-mon[27946]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T06:39:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:48.927 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:48 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:48.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:48 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:48.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:48.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:48 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:39:49.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 -- 172.21.15.193:0/1655878659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48241019a0 msgr2=0x7f4824101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:49.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 --2- 172.21.15.193:0/1655878659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48241019a0 0x7f4824101dc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4814009990 tx=0x7f4814009ca0 comp rx=0 tx=0).stop 2024-04-22T06:39:49.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 -- 172.21.15.193:0/1655878659 shutdown_connections 2024-04-22T06:39:49.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 --2- 172.21.15.193:0/1655878659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4824103540 0x7f4824109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:49.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 --2- 172.21.15.193:0/1655878659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48241019a0 0x7f4824101dc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:49.438 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 --2- 172.21.15.193:0/1655878659 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4824102ba0 0x7f4824103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 -- 172.21.15.193:0/1655878659 >> 172.21.15.193:0/1655878659 conn(0x7f48240fcf40 msgr2=0x7f48240ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.433+0000 7f482aa38700 1 -- 172.21.15.193:0/1655878659 shutdown_connections 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.434+0000 7f482aa38700 1 -- 172.21.15.193:0/1655878659 wait complete. 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.434+0000 7f482aa38700 1 Processor -- start 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.434+0000 7f482aa38700 1 -- start start 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.434+0000 7f482aa38700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48241019a0 0x7f4824198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4824102ba0 0x7f4824199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 0x7f48241a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f482410d430 con 0x7f48241019a0 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f482410d2b0 con 0x7f4824102ba0 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f482410d5b0 con 0x7f4824103540 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 0x7f48241a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 0x7f48241a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:34306/0 (socket says 172.21.15.193:34306) 2024-04-22T06:39:49.439 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 -- 172.21.15.193:0/2348319187 learned_addr learned my addr 172.21.15.193:0/2348319187 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4823fff700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48241019a0 0x7f4824198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f48237fe700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4824102ba0 0x7f4824199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 -- 172.21.15.193:0/2348319187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4824102ba0 msgr2=0x7f4824199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4824102ba0 0x7f4824199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 -- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48241019a0 msgr2=0x7f4824198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48241019a0 0x7f4824198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 -- 172.21.15.193:0/2348319187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48241a0870 con 0x7f4824103540 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f4828fd5700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 0x7f48241a01a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f481c00ef30 tx=0x7f481c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c0092e0 con 0x7f4824103540 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f481c004bb0 con 0x7f4824103540 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48241a0b00 con 0x7f4824103540 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c004280 con 0x7f4824103540 2024-04-22T06:39:49.440 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.435+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48241a1040 con 0x7f4824103540 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.436+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4824069730 con 0x7f4824103540 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.437+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f481c01b030 con 0x7f4824103540 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.438+0000 7f48217fa700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f480c0a3c40 0x7f480c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.438+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f481c0cfd40 con 0x7f4824103540 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.438+0000 7f4823fff700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f480c0a3c40 0x7f480c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.439+0000 7f4823fff700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f480c0a3c40 0x7f480c0a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f482419a100 tx=0x7f481401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:49.441 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.442+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f481c022020 con 0x7f4824103540 2024-04-22T06:39:49.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:39:49.647+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f482406ae60 con 0x7f480c0a3c40 2024-04-22T06:39:49.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi190 2024-04-22T06:39:49.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi190 to 5696M 2024-04-22T06:39:49.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:49.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:39:49.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:39:49.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:49.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:49.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:49 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi190 2024-04-22T06:39:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi190 to 5696M 2024-04-22T06:39:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:39:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:39:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:49.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:49.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:49 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi190 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi190 to 5696M 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:49 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:50.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:50 smithi193 ceph-mon[31815]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:50.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:50 smithi193 ceph-mon[31815]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:39:50.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:50 smithi190 ceph-mon[31421]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:50.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:50 smithi190 ceph-mon[31421]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:39:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:50 smithi079 ceph-mon[27946]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:50 smithi079 ceph-mon[27946]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:39:51.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:51 smithi193 ceph-mon[31815]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:51.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:51 smithi190 ceph-mon[31421]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:51 smithi079 ceph-mon[27946]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:52.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:52 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/4001156509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]: dispatch 2024-04-22T06:39:52.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:52 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]: dispatch 2024-04-22T06:39:52.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:52 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]': finished 2024-04-22T06:39:52.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:52 smithi190 ceph-mon[31421]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T06:39:52.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:39:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:52 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/4001156509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]: dispatch 2024-04-22T06:39:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:52 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]: dispatch 2024-04-22T06:39:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:52 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]': finished 2024-04-22T06:39:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:52 smithi079 ceph-mon[27946]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T06:39:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:39:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:52 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/4001156509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]: dispatch 2024-04-22T06:39:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:52 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]: dispatch 2024-04-22T06:39:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:52 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6b199d-ad3e-4ac8-815c-4b30b3a7e335"}]': finished 2024-04-22T06:39:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:52 smithi193 ceph-mon[31815]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T06:39:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:39:53.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:53 smithi190 ceph-mon[31421]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:53.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:53 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/309223430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:53 smithi079 ceph-mon[27946]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:53 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/309223430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:53 smithi193 ceph-mon[31815]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:53 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/309223430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:55 smithi190 ceph-mon[31421]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:55 smithi079 ceph-mon[27946]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:56.126 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:55 smithi193 ceph-mon[31815]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:57.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:57 smithi190 ceph-mon[31421]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:57.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:57 smithi079 ceph-mon[27946]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:58.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:57 smithi193 ceph-mon[31815]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:59.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:39:59 smithi079 ceph-mon[27946]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:00.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:39:59 smithi190 ceph-mon[31421]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:00.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:39:59 smithi193 ceph-mon[31815]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:00.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:00 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:40:00.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:00 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:00.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:00 smithi193 ceph-mon[31815]: overall HEALTH_OK 2024-04-22T06:40:00.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:00 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:40:00.971 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:00 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:00.971 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:00 smithi190 ceph-mon[31421]: overall HEALTH_OK 2024-04-22T06:40:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:00 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:40:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:00 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:00.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:00 smithi079 ceph-mon[27946]: overall HEALTH_OK 2024-04-22T06:40:01.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:01 smithi193 ceph-mon[31815]: Deploying daemon osd.8 on smithi193 2024-04-22T06:40:01.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:01 smithi193 ceph-mon[31815]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:01.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:01 smithi079 ceph-mon[27946]: Deploying daemon osd.8 on smithi193 2024-04-22T06:40:01.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:01 smithi079 ceph-mon[27946]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:02.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:01 smithi190 ceph-mon[31421]: Deploying daemon osd.8 on smithi193 2024-04-22T06:40:02.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:01 smithi190 ceph-mon[31421]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:03.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:03 smithi079 ceph-mon[27946]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:04.022 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:03 smithi193 ceph-mon[31815]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:04.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:03 smithi190 ceph-mon[31421]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:05.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:05 smithi193 ceph-mon[31815]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:05 smithi079 ceph-mon[27946]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:06.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:05 smithi190 ceph-mon[31421]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:07.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:07.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:07.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:07.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:07.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:07.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:07.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:07 smithi193 ceph-mon[31815]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:08.141 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:07 smithi079 ceph-mon[27946]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:07 smithi190 ceph-mon[31421]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:09.215 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-04-22T06:40:09.215 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.216+0000 7f48217fa700 1 -- 172.21.15.193:0/2348319187 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f482406ae60 con 0x7f480c0a3c40 2024-04-22T06:40:09.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.218+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f480c0a3c40 msgr2=0x7f480c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.218+0000 7f482aa38700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f480c0a3c40 0x7f480c0a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f482419a100 tx=0x7f481401a040 comp rx=0 tx=0).stop 2024-04-22T06:40:09.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 msgr2=0x7f48241a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 0x7f48241a01a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f481c00ef30 tx=0x7f481c00c3a0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 shutdown_connections 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4824103540 0x7f48241a01a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f480c0a3c40 0x7f480c0a6100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48241019a0 0x7f4824198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 --2- 172.21.15.193:0/2348319187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4824102ba0 0x7f4824199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 >> 172.21.15.193:0/2348319187 conn(0x7f48240fcf40 msgr2=0x7f482406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 shutdown_connections 2024-04-22T06:40:09.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:40:09.219+0000 7f482aa38700 1 -- 172.21.15.193:0/2348319187 wait complete. 2024-04-22T06:40:09.818 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:09 smithi193 ceph-mon[31815]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:09.818 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:09 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:09.818 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:09 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:09.818 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:09 smithi193 ceph-mon[31815]: from='osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:09.818 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:09 smithi193 ceph-mon[31815]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:09.858 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.8.service 2024-04-22T06:40:09.861 INFO:tasks.cephadm:Deploying osd.9 on smithi079 with /dev/vg_nvme/lv_1... 2024-04-22T06:40:09.861 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:40:10.099 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:40:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:09 smithi190 ceph-mon[31421]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:09 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:09 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:09 smithi190 ceph-mon[31421]: from='osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:09 smithi190 ceph-mon[31421]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:10.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:09 smithi079 ceph-mon[27946]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:10.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:09 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:10.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:09 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:10.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:09 smithi079 ceph-mon[27946]: from='osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:10.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:09 smithi079 ceph-mon[27946]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:11.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:11.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:11.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:40:11.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T06:40:11.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:11.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: from='osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:40:11.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:40:11.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:10 smithi193 ceph-mon[31815]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: from='osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:40:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:40:11.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:10 smithi190 ceph-mon[31421]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: from='osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:40:11.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:10 smithi079 ceph-mon[27946]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:11.850 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:40:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:40:11.643+0000 7f74ed8cf700 -1 osd.8 0 waiting for initial osdmap 2024-04-22T06:40:11.850 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:40:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:40:11.654+0000 7f74e4eb9700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:40:12.247 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:40:12.247 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:40:12.247 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-04-22T06:40:12.247 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-04-22T06:40:12.247 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113737 s, 922 MB/s 2024-04-22T06:40:12.247 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-04-22T06:40:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:12 smithi079 ceph-mon[27946]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:40:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:12 smithi079 ceph-mon[27946]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:40:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:12 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:12 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:12 smithi079 ceph-mon[27946]: from='osd.8 ' entity='osd.8' 2024-04-22T06:40:12.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:12 smithi193 ceph-mon[31815]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:40:12.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:12 smithi193 ceph-mon[31815]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:40:12.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:12 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:12.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:12 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:12.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:12 smithi193 ceph-mon[31815]: from='osd.8 ' entity='osd.8' 2024-04-22T06:40:12.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:12 smithi190 ceph-mon[31421]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:40:12.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:12 smithi190 ceph-mon[31421]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:40:12.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:12 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:12.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:12 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:12.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:12 smithi190 ceph-mon[31421]: from='osd.8 ' entity='osd.8' 2024-04-22T06:40:12.891 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_1 2024-04-22T06:40:13.128 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:40:13.335 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:13 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:40:13.335 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:13 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:40:13.335 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:13 smithi193 ceph-mon[31815]: osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] boot 2024-04-22T06:40:13.335 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:13 smithi193 ceph-mon[31815]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T06:40:13.336 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:13.336 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:13 smithi193 ceph-mon[31815]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:13 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:40:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:13 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:40:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:13 smithi079 ceph-mon[27946]: osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] boot 2024-04-22T06:40:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:13 smithi079 ceph-mon[27946]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T06:40:13.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:13.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:13 smithi079 ceph-mon[27946]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:13 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:40:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:13 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:40:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:13 smithi190 ceph-mon[31421]: osd.8 [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] boot 2024-04-22T06:40:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:13 smithi190 ceph-mon[31421]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T06:40:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:13.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:13 smithi190 ceph-mon[31421]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T06:40:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi193 2024-04-22T06:40:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi193 to 5696M 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:14 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi193 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi193 to 5696M 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:14.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:14.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:14.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:14.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:14.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:14 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi193 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi193 to 5696M 2024-04-22T06:40:14.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:14.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:14 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 -- 172.21.15.79:0/447829598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 msgr2=0x7fc800109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 --2- 172.21.15.79:0/447829598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800109ee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800b0d0 tx=0x7fc7f800b3e0 comp rx=0 tx=0).stop 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 -- 172.21.15.79:0/447829598 shutdown_connections 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 --2- 172.21.15.79:0/447829598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800109ee0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 --2- 172.21.15.79:0/447829598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 0x7fc800103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 --2- 172.21.15.79:0/447829598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc800101c80 0x7fc8001020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 -- 172.21.15.79:0/447829598 >> 172.21.15.79:0/447829598 conn(0x7fc8000fc6b0 msgr2=0x7fc8000feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.762+0000 7fc807ca9700 1 -- 172.21.15.79:0/447829598 shutdown_connections 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 -- 172.21.15.79:0/447829598 wait complete. 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 Processor -- start 2024-04-22T06:40:14.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 -- start start 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc800101c80 0x7fc800197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 0x7fc80019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010d4a0 con 0x7fc800102620 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010d320 con 0x7fc800101c80 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.763+0000 7fc807ca9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010d620 con 0x7fc800102f40 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48796/0 (socket says 172.21.15.79:48796) 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 -- 172.21.15.79:0/4047607915 learned_addr learned my addr 172.21.15.79:0/4047607915 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:40:14.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc806246700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 0x7fc80019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805a45700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc800101c80 0x7fc800197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 -- 172.21.15.79:0/4047607915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 msgr2=0x7fc80019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 0x7fc80019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 -- 172.21.15.79:0/4047607915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc800101c80 msgr2=0x7fc800197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc800101c80 0x7fc800197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 -- 172.21.15.79:0/4047607915 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80019f6f0 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc806246700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 0x7fc80019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc805244700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800197fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b560 tx=0x7fc7fc00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc015020 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80019f9e0 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7fc011ce0 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc010da0 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.764+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80019fff0 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.765+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc800069730 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.766+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc7fc018020 con 0x7fc800102620 2024-04-22T06:40:14.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.767+0000 7fc7f2ffd700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc7ec0a3c40 0x7fc7ec0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:14.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.767+0000 7fc805a45700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc7ec0a3c40 0x7fc7ec0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:14.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.767+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7fc7fc0cf7a0 con 0x7fc800102620 2024-04-22T06:40:14.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.767+0000 7fc805a45700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc7ec0a3c40 0x7fc7ec0a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc80006af70 tx=0x7fc7f4005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:14.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.771+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7fc01f020 con 0x7fc800102620 2024-04-22T06:40:14.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:14.981+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc800100b70 con 0x7fc7ec0a3c40 2024-04-22T06:40:15.231 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:15 smithi079 ceph-mon[27946]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T06:40:15.231 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:15 smithi079 ceph-mon[27946]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:15.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:15.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:15.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:15.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:15 smithi193 ceph-mon[31815]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T06:40:15.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:15 smithi193 ceph-mon[31815]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:15.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:15.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:15.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:15.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:15 smithi190 ceph-mon[31421]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T06:40:15.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:15 smithi190 ceph-mon[31421]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:15.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:15.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:15.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:16.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:16 smithi079 ceph-mon[27946]: from='client.14391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:16.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:16 smithi079 ceph-mon[27946]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:40:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:16 smithi193 ceph-mon[31815]: from='client.14391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:16 smithi193 ceph-mon[31815]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:40:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:16 smithi190 ceph-mon[31421]: from='client.14391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:16 smithi190 ceph-mon[31421]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:40:17.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:17 smithi079 ceph-mon[27946]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:40:17.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:17 smithi079 ceph-mon[27946]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:17.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:17 smithi193 ceph-mon[31815]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:40:17.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:17 smithi193 ceph-mon[31815]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:17.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:17 smithi190 ceph-mon[31421]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:40:17.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:17 smithi190 ceph-mon[31421]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:18.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1067178941' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a68e58f-14cc-4c45-a45b-5cd0505bff3f"}]: dispatch 2024-04-22T06:40:18.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1067178941' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a68e58f-14cc-4c45-a45b-5cd0505bff3f"}]': finished 2024-04-22T06:40:18.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:18 smithi079 ceph-mon[27946]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T06:40:18.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:18.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1067178941' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a68e58f-14cc-4c45-a45b-5cd0505bff3f"}]: dispatch 2024-04-22T06:40:18.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1067178941' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a68e58f-14cc-4c45-a45b-5cd0505bff3f"}]': finished 2024-04-22T06:40:18.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:18 smithi193 ceph-mon[31815]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T06:40:18.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:18.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1067178941' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a68e58f-14cc-4c45-a45b-5cd0505bff3f"}]: dispatch 2024-04-22T06:40:18.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1067178941' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a68e58f-14cc-4c45-a45b-5cd0505bff3f"}]': finished 2024-04-22T06:40:18.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:18 smithi190 ceph-mon[31421]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T06:40:18.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:19.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:19 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/515443670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:19.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:19 smithi193 ceph-mon[31815]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-22T06:40:19.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:19 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/515443670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:19.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:19 smithi190 ceph-mon[31421]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-22T06:40:19.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:19 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/515443670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:19.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:19 smithi079 ceph-mon[27946]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-22T06:40:21.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:21 smithi193 ceph-mon[31815]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:40:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:21 smithi190 ceph-mon[31421]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:40:21.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:21 smithi079 ceph-mon[27946]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:40:23.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:23 smithi193 ceph-mon[31815]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-22T06:40:23.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:23 smithi190 ceph-mon[31421]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-22T06:40:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:23 smithi079 ceph-mon[27946]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-22T06:40:25.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:25 smithi193 ceph-mon[31815]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-22T06:40:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:25 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:40:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:25 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:25 smithi190 ceph-mon[31421]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-22T06:40:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:25 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:40:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:25 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:25 smithi079 ceph-mon[27946]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-22T06:40:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:25 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:40:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:25 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:26.942 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:26 smithi193 ceph-mon[31815]: Deploying daemon osd.9 on smithi079 2024-04-22T06:40:26.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:26 smithi190 ceph-mon[31421]: Deploying daemon osd.9 on smithi079 2024-04-22T06:40:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:26 smithi079 ceph-mon[27946]: Deploying daemon osd.9 on smithi079 2024-04-22T06:40:27.916 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:27 smithi079 ceph-mon[27946]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:40:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:27 smithi193 ceph-mon[31815]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:40:27.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:27 smithi190 ceph-mon[31421]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:40:29.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:29 smithi079 ceph-mon[27946]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:29.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:29 smithi193 ceph-mon[31815]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:29.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:29 smithi190 ceph-mon[31421]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:31.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:31 smithi079 ceph-mon[27946]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:31.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:31.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:31.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:31.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:31 smithi193 ceph-mon[31815]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:31.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:31.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:31.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:31.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:31 smithi190 ceph-mon[31421]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:31.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:31.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:31.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:33.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:33 smithi193 ceph-mon[31815]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:33.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:33 smithi190 ceph-mon[31421]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:33.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:33 smithi079 ceph-mon[27946]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.238+0000 7fc7f2ffd700 1 -- 172.21.15.79:0/4047607915 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc800100b70 con 0x7fc7ec0a3c40 2024-04-22T06:40:35.238 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 9 on host 'smithi079' 2024-04-22T06:40:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc7ec0a3c40 msgr2=0x7fc7ec0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc7ec0a3c40 0x7fc7ec0a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc80006af70 tx=0x7fc7f4005cf0 comp rx=0 tx=0).stop 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 msgr2=0x7fc800197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800197fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b560 tx=0x7fc7fc00b920 comp rx=0 tx=0).stop 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 shutdown_connections 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc800102f40 0x7fc80019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc800101c80 0x7fc800197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc7ec0a3c40 0x7fc7ec0a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 --2- 172.21.15.79:0/4047607915 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc800102620 0x7fc800197fd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 >> 172.21.15.79:0/4047607915 conn(0x7fc8000fc6b0 msgr2=0x7fc800105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 shutdown_connections 2024-04-22T06:40:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:40:35.241+0000 7fc807ca9700 1 -- 172.21.15.79:0/4047607915 wait complete. 2024-04-22T06:40:35.886 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:35 smithi079 ceph-mon[27946]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:35.886 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:35 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:35.886 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:35 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:35.886 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:35 smithi079 ceph-mon[27946]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:40:35.913 DEBUG:teuthology.orchestra.run.smithi079:osd.9> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.9.service 2024-04-22T06:40:35.915 INFO:tasks.cephadm:Deploying osd.10 on smithi190 with /dev/vg_nvme/lv_1... 2024-04-22T06:40:35.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:40:35.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:35 smithi190 ceph-mon[31421]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:35.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:35 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:35.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:35 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:35.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:35 smithi190 ceph-mon[31421]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:40:36.164 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:40:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:35 smithi193 ceph-mon[31815]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:35 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:35 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:35 smithi193 ceph-mon[31815]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-mon[27946]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:37.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:37.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:37.369 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:40:37.369 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T06:40:37.369 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:40:37.369 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:37.369 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:37 smithi190 ceph-mon[31421]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:37.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:37.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:37.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:40:37.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T06:40:37.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-04-22T06:40:37.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:37.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:37 smithi193 ceph-mon[31815]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:37.982 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:40:37.673+0000 7fc961b41700 -1 osd.9 0 waiting for initial osdmap 2024-04-22T06:40:37.982 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:40:37 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:40:37.685+0000 7fc95992c700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:40:38.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:38 smithi079 ceph-mon[27946]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:40:38.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:38 smithi079 ceph-mon[27946]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T06:40:38.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:38 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:38.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:38 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:38.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:38 smithi079 ceph-mon[27946]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' 2024-04-22T06:40:38.301 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:40:38.301 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:40:38.301 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-22T06:40:38.301 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-22T06:40:38.301 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112956 s, 928 MB/s 2024-04-22T06:40:38.301 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-22T06:40:38.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:38 smithi190 ceph-mon[31421]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:40:38.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:38 smithi190 ceph-mon[31421]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T06:40:38.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:38 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:38.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:38 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:38.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:38 smithi190 ceph-mon[31421]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' 2024-04-22T06:40:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:38 smithi193 ceph-mon[31815]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-04-22T06:40:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:38 smithi193 ceph-mon[31815]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T06:40:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:38 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:38 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:38.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:38 smithi193 ceph-mon[31815]: from='osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131]' entity='osd.9' 2024-04-22T06:40:39.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2024-04-22T06:40:39.291 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] boot 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T06:40:39.450 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:39.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:40:39.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:40:39.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:39.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:39.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] boot 2024-04-22T06:40:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T06:40:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:40:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:40:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:39.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:39.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: osd.9 [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] boot 2024-04-22T06:40:39.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T06:40:39.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:40.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 -- 172.21.15.190:0/3921683997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 msgr2=0x7f76141041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 --2- 172.21.15.190:0/3921683997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f76141041a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7604009990 tx=0x7f7604009ca0 comp rx=0 tx=0).stop 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 -- 172.21.15.190:0/3921683997 shutdown_connections 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 --2- 172.21.15.190:0/3921683997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76141046e0 0x7f761410af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 --2- 172.21.15.190:0/3921683997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7614100d60 0x7f7614101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 --2- 172.21.15.190:0/3921683997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f76141041a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 -- 172.21.15.190:0/3921683997 >> 172.21.15.190:0/3921683997 conn(0x7f76140fc6a0 msgr2=0x7f76140feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 -- 172.21.15.190:0/3921683997 shutdown_connections 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.976+0000 7f7619a8c700 1 -- 172.21.15.190:0/3921683997 wait complete. 2024-04-22T06:40:40.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 Processor -- start 2024-04-22T06:40:40.978 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 -- start start 2024-04-22T06:40:40.978 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7614100d60 0x7f7614198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:40.978 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f7614199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76141046e0 0x7f76141a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f761410e520 con 0x7f76141046e0 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f761410e3a0 con 0x7f7614103d40 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7619a8c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f761410e6a0 con 0x7f7614100d60 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7612ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f7614199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7612ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f7614199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:40876/0 (socket says 172.21.15.190:40876) 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f7612ffd700 1 -- 172.21.15.190:0/2314706984 learned_addr learned my addr 172.21.15.190:0/2314706984 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.977+0000 7f76137fe700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7614100d60 0x7f7614198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7613fff700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76141046e0 0x7f76141a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7612ffd700 1 -- 172.21.15.190:0/2314706984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7614100d60 msgr2=0x7f7614198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7612ffd700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7614100d60 0x7f7614198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7612ffd700 1 -- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76141046e0 msgr2=0x7f76141a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7612ffd700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76141046e0 0x7f76141a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:40.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7612ffd700 1 -- 172.21.15.190:0/2314706984 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76141a0820 con 0x7f7614103d40 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7612ffd700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f7614199100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7604006010 tx=0x7f76040047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7604027070 con 0x7f7614103d40 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76141a0ab0 con 0x7f7614103d40 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7604004080 con 0x7f7614103d40 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7604022dc0 con 0x7f7614103d40 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.978+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76141a0f60 con 0x7f7614103d40 2024-04-22T06:40:40.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.979+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7614069730 con 0x7f7614103d40 2024-04-22T06:40:40.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.981+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f760402a070 con 0x7f7614103d40 2024-04-22T06:40:40.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.981+0000 7f7610ff9700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75fc0a3c40 0x7f75fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:40.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.982+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f76040d1230 con 0x7f7614103d40 2024-04-22T06:40:40.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.982+0000 7f76137fe700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75fc0a3c40 0x7f75fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:40.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.982+0000 7f76137fe700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75fc0a3c40 0x7f75fc0a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f761406b010 tx=0x7f760800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:40.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:40.985+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7604031070 con 0x7f7614103d40 2024-04-22T06:40:41.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi079 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi079 to 4272M 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:41.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:41.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:41.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T06:40:41.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:40 smithi193 ceph-mon[31815]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:41.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:40:41.195+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7614062f40 con 0x7f75fc0a3c40 2024-04-22T06:40:41.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi079 2024-04-22T06:40:41.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi079 to 4272M 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T06:40:41.195 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:40 smithi190 ceph-mon[31421]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi079 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi079 to 4272M 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T06:40:41.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:40 smithi079 ceph-mon[27946]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:42.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:41 smithi190 ceph-mon[31421]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:42.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:42.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:42.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:41 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:42.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:41 smithi193 ceph-mon[31815]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:42.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:42.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:42.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:41 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:42.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:41 smithi079 ceph-mon[27946]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:42.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:42.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:42.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:41 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:43.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:42 smithi190 ceph-mon[31421]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:42 smithi193 ceph-mon[31815]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:43.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:42 smithi079 ceph-mon[27946]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:44.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:44 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/2215676749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]: dispatch 2024-04-22T06:40:44.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:44 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]: dispatch 2024-04-22T06:40:44.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:44 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]': finished 2024-04-22T06:40:44.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:44 smithi190 ceph-mon[31421]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T06:40:44.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:40:44.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:44 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/2215676749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]: dispatch 2024-04-22T06:40:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:44 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]: dispatch 2024-04-22T06:40:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:44 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]': finished 2024-04-22T06:40:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:44 smithi193 ceph-mon[31815]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T06:40:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:40:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:44 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/2215676749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]: dispatch 2024-04-22T06:40:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:44 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]: dispatch 2024-04-22T06:40:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:44 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c7a37b4-3474-4e05-b976-61c5b99dbb98"}]': finished 2024-04-22T06:40:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:44 smithi079 ceph-mon[27946]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T06:40:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:40:45.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:45 smithi193 ceph-mon[31815]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:45 smithi193 ceph-mon[31815]: from='client.? 172.21.15.190:0/1276213518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:45 smithi190 ceph-mon[31421]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:45 smithi190 ceph-mon[31421]: from='client.? 172.21.15.190:0/1276213518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:45 smithi079 ceph-mon[27946]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:45 smithi079 ceph-mon[27946]: from='client.? 172.21.15.190:0/1276213518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:47.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:47 smithi193 ceph-mon[31815]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:47.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:47 smithi190 ceph-mon[31421]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:47 smithi079 ceph-mon[27946]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:49.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:49 smithi193 ceph-mon[31815]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:49.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:49 smithi190 ceph-mon[31421]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:49 smithi079 ceph-mon[27946]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:51.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:51 smithi193 ceph-mon[31815]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:51.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:51 smithi190 ceph-mon[31421]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:51 smithi079 ceph-mon[27946]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:40:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:52 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:52.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:52 smithi193 ceph-mon[31815]: Deploying daemon osd.10 on smithi190 2024-04-22T06:40:52.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:40:52.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:52 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:52.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:52 smithi190 ceph-mon[31421]: Deploying daemon osd.10 on smithi190 2024-04-22T06:40:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:40:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:52 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:52 smithi079 ceph-mon[27946]: Deploying daemon osd.10 on smithi190 2024-04-22T06:40:53.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:53 smithi190 ceph-mon[31421]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:53.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:53 smithi193 ceph-mon[31815]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:53 smithi079 ceph-mon[27946]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:55.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:55 smithi193 ceph-mon[31815]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:55.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:55 smithi190 ceph-mon[31421]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:55 smithi079 ceph-mon[27946]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:57.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:57 smithi190 ceph-mon[31421]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:57.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:57 smithi079 ceph-mon[27946]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:58.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:57 smithi193 ceph-mon[31815]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:58.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:58 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:58.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:58 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:58.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:58 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:58.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:58 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:58.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:58 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:58.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:58 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:59.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:58 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:59.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:58 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:59.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:58 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:40:59.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:40:59 smithi190 ceph-mon[31421]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:59.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:40:59 smithi079 ceph-mon[27946]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:00.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:40:59 smithi193 ceph-mon[31815]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:01.764 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 10 on host 'smithi190' 2024-04-22T06:41:01.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.766+0000 7f7610ff9700 1 -- 172.21.15.190:0/2314706984 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7614062f40 con 0x7f75fc0a3c40 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75fc0a3c40 msgr2=0x7f75fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75fc0a3c40 0x7f75fc0a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f761406b010 tx=0x7f760800b410 comp rx=0 tx=0).stop 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 msgr2=0x7f7614199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f7614199100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7604006010 tx=0x7f76040047b0 comp rx=0 tx=0).stop 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 shutdown_connections 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75fc0a3c40 0x7f75fc0a6100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76141046e0 0x7f76141a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7614100d60 0x7f7614198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:01.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 --2- 172.21.15.190:0/2314706984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7614103d40 0x7f7614199100 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:01.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 >> 172.21.15.190:0/2314706984 conn(0x7f76140fc6a0 msgr2=0x7f7614107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:01.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 shutdown_connections 2024-04-22T06:41:01.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-22T06:41:01.769+0000 7f7619a8c700 1 -- 172.21.15.190:0/2314706984 wait complete. 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:01 smithi190 ceph-mon[31421]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:02.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:01 smithi193 ceph-mon[31815]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:02.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:02.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:01 smithi079 ceph-mon[27946]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:02.469 DEBUG:teuthology.orchestra.run.smithi190:osd.10> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.10.service 2024-04-22T06:41:02.471 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-04-22T06:41:02.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:41:02.716 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:02.839 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:02 smithi193 ceph-mon[31815]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:41:02.962 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:02.963 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:02 smithi190 ceph-mon[31421]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:41:03.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:03.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:03.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:03.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:41:03.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:41:03.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T06:41:03.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:03.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:02 smithi079 ceph-mon[27946]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-22T06:41:03.453 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:41:02.999+0000 7fc855b73700 -1 osd.10 0 waiting for initial osdmap 2024-04-22T06:41:03.453 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:41:03.012+0000 7fc851966700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:41:04.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:03 smithi193 ceph-mon[31815]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:04.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:03 smithi193 ceph-mon[31815]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:41:04.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:03 smithi193 ceph-mon[31815]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T06:41:04.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:03 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:04.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:03 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:04.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-mon[31421]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:04.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-mon[31421]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:41:04.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-mon[31421]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T06:41:04.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:04.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:03 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:03 smithi079 ceph-mon[27946]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:03 smithi079 ceph-mon[27946]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-22T06:41:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:03 smithi079 ceph-mon[27946]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T06:41:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:03 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:03 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112627 s, 931 MB/s 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 2024-04-22T06:41:04.871 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] boot 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T06:41:05.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:04 smithi190 ceph-mon[31421]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] boot 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T06:41:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:04 smithi193 ceph-mon[31815]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: osd.10 [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] boot 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T06:41:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:04 smithi079 ceph-mon[27946]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:05.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-04-22T06:41:05.751 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:41:06.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi190 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi190 to 4272M 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:06.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:06 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi190 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi190 to 4272M 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:06 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi190 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi190 to 4272M 2024-04-22T06:41:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:06.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:06.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:06.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:06 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:07.396 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 -- 172.21.15.193:0/4012884313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 msgr2=0x7f1764105970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:07.396 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4012884313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764105970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1760009990 tx=0x7f1760009ca0 comp rx=0 tx=0).stop 2024-04-22T06:41:07.396 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 -- 172.21.15.193:0/4012884313 shutdown_connections 2024-04-22T06:41:07.396 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4012884313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17641087e0 0x7f176410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:07.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4012884313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764105970 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:07.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4012884313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 0x7f17641082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:07.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.397+0000 7f176b8d3700 1 -- 172.21.15.193:0/4012884313 >> 172.21.15.193:0/4012884313 conn(0x7f17640fcf30 msgr2=0x7f17640ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:07.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 -- 172.21.15.193:0/4012884313 shutdown_connections 2024-04-22T06:41:07.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 -- 172.21.15.193:0/4012884313 wait complete. 2024-04-22T06:41:07.398 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 Processor -- start 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 -- start start 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 0x7f1764101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17641087e0 0x7f1764101a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f176410e610 con 0x7f1764105eb0 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f176410e490 con 0x7f17641087e0 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176b8d3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f176410e790 con 0x7f1764103580 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.398+0000 7f176966f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764102ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:53656/0 (socket says 172.21.15.193:53656) 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 -- 172.21.15.193:0/4271223858 learned_addr learned my addr 172.21.15.193:0/4271223858 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f1768e6e700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 0x7f1764101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 -- 172.21.15.193:0/4271223858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17641087e0 msgr2=0x7f1764101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17641087e0 0x7f1764101a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:07.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 -- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 msgr2=0x7f1764101530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 0x7f1764101530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 -- 172.21.15.193:0/4271223858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1764102320 con 0x7f1764103580 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f1768e6e700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 0x7f1764101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176966f700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764102ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17600036f0 tx=0x7f17600039f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760027070 con 0x7f1764103580 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1760022470 con 0x7f1764103580 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17641a2e10 con 0x7f1764103580 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176000f6c0 con 0x7f1764103580 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.399+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17641a3230 con 0x7f1764103580 2024-04-22T06:41:07.400 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.400+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f176402df80 con 0x7f1764103580 2024-04-22T06:41:07.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.401+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f176002a070 con 0x7f1764103580 2024-04-22T06:41:07.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.402+0000 7f175a7fc700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f17500a3c40 0x7f17500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:07.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.402+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f17600d11d0 con 0x7f1764103580 2024-04-22T06:41:07.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.402+0000 7f1768e6e700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f17500a3c40 0x7f17500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:07.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.402+0000 7f1768e6e700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f17500a3c40 0x7f17500a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f17540099d0 tx=0x7f1754008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:07.404 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.406+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1760013070 con 0x7f1764103580 2024-04-22T06:41:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:07 smithi193 ceph-mon[31815]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T06:41:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:07 smithi193 ceph-mon[31815]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:07 smithi193 ceph-mon[31815]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:41:07.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:07 smithi190 ceph-mon[31421]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T06:41:07.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:07 smithi190 ceph-mon[31421]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:07.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:07 smithi190 ceph-mon[31421]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:41:07.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:07 smithi079 ceph-mon[27946]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T06:41:07.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:07 smithi079 ceph-mon[27946]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:07.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:07 smithi079 ceph-mon[27946]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:41:07.610 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:07.612+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1764062e80 con 0x7f17500a3c40 2024-04-22T06:41:08.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:08 smithi193 ceph-mon[31815]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:08.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:08 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:41:08.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:08 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:41:08.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:08 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:08.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:08 smithi190 ceph-mon[31421]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:08.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:08 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:41:08.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:08 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:41:08.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:08 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:08 smithi079 ceph-mon[27946]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:08 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:41:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:08 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:41:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:08 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:09.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:09 smithi193 ceph-mon[31815]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:41:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:09 smithi190 ceph-mon[31421]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:41:09.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:09 smithi079 ceph-mon[27946]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T06:41:10.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:10 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/206083572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]: dispatch 2024-04-22T06:41:10.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:10 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]: dispatch 2024-04-22T06:41:10.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:10 smithi193 ceph-mon[31815]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]': finished 2024-04-22T06:41:10.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:10 smithi193 ceph-mon[31815]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T06:41:10.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:10.872 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:10 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/206083572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]: dispatch 2024-04-22T06:41:10.872 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:10 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]: dispatch 2024-04-22T06:41:10.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:10 smithi190 ceph-mon[31421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]': finished 2024-04-22T06:41:10.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:10 smithi190 ceph-mon[31421]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T06:41:10.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/206083572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]: dispatch 2024-04-22T06:41:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:10 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]: dispatch 2024-04-22T06:41:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:10 smithi079 ceph-mon[27946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "343a8050-22fd-4408-a1d5-e058b623348d"}]': finished 2024-04-22T06:41:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:10 smithi079 ceph-mon[27946]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T06:41:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:11.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:11 smithi193 ceph-mon[31815]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-22T06:41:11.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:11 smithi193 ceph-mon[31815]: from='client.? 172.21.15.193:0/3644883554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:41:11.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:11 smithi190 ceph-mon[31421]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-22T06:41:11.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:11 smithi190 ceph-mon[31421]: from='client.? 172.21.15.193:0/3644883554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:41:11.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:11 smithi079 ceph-mon[27946]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-22T06:41:11.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:11 smithi079 ceph-mon[27946]: from='client.? 172.21.15.193:0/3644883554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:41:13.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:13 smithi193 ceph-mon[31815]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T06:41:13.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:13 smithi190 ceph-mon[31421]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T06:41:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:13 smithi079 ceph-mon[27946]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T06:41:15.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:15 smithi193 ceph-mon[31815]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:41:15.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:15 smithi190 ceph-mon[31421]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:41:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:15 smithi079 ceph-mon[27946]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:41:17.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:17 smithi193 ceph-mon[31815]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:17 smithi190 ceph-mon[31421]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:17.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:17 smithi079 ceph-mon[27946]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:18.813 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:41:18.813 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:41:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:41:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:19.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:19 smithi193 ceph-mon[31815]: Deploying daemon osd.11 on smithi193 2024-04-22T06:41:19.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:19 smithi193 ceph-mon[31815]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:19.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:19 smithi190 ceph-mon[31421]: Deploying daemon osd.11 on smithi193 2024-04-22T06:41:19.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:19 smithi190 ceph-mon[31421]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:19.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:19 smithi079 ceph-mon[27946]: Deploying daemon osd.11 on smithi193 2024-04-22T06:41:19.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:19 smithi079 ceph-mon[27946]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:21.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:21 smithi190 ceph-mon[31421]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:21.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:21 smithi193 ceph-mon[31815]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:21.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:21 smithi079 ceph-mon[27946]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:23 smithi190 ceph-mon[31421]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:23 smithi079 ceph-mon[27946]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:24.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:23 smithi193 ceph-mon[31815]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:25.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:25.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:24 smithi193 ceph-mon[31815]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:25.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:25.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:24 smithi190 ceph-mon[31421]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:25.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:25.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:25.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:25.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:24 smithi079 ceph-mon[27946]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:27.732 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.733+0000 7f175a7fc700 1 -- 172.21.15.193:0/4271223858 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1764062e80 con 0x7f17500a3c40 2024-04-22T06:41:27.732 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-04-22T06:41:27.734 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f17500a3c40 msgr2=0x7f17500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:27.734 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f17500a3c40 0x7f17500a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f17540099d0 tx=0x7f1754008040 comp rx=0 tx=0).stop 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 msgr2=0x7f1764102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764102ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17600036f0 tx=0x7f17600039f0 comp rx=0 tx=0).stop 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 shutdown_connections 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17641087e0 0x7f1764101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1764103580 0x7f1764102ee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f17500a3c40 0x7f17500a6100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 --2- 172.21.15.193:0/4271223858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1764105eb0 0x7f1764101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 >> 172.21.15.193:0/4271223858 conn(0x7f17640fcf30 msgr2=0x7f1764103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 shutdown_connections 2024-04-22T06:41:27.735 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:27.736+0000 7f176b8d3700 1 -- 172.21.15.193:0/4271223858 wait complete. 2024-04-22T06:41:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:27 smithi193 ceph-mon[31815]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:27 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:27 smithi190 ceph-mon[31421]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:27 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:27 smithi079 ceph-mon[27946]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:27 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.390 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.11.service 2024-04-22T06:41:28.392 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-22T06:41:28.392 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T06:41:28.636 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:28.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:28 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:28 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:28 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:28.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:28 smithi193 ceph-mon[31815]: from='osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:28.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:28 smithi193 ceph-mon[31815]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:28 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:28 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:28 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:28 smithi190 ceph-mon[31421]: from='osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:28 smithi190 ceph-mon[31421]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:28 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:28 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:28 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:28 smithi079 ceph-mon[27946]: from='osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:28.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:28 smithi079 ceph-mon[27946]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:29.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:29 smithi193 ceph-mon[31815]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:29.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:29 smithi193 ceph-mon[31815]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:41:29.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:29 smithi193 ceph-mon[31815]: from='osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:41:29.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:29 smithi193 ceph-mon[31815]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T06:41:29.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:29 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:29.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:29 smithi193 ceph-mon[31815]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:41:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:29 smithi079 ceph-mon[27946]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:29 smithi079 ceph-mon[27946]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:41:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:29 smithi079 ceph-mon[27946]: from='osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:41:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:29 smithi079 ceph-mon[27946]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T06:41:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:29 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:29 smithi079 ceph-mon[27946]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:41:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:29 smithi190 ceph-mon[31421]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:29 smithi190 ceph-mon[31421]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:41:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:29 smithi190 ceph-mon[31421]: from='osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:41:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:29 smithi190 ceph-mon[31421]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T06:41:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:29 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:29 smithi190 ceph-mon[31421]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 -- 172.21.15.79:0/623257858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188102ac0 msgr2=0x7f4188102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 --2- 172.21.15.79:0/623257858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188102ac0 0x7f4188102ee0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f418000b0d0 tx=0x7f418000b3e0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 -- 172.21.15.79:0/623257858 shutdown_connections 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 --2- 172.21.15.79:0/623257858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4188104660 0x7f418810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 --2- 172.21.15.79:0/623257858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188102ac0 0x7f4188102ee0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 --2- 172.21.15.79:0/623257858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4188103cc0 0x7f4188104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 -- 172.21.15.79:0/623257858 >> 172.21.15.79:0/623257858 conn(0x7f41880fe060 msgr2=0x7f41881004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 -- 172.21.15.79:0/623257858 shutdown_connections 2024-04-22T06:41:30.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.439+0000 7f418c852700 1 -- 172.21.15.79:0/623257858 wait complete. 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 Processor -- start 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 -- start start 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4188102ac0 0x7f4188198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4188103cc0 0x7f4188199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 0x7f41881a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f418810e550 con 0x7f4188104660 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f418810e3d0 con 0x7f4188103cc0 2024-04-22T06:41:30.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418c852700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f418810e6d0 con 0x7f4188102ac0 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 0x7f41881a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 0x7f41881a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43268/0 (socket says 172.21.15.79:43268) 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 -- 172.21.15.79:0/581101419 learned_addr learned my addr 172.21.15.79:0/581101419 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f418659c700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4188102ac0 0x7f4188198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4185d9b700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4188103cc0 0x7f4188199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 -- 172.21.15.79:0/581101419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4188102ac0 msgr2=0x7f4188198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4188102ac0 0x7f4188198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 -- 172.21.15.79:0/581101419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4188103cc0 msgr2=0x7f4188199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4188103cc0 0x7f4188199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 -- 172.21.15.79:0/581101419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41881a0830 con 0x7f4188104660 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.440+0000 7f4186d9d700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 0x7f41881a0190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f417800e970 tx=0x7f417800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.441+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178004a30 con 0x7f4188104660 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.441+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41780054d0 con 0x7f4188104660 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.441+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41881a0b20 con 0x7f4188104660 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.441+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417800cd80 con 0x7f4188104660 2024-04-22T06:41:30.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.441+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41881a1010 con 0x7f4188104660 2024-04-22T06:41:30.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.442+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4188069730 con 0x7f4188104660 2024-04-22T06:41:30.442 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:41:30.154+0000 7fb3c59dd700 -1 osd.11 0 waiting for initial osdmap 2024-04-22T06:41:30.442 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:41:30.168+0000 7fb3bd7c8700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:41:30.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.443+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4178004fe0 con 0x7f4188104660 2024-04-22T06:41:30.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.443+0000 7f41777fe700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f41700a3c40 0x7f41700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:30.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.444+0000 7f418659c700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f41700a3c40 0x7f41700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:30.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.444+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f41780cff90 con 0x7f4188104660 2024-04-22T06:41:30.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.444+0000 7f418659c700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f41700a3c40 0x7f41700a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4180009be0 tx=0x7f418000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:30.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.448+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f417809cf30 con 0x7f4188104660 2024-04-22T06:41:30.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.645+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f418802e050 con 0x7f4188104660 2024-04-22T06:41:30.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.645+0000 7f41777fe700 1 -- 172.21.15.79:0/581101419 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f417809c830 con 0x7f4188104660 2024-04-22T06:41:30.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:30.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f41700a3c40 msgr2=0x7f41700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:30.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f41700a3c40 0x7f41700a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4180009be0 tx=0x7f418000b040 comp rx=0 tx=0).stop 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 msgr2=0x7f41881a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 0x7f41881a0190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f417800e970 tx=0x7f417800ec80 comp rx=0 tx=0).stop 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 shutdown_connections 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f41700a3c40 0x7f41700a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4188104660 0x7f41881a0190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4188102ac0 0x7f4188198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 --2- 172.21.15.79:0/581101419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4188103cc0 0x7f4188199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 >> 172.21.15.79:0/581101419 conn(0x7f41880fe060 msgr2=0x7f4188106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 shutdown_connections 2024-04-22T06:41:30.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:30.648+0000 7f418c852700 1 -- 172.21.15.79:0/581101419 wait complete. 2024-04-22T06:41:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:30 smithi079 ceph-mon[27946]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:41:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:30 smithi079 ceph-mon[27946]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T06:41:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:30 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:30 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:30 smithi079 ceph-mon[27946]: from='osd.11 ' entity='osd.11' 2024-04-22T06:41:30.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:30 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/581101419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-mon[31815]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:41:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-mon[31815]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T06:41:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-mon[31815]: from='osd.11 ' entity='osd.11' 2024-04-22T06:41:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:30 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/581101419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:31.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:30 smithi190 ceph-mon[31421]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-22T06:41:31.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:30 smithi190 ceph-mon[31421]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T06:41:31.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:30 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:31.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:30 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:31.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:30 smithi190 ceph-mon[31421]: from='osd.11 ' entity='osd.11' 2024-04-22T06:41:31.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:30 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/581101419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:31.489 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1713768063,"num_in_osds":12,"osd_in_since":1713768070,"num_remapped_pgs":0} 2024-04-22T06:41:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: purged_snaps scrub starts 2024-04-22T06:41:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: purged_snaps scrub ok 2024-04-22T06:41:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] boot 2024-04-22T06:41:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:31.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:31 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: purged_snaps scrub starts 2024-04-22T06:41:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: purged_snaps scrub ok 2024-04-22T06:41:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] boot 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:31 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: purged_snaps scrub starts 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: purged_snaps scrub ok 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: osd.11 [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] boot 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:32.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:32.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:31 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:32.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T06:41:32.729 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:32.851 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:32 smithi079 ceph-mon[27946]: Detected new or changed devices on smithi193 2024-04-22T06:41:32.851 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:32 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi193 to 4272M 2024-04-22T06:41:33.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:32 smithi193 ceph-mon[31815]: Detected new or changed devices on smithi193 2024-04-22T06:41:33.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:32 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi193 to 4272M 2024-04-22T06:41:33.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:32 smithi190 ceph-mon[31421]: Detected new or changed devices on smithi193 2024-04-22T06:41:33.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:32 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi193 to 4272M 2024-04-22T06:41:33.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:33 smithi079 ceph-mon[27946]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:33.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:33 smithi079 ceph-mon[27946]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T06:41:34.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:33 smithi193 ceph-mon[31815]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:34.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:33 smithi193 ceph-mon[31815]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T06:41:34.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:33 smithi190 ceph-mon[31421]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:34.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:33 smithi190 ceph-mon[31421]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T06:41:34.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.459+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4135458529 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc103530 msgr2=0x7f2ebc105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:34.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.459+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4135458529 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc103530 0x7f2ebc105920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009990 tx=0x7f2eb8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.459+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4135458529 shutdown_connections 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.459+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4135458529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 0x7f2ebc10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.459+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4135458529 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc103530 0x7f2ebc105920 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.459+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4135458529 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc105e60 0x7f2ebc108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.460+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4135458529 >> 172.21.15.79:0/4135458529 conn(0x7f2ebc0fcf40 msgr2=0x7f2ebc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.460+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4135458529 shutdown_connections 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.460+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4135458529 wait complete. 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.460+0000 7f2ec34b1700 1 Processor -- start 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.460+0000 7f2ec34b1700 1 -- start start 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc103530 0x7f2ebc198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 0x7f2ebc199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 0x7f2ebc1a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc10e620 con 0x7f2ebc105e60 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc10e4a0 con 0x7f2ebc103530 2024-04-22T06:41:34.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc10e7a0 con 0x7f2ebc108790 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 0x7f2ebc199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 0x7f2ebc199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57238/0 (socket says 172.21.15.79:57238) 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 -- 172.21.15.79:0/4166130960 learned_addr learned my addr 172.21.15.79:0/4166130960 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec124d700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc103530 0x7f2ebc198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec1a4e700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 0x7f2ebc1a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 -- 172.21.15.79:0/4166130960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 msgr2=0x7f2ebc1a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 0x7f2ebc1a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 -- 172.21.15.79:0/4166130960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc103530 msgr2=0x7f2ebc198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc103530 0x7f2ebc198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 -- 172.21.15.79:0/4166130960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc1a0980 con 0x7f2ebc105e60 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec1a4e700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 0x7f2ebc1a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec0a4c700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 0x7f2ebc199270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00b530 tx=0x7f2eac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac015020 con 0x7f2ebc105e60 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eac011ce0 con 0x7f2ebc105e60 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ebc1a0c70 con 0x7f2ebc105e60 2024-04-22T06:41:34.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac010e40 con 0x7f2ebc105e60 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.461+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ebc1a11b0 con 0x7f2ebc105e60 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.462+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ebc069730 con 0x7f2ebc105e60 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.463+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2eac018020 con 0x7f2ebc105e60 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.464+0000 7f2eb27fc700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ea80a3cc0 0x7f2ea80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.464+0000 7f2ec124d700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ea80a3cc0 0x7f2ea80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.464+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2eac0cfd40 con 0x7f2ebc105e60 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.465+0000 7f2ec124d700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ea80a3cc0 0x7f2ea80a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80038a0 tx=0x7f2eb8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:34.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.468+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2eac01f020 con 0x7f2ebc105e60 2024-04-22T06:41:34.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.668+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2ebc02e190 con 0x7f2ebc105e60 2024-04-22T06:41:34.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.669+0000 7f2eb27fc700 1 -- 172.21.15.79:0/4166130960 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f2eac09cb20 con 0x7f2ebc105e60 2024-04-22T06:41:34.667 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:34.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ea80a3cc0 msgr2=0x7f2ea80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ea80a3cc0 0x7f2ea80a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80038a0 tx=0x7f2eb8005fb0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 msgr2=0x7f2ebc199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 0x7f2ebc199270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00b530 tx=0x7f2eac00b840 comp rx=0 tx=0).stop 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 shutdown_connections 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ebc108790 0x7f2ebc1a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ebc103530 0x7f2ebc198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ea80a3cc0 0x7f2ea80a6180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 --2- 172.21.15.79:0/4166130960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2ebc105e60 0x7f2ebc199270 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 >> 172.21.15.79:0/4166130960 conn(0x7f2ebc0fcf40 msgr2=0x7f2ebc103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 shutdown_connections 2024-04-22T06:41:34.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:34.671+0000 7f2ec34b1700 1 -- 172.21.15.79:0/4166130960 wait complete. 2024-04-22T06:41:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4166130960' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4166130960' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:35.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4166130960' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:35.309 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713768091,"num_in_osds":12,"osd_in_since":1713768070,"num_remapped_pgs":0} 2024-04-22T06:41:35.309 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:41:35.546 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:35.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:35 smithi079 ceph-mon[27946]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:35 smithi193 ceph-mon[31815]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:36.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:35 smithi190 ceph-mon[31421]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1991521081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c41019d0 msgr2=0x7fb0c4101e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1991521081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c41019d0 0x7fb0c4101e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800b0a0 tx=0x7fb0b800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1991521081 shutdown_connections 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1991521081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c4102390 0x7fb0c410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1991521081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c4107940 0x7fb0c4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1991521081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c41019d0 0x7fb0c4101e50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1991521081 >> 172.21.15.79:0/1991521081 conn(0x7fb0c4077990 msgr2=0x7fb0c4077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1991521081 shutdown_connections 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.315+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1991521081 wait complete. 2024-04-22T06:41:37.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 Processor -- start 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 -- start start 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c41019d0 0x7fb0c419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 0x7fb0c419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c4107940 0x7fb0c41a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c4111be0 con 0x7fb0c4102390 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c4111a60 con 0x7fb0c41019d0 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.316+0000 7fb0c91fd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c4111d60 con 0x7fb0c4107940 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 0x7fb0c419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 0x7fb0c419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57260/0 (socket says 172.21.15.79:57260) 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 -- 172.21.15.79:0/1088629072 learned_addr learned my addr 172.21.15.79:0/1088629072 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c2d9d700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c41019d0 0x7fb0c419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c359e700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c4107940 0x7fb0c41a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 -- 172.21.15.79:0/1088629072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c4107940 msgr2=0x7fb0c41a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:37.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c4107940 0x7fb0c41a5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 -- 172.21.15.79:0/1088629072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c41019d0 msgr2=0x7fb0c419a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c41019d0 0x7fb0c419a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 -- 172.21.15.79:0/1088629072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c41a5660 con 0x7fb0c4102390 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c2d9d700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c41019d0 0x7fb0c419a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c259c700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 0x7fb0c419a5e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8006070 tx=0x7fb0b8004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8021070 con 0x7fb0c4102390 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c41a5830 con 0x7fb0c4102390 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b8007df0 con 0x7fb0c4102390 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b801d450 con 0x7fb0c4102390 2024-04-22T06:41:37.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.317+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c41a5d30 con 0x7fb0c4102390 2024-04-22T06:41:37.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.318+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c4069730 con 0x7fb0c4102390 2024-04-22T06:41:37.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.319+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb0b80040b0 con 0x7fb0c4102390 2024-04-22T06:41:37.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.320+0000 7fb0abfff700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0ac0a3c40 0x7fb0ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:37.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.320+0000 7fb0c2d9d700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0ac0a3c40 0x7fb0ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:37.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.320+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb0b800f020 con 0x7fb0c4102390 2024-04-22T06:41:37.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.320+0000 7fb0c2d9d700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0ac0a3c40 0x7fb0ac0a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0099d0 tx=0x7fb0bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:37.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.323+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0b802a020 con 0x7fb0c4102390 2024-04-22T06:41:37.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.524+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0c402df80 con 0x7fb0c4102390 2024-04-22T06:41:37.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.525+0000 7fb0abfff700 1 -- 172.21.15.79:0/1088629072 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18454 (secure 0 0 0) 0x7fb0b809e980 con 0x7fb0c4102390 2024-04-22T06:41:37.523 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:37.524 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":81,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:41:32.712071+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:41:37.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0ac0a3c40 msgr2=0x7fb0ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0ac0a3c40 0x7fb0ac0a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0099d0 tx=0x7fb0bc008040 comp rx=0 tx=0).stop 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 msgr2=0x7fb0c419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 0x7fb0c419a5e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8006070 tx=0x7fb0b8004ef0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 shutdown_connections 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c4107940 0x7fb0c41a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0c41019d0 0x7fb0c419a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0ac0a3c40 0x7fb0ac0a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 --2- 172.21.15.79:0/1088629072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0c4102390 0x7fb0c419a5e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 >> 172.21.15.79:0/1088629072 conn(0x7fb0c4077990 msgr2=0x7fb0c40ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 shutdown_connections 2024-04-22T06:41:37.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:37.527+0000 7fb0c91fd700 1 -- 172.21.15.79:0/1088629072 wait complete. 2024-04-22T06:41:37.843 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:37 smithi079 ceph-mon[27946]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:37.844 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:37 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1088629072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:38.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:37 smithi193 ceph-mon[31815]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:38.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:37 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1088629072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:38.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:37 smithi190 ceph-mon[31421]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:38.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:37 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1088629072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:38.299 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:41:38.300 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T06:41:38.535 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:39.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:39 smithi079 ceph-mon[27946]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:40.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:39 smithi193 ceph-mon[31815]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:40.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:39 smithi190 ceph-mon[31421]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:40.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/307997345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac06bb70 msgr2=0x7fdeac10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/307997345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac06bb70 0x7fdeac10f900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdea400b0d0 tx=0x7fdea400b3e0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/307997345 shutdown_connections 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/307997345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac06bb70 0x7fdeac10f900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/307997345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeac108a50 0x7fdeac108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/307997345 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06b1b0 0x7fdeac06b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/307997345 >> 172.21.15.79:0/307997345 conn(0x7fdeac077990 msgr2=0x7fdeac077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.227+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/307997345 shutdown_connections 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/307997345 wait complete. 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 Processor -- start 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 -- start start 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeac06b1b0 0x7fdeac1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06bb70 0x7fdeac199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:40.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 0x7fdeac19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac112c40 con 0x7fdeac108a50 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac112ac0 con 0x7fdeac06b1b0 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb2aa3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac112dc0 con 0x7fdeac06bb70 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb1040700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 0x7fdeac19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb1040700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 0x7fdeac19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57282/0 (socket says 172.21.15.79:57282) 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeb1040700 1 -- 172.21.15.79:0/1224726841 learned_addr learned my addr 172.21.15.79:0/1224726841 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.228+0000 7fdeabfff700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06bb70 0x7fdeac199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb083f700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeac06b1b0 0x7fdeac1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb1040700 1 -- 172.21.15.79:0/1224726841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06bb70 msgr2=0x7fdeac199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb1040700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06bb70 0x7fdeac199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb1040700 1 -- 172.21.15.79:0/1224726841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeac06b1b0 msgr2=0x7fdeac1990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb1040700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeac06b1b0 0x7fdeac1990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb1040700 1 -- 172.21.15.79:0/1224726841 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac19e0e0 con 0x7fdeac108a50 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeabfff700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06bb70 0x7fdeac199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:40.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb1040700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 0x7fdeac19d9a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009850 tx=0x7fdea4007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea401f070 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea4005520 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac19e370 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea401b870 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.229+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac1a5c50 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.230+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeac069730 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.231+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdea4022070 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.232+0000 7fdea9ffb700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fde9c0a3cc0 0x7fde9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.232+0000 7fdeb083f700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fde9c0a3cc0 0x7fde9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.232+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdea400f020 con 0x7fdeac108a50 2024-04-22T06:41:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.232+0000 7fdeb083f700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fde9c0a3cc0 0x7fde9c0a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdea000dad0 tx=0x7fdea000d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:40.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.235+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdea4029020 con 0x7fdeac108a50 2024-04-22T06:41:40.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.431+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdeac19a430 con 0x7fdeac108a50 2024-04-22T06:41:40.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.431+0000 7fdea9ffb700 1 -- 172.21.15.79:0/1224726841 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fdea409e3b0 con 0x7fdeac108a50 2024-04-22T06:41:40.430 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:41:40.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fde9c0a3cc0 msgr2=0x7fde9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:40.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fde9c0a3cc0 0x7fde9c0a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdea000dad0 tx=0x7fdea000d4b0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 msgr2=0x7fdeac19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 0x7fdeac19d9a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009850 tx=0x7fdea4007ad0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 shutdown_connections 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fde9c0a3cc0 0x7fde9c0a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdeac108a50 0x7fdeac19d9a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeac06b1b0 0x7fdeac1990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 --2- 172.21.15.79:0/1224726841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeac06bb70 0x7fdeac199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 >> 172.21.15.79:0/1224726841 conn(0x7fdeac077990 msgr2=0x7fdeac1006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 shutdown_connections 2024-04-22T06:41:40.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:40.434+0000 7fdeb2aa3700 1 -- 172.21.15.79:0/1224726841 wait complete. 2024-04-22T06:41:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:40 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1224726841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:40.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:40 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1224726841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:41.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:40 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1224726841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:41.256 INFO:tasks.cephadm:Adding mds.a on smithi079 2024-04-22T06:41:41.256 INFO:tasks.cephadm:Adding mds.d on smithi079 2024-04-22T06:41:41.256 INFO:tasks.cephadm:Adding mds.g on smithi079 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.j on smithi079 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.b on smithi190 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.e on smithi190 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.h on smithi190 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.k on smithi190 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-04-22T06:41:41.257 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-04-22T06:41:41.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-04-22T06:41:41.510 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:41:41.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:41 smithi193 ceph-mon[31815]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:41.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:41 smithi079 ceph-mon[27946]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:42.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:41 smithi190 ceph-mon[31421]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:43.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.078+0000 7f22c3145700 1 -- 172.21.15.193:0/535075073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc105f20 msgr2=0x7f22bc108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:43.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.078+0000 7f22c3145700 1 --2- 172.21.15.193:0/535075073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc105f20 0x7f22bc108310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009990 tx=0x7f22a8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- 172.21.15.193:0/535075073 shutdown_connections 2024-04-22T06:41:43.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 --2- 172.21.15.193:0/535075073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22bc108850 0x7f22bc10ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 --2- 172.21.15.193:0/535075073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 0x7f22bc1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 --2- 172.21.15.193:0/535075073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc105f20 0x7f22bc108310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- 172.21.15.193:0/535075073 >> 172.21.15.193:0/535075073 conn(0x7f22bc0fcfa0 msgr2=0x7f22bc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- 172.21.15.193:0/535075073 shutdown_connections 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- 172.21.15.193:0/535075073 wait complete. 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 Processor -- start 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- start start 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 0x7f22bc198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22bc105f20 0x7f22bc1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 0x7f22bc1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f22bc10e540 con 0x7f22bc105f20 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f22bc10e3c0 con 0x7f22bc1035f0 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.079+0000 7f22c3145700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22bc10e6c0 con 0x7f22bc108850 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 0x7f22bc1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:43.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 0x7f22bc1a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:53630/0 (socket says 172.21.15.193:53630) 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 -- 172.21.15.193:0/760684698 learned_addr learned my addr 172.21.15.193:0/760684698 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22bbfff700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22bc105f20 0x7f22bc1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c0ee1700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 0x7f22bc198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 -- 172.21.15.193:0/760684698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 msgr2=0x7f22bc198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 0x7f22bc198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 -- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22bc105f20 msgr2=0x7f22bc1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22bc105f20 0x7f22bc1990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 -- 172.21.15.193:0/760684698 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22bc1a07e0 con 0x7f22bc108850 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c0ee1700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 0x7f22bc198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c16e2700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 0x7f22bc1a0140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f22b4007d30 tx=0x7f22b400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b400ad60 con 0x7f22bc108850 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b4004030 con 0x7f22bc108850 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22bc1a0ad0 con 0x7f22bc108850 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.080+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4012680 con 0x7f22bc108850 2024-04-22T06:41:43.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.081+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22bc1a0fc0 con 0x7f22bc108850 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.082+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22bc069730 con 0x7f22bc108850 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.083+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f22b4003a00 con 0x7f22bc108850 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.083+0000 7f22b9ffb700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22ac0a3c40 0x7f22ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.083+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22b40d01b0 con 0x7f22bc108850 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.083+0000 7f22c0ee1700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22ac0a3c40 0x7f22ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.084+0000 7f22c0ee1700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22ac0a3c40 0x7f22ac0a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f22b0005fd0 tx=0x7f22b0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:43.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.088+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22b409d030 con 0x7f22bc108850 2024-04-22T06:41:43.297 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.299+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) -- 0x7f22bc063080 con 0x7f22ac0a3c40 2024-04-22T06:41:43.306 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.308+0000 7f22b9ffb700 1 -- 172.21.15.193:0/760684698 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f22bc063080 con 0x7f22ac0a3c40 2024-04-22T06:41:43.306 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-04-22T06:41:43.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22ac0a3c40 msgr2=0x7f22ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:43.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22ac0a3c40 0x7f22ac0a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f22b0005fd0 tx=0x7f22b0005ee0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 msgr2=0x7f22bc1a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 0x7f22bc1a0140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f22b4007d30 tx=0x7f22b400a340 comp rx=0 tx=0).stop 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 shutdown_connections 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22bc108850 0x7f22bc1a0140 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22bc1035f0 0x7f22bc198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22ac0a3c40 0x7f22ac0a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 --2- 172.21.15.193:0/760684698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22bc105f20 0x7f22bc1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 >> 172.21.15.193:0/760684698 conn(0x7f22bc0fcfa0 msgr2=0x7f22bc104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 shutdown_connections 2024-04-22T06:41:43.309 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-22T06:41:43.310+0000 7f22c3145700 1 -- 172.21.15.193:0/760684698 wait complete. 2024-04-22T06:41:43.941 DEBUG:teuthology.orchestra.run.smithi079:mds.a> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.a.service 2024-04-22T06:41:43.944 DEBUG:teuthology.orchestra.run.smithi079:mds.d> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.d.service 2024-04-22T06:41:43.987 DEBUG:teuthology.orchestra.run.smithi079:mds.g> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.g.service 2024-04-22T06:41:43.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:43 smithi079 ceph-mon[27946]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:43 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:43.990 DEBUG:teuthology.orchestra.run.smithi079:mds.j> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.j.service 2024-04-22T06:41:44.033 DEBUG:teuthology.orchestra.run.smithi190:mds.b> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.b.service 2024-04-22T06:41:44.036 DEBUG:teuthology.orchestra.run.smithi190:mds.e> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service 2024-04-22T06:41:44.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:43 smithi190 ceph-mon[31421]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:44.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:44.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:43 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:44.056 DEBUG:teuthology.orchestra.run.smithi190:mds.h> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service 2024-04-22T06:41:44.098 DEBUG:teuthology.orchestra.run.smithi190:mds.k> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.k.service 2024-04-22T06:41:44.141 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service 2024-04-22T06:41:44.144 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service 2024-04-22T06:41:44.162 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:43 smithi193 ceph-mon[31815]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:44.163 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:44.163 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:43 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:44.164 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service 2024-04-22T06:41:44.206 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.l.service 2024-04-22T06:41:44.249 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-22T06:41:44.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:41:44.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 -- 172.21.15.79:0/839494465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020107c60 msgr2=0x7f602010a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 --2- 172.21.15.79:0/839494465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020107c60 0x7f602010a0b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6018009ca0 tx=0x7f601802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 -- 172.21.15.79:0/839494465 shutdown_connections 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 --2- 172.21.15.79:0/839494465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020107c60 0x7f602010a0b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 --2- 172.21.15.79:0/839494465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6020105330 0x7f6020107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 --2- 172.21.15.79:0/839494465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6020102a00 0x7f6020104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 -- 172.21.15.79:0/839494465 >> 172.21.15.79:0/839494465 conn(0x7f60200fc410 msgr2=0x7f60200fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 -- 172.21.15.79:0/839494465 shutdown_connections 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.368+0000 7f60263d0640 1 -- 172.21.15.79:0/839494465 wait complete. 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 Processor -- start 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 -- start start 2024-04-22T06:41:44.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6020102a00 0x7f6020198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 0x7f60201986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6020107c60 0x7f602019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f602010dac0 con 0x7f6020105330 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f602010d940 con 0x7f6020107c60 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f60263d0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f602010dc40 con 0x7f6020102a00 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 0x7f60201986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 0x7f60201986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42780/0 (socket says 172.21.15.79:42780) 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 -- 172.21.15.79:0/849573533 learned_addr learned my addr 172.21.15.79:0/849573533 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f602496d640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6020107c60 0x7f602019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601ffff640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6020102a00 0x7f6020198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 -- 172.21.15.79:0/849573533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6020102a00 msgr2=0x7f6020198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6020102a00 0x7f6020198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 -- 172.21.15.79:0/849573533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6020107c60 msgr2=0x7f602019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6020107c60 0x7f602019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.369+0000 7f601f7fe640 1 -- 172.21.15.79:0/849573533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602019fd80 con 0x7f6020105330 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.370+0000 7f601f7fe640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 0x7f60201986a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f601000e9f0 tx=0x7f601000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.370+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601000ccc0 con 0x7f6020105330 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.370+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60201a0070 con 0x7f6020105330 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.370+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60100054d0 con 0x7f6020105330 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.370+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010010430 con 0x7f6020105330 2024-04-22T06:41:44.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.370+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60201a05c0 con 0x7f6020105330 2024-04-22T06:41:44.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.371+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60200fdf00 con 0x7f6020105330 2024-04-22T06:41:44.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.372+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f60100046c0 con 0x7f6020105330 2024-04-22T06:41:44.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.372+0000 7f601d7fa640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f600c0a3bb0 0x7f600c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.372+0000 7f601ffff640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f600c0a3bb0 0x7f600c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.373+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f60100d0490 con 0x7f6020105330 2024-04-22T06:41:44.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.373+0000 7f601ffff640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f600c0a3bb0 0x7f600c0a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f600800bbe0 tx=0x7f600800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:44.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.376+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6010053ff0 con 0x7f6020105330 2024-04-22T06:41:44.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.554+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f602002d7c0 con 0x7f6020105330 2024-04-22T06:41:44.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.555+0000 7f601d7fa640 1 -- 172.21.15.79:0/849573533 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18454 (secure 0 0 0) 0x7f601009d310 con 0x7f6020105330 2024-04-22T06:41:44.554 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:44.554 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":81,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:41:32.712071+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f600c0a3bb0 msgr2=0x7f600c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f600c0a3bb0 0x7f600c0a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f600800bbe0 tx=0x7f600800b390 comp rx=0 tx=0).stop 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 msgr2=0x7f60201986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 0x7f60201986a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f601000e9f0 tx=0x7f601000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 shutdown_connections 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f600c0a3bb0 0x7f600c0a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6020107c60 0x7f602019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6020105330 0x7f60201986a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 --2- 172.21.15.79:0/849573533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6020102a00 0x7f6020198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 >> 172.21.15.79:0/849573533 conn(0x7f60200fc410 msgr2=0x7f6020103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:44.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 shutdown_connections 2024-04-22T06:41:44.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.557+0000 7f60263d0640 1 -- 172.21.15.79:0/849573533 wait complete. 2024-04-22T06:41:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:41:44.567 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 -- 172.21.15.79:0/1142410817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 msgr2=0x7f5fdc1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1142410817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc1012b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009ca0 tx=0x7f5fcc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 -- 172.21.15.79:0/1142410817 shutdown_connections 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1142410817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc1029a0 0x7f5fdc1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1142410817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc101fe0 0x7f5fdc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1142410817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc1012b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 -- 172.21.15.79:0/1142410817 >> 172.21.15.79:0/1142410817 conn(0x7f5fdc0fc400 msgr2=0x7f5fdc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 -- 172.21.15.79:0/1142410817 shutdown_connections 2024-04-22T06:41:44.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 -- 172.21.15.79:0/1142410817 wait complete. 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 Processor -- start 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.670+0000 7f5fe2674640 1 -- start start 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc101fe0 0x7f5fdc19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1029a0 0x7f5fdc19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc10c9e0 con 0x7f5fdc100e90 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc10c860 con 0x7f5fdc101fe0 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc10cb60 con 0x7f5fdc1029a0 2024-04-22T06:41:44.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42802/0 (socket says 172.21.15.79:42802) 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 -- 172.21.15.79:0/1889574372 learned_addr learned my addr 172.21.15.79:0/1889574372 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fd37fe640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc101fe0 0x7f5fdc19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe0c11640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1029a0 0x7f5fdc19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 -- 172.21.15.79:0/1889574372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1029a0 msgr2=0x7f5fdc19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1029a0 0x7f5fdc19f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 -- 172.21.15.79:0/1889574372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc101fe0 msgr2=0x7f5fdc19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc101fe0 0x7f5fdc19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 -- 172.21.15.79:0/1889574372 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc19fee0 con 0x7f5fdc100e90 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fd37fe640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc101fe0 0x7f5fdc19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fdbfff640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc069c50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc005e30 tx=0x7f5fcc034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc039040 con 0x7f5fdc100e90 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fcc0325c0 con 0x7f5fdc100e90 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fdc1a0110 con 0x7f5fdc100e90 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0042a0 con 0x7f5fdc100e90 2024-04-22T06:41:44.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.671+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fdc1a05d0 con 0x7f5fdc100e90 2024-04-22T06:41:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.672+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fdc1082c0 con 0x7f5fdc100e90 2024-04-22T06:41:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.673+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5fcc046030 con 0x7f5fdc100e90 2024-04-22T06:41:44.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.674+0000 7f5fd9ffb640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5fc80a3bb0 0x7f5fc80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.674+0000 7f5fd37fe640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5fc80a3bb0 0x7f5fc80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.674+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5fcc0f19f0 con 0x7f5fdc100e90 2024-04-22T06:41:44.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.674+0000 7f5fd37fe640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5fc80a3bb0 0x7f5fc80a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0699c0 tx=0x7f5fc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:44.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.679+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fcc04d060 con 0x7f5fdc100e90 2024-04-22T06:41:44.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.857+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5fdc02d7e0 con 0x7f5fdc100e90 2024-04-22T06:41:44.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.857+0000 7f5fd9ffb640 1 -- 172.21.15.79:0/1889574372 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f5fcc046320 con 0x7f5fdc100e90 2024-04-22T06:41:44.856 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:41:44.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5fc80a3bb0 msgr2=0x7f5fc80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5fc80a3bb0 0x7f5fc80a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0699c0 tx=0x7f5fc4005f90 comp rx=0 tx=0).stop 2024-04-22T06:41:44.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 msgr2=0x7f5fdc069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc069c50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc005e30 tx=0x7f5fcc034760 comp rx=0 tx=0).stop 2024-04-22T06:41:44.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 shutdown_connections 2024-04-22T06:41:44.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5fc80a3bb0 0x7f5fc80a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5fdc101fe0 0x7f5fdc19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1029a0 0x7f5fdc19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 --2- 172.21.15.79:0/1889574372 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fdc100e90 0x7f5fdc069c50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.859+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 >> 172.21.15.79:0/1889574372 conn(0x7f5fdc0fc400 msgr2=0x7f5fdc105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:44.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.860+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 shutdown_connections 2024-04-22T06:41:44.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.860+0000 7f5fe2674640 1 -- 172.21.15.79:0/1889574372 wait complete. 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: Saving service mds.all spec with placement smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:44.868 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:44 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/849573533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:44.869 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/3681725444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 msgr2=0x7f7fe0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/3681725444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0104e10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4009900 tx=0x7f7fd402bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/3681725444 shutdown_connections 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/3681725444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe0105350 0x7f7fe0107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/3681725444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 0x7f7fe010a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/3681725444 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0104e10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/3681725444 >> 172.21.15.79:0/3681725444 conn(0x7f7fe00fc410 msgr2=0x7f7fe00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:44.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/3681725444 shutdown_connections 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/3681725444 wait complete. 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 Processor -- start 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- start start 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe0105350 0x7f7fe0198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 0x7f7fe019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe010dae0 con 0x7f7fe0102a20 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe010d960 con 0x7f7fe0105350 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe6d2b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe010dc60 con 0x7f7fe0107c80 2024-04-22T06:41:44.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe4ac7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe4ac7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42810/0 (socket says 172.21.15.79:42810) 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.969+0000 7f7fe4ac7640 1 -- 172.21.15.79:0/2223962466 learned_addr learned my addr 172.21.15.79:0/2223962466 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe52c8640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 0x7f7fe019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe4ac7640 1 -- 172.21.15.79:0/2223962466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 msgr2=0x7f7fe019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe4ac7640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 0x7f7fe019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe4ac7640 1 -- 172.21.15.79:0/2223962466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe0105350 msgr2=0x7f7fe0198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe4ac7640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe0105350 0x7f7fe0198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe4ac7640 1 -- 172.21.15.79:0/2223962466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe019fd20 con 0x7f7fe0102a20 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe52c8640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 0x7f7fe019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe4ac7640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0198100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4006010 tx=0x7f7fd40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4043020 con 0x7f7fe0102a20 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd4004aa0 con 0x7f7fe0102a20 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe019ffb0 con 0x7f7fe0102a20 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4003c70 con 0x7f7fe0102a20 2024-04-22T06:41:44.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.970+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe01a04c0 con 0x7f7fe0102a20 2024-04-22T06:41:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.971+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe00fdf00 con 0x7f7fe0102a20 2024-04-22T06:41:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.972+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7fd4046020 con 0x7f7fe0102a20 2024-04-22T06:41:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.973+0000 7f7fddffb640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7fd00a3c30 0x7f7fd00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:44.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.973+0000 7f7fdffff640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7fd00a3c30 0x7f7fd00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.973+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7fd40f2350 con 0x7f7fe0102a20 2024-04-22T06:41:44.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.973+0000 7f7fdffff640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7fd00a3c30 0x7f7fd00a60f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0199610 tx=0x7f7fcc005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:44.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:44.977+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fd404d020 con 0x7f7fe0102a20 2024-04-22T06:41:44.983 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:44.983 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: Saving service mds.all spec with placement smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-22T06:41:44.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:44.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:44.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:44.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:44.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:44.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:44.985 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:44 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/849573533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.193+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fe002d830 con 0x7f7fe0102a20 2024-04-22T06:41:45.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.194+0000 7f7fddffb640 1 -- 172.21.15.79:0/2223962466 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7fd40bf1d0 con 0x7f7fe0102a20 2024-04-22T06:41:45.192 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:45.192 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-22T06:41:45.193 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 1 2024-04-22T06:41:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7fd00a3c30 msgr2=0x7f7fd00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7fd00a3c30 0x7f7fd00a60f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0199610 tx=0x7f7fcc005c70 comp rx=0 tx=0).stop 2024-04-22T06:41:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 msgr2=0x7f7fe0198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0198100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4006010 tx=0x7f7fd40042f0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 shutdown_connections 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7fd00a3c30 0x7f7fd00a60f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fe0105350 0x7f7fe0198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe0107c80 0x7f7fe019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 --2- 172.21.15.79:0/2223962466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fe0102a20 0x7f7fe0198100 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 >> 172.21.15.79:0/2223962466 conn(0x7f7fe00fc410 msgr2=0x7f7fe01032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 shutdown_connections 2024-04-22T06:41:45.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:45.196+0000 7f7fe6d2b640 1 -- 172.21.15.79:0/2223962466 wait complete. 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: Saving service mds.all spec with placement smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:45.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:44 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/849573533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:45 smithi193 ceph-mon[31815]: Deploying daemon mds.l on smithi193 2024-04-22T06:41:46.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:45 smithi193 ceph-mon[31815]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:46.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:45 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1889574372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:46.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:45 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2223962466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:45 smithi190 ceph-mon[31421]: Deploying daemon mds.l on smithi193 2024-04-22T06:41:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:45 smithi190 ceph-mon[31421]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:45 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1889574372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:45 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2223962466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:45 smithi079 ceph-mon[27946]: Deploying daemon mds.l on smithi193 2024-04-22T06:41:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:45 smithi079 ceph-mon[27946]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:45 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1889574372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:46.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:45 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2223962466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.393 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 systemd[1]: Starting Ceph mds.l for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:46.676 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 podman[62305]: 2024-04-22 06:41:46.298683034 +0000 UTC m=+0.017761739 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:46.676 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 podman[62305]: 2024-04-22 06:41:46.434318537 +0000 UTC m=+0.153397260 container create 54adb40b1d86167d7baf35d9d721d502f12028f143fc747fe34fb0e56c1f8bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:41:46.677 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 podman[62305]: 2024-04-22 06:41:46.67882616 +0000 UTC m=+0.397904838 container init 54adb40b1d86167d7baf35d9d721d502f12028f143fc747fe34fb0e56c1f8bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T06:41:46.939 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 podman[62305]: 2024-04-22 06:41:46.681762468 +0000 UTC m=+0.400841146 container start 54adb40b1d86167d7baf35d9d721d502f12028f143fc747fe34fb0e56c1f8bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T06:41:46.939 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 ceph-mds[62324]: -- 172.21.15.193:0/1541765942 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce1f947a40 con 0x55ce1f91ac00 2024-04-22T06:41:46.939 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l[62320]: starting mds.l at 2024-04-22T06:41:46.940 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 bash[62305]: 54adb40b1d86167d7baf35d9d721d502f12028f143fc747fe34fb0e56c1f8bfa 2024-04-22T06:41:46.940 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:41:46 smithi193 systemd[1]: Started Ceph mds.l for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:47.206 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 -- 172.21.15.79:0/1291245694 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 msgr2=0x7f550c102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 --2- 172.21.15.79:0/1291245694 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c102c20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5504009ca0 tx=0x7f550402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 -- 172.21.15.79:0/1291245694 shutdown_connections 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 --2- 172.21.15.79:0/1291245694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c103160 0x7f550c1055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 --2- 172.21.15.79:0/1291245694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 0x7f550c105f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 --2- 172.21.15.79:0/1291245694 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c102c20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 -- 172.21.15.79:0/1291245694 >> 172.21.15.79:0/1291245694 conn(0x7f550c0fc410 msgr2=0x7f550c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 -- 172.21.15.79:0/1291245694 shutdown_connections 2024-04-22T06:41:47.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.318+0000 7f5513e74640 1 -- 172.21.15.79:0/1291245694 wait complete. 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 Processor -- start 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 -- start start 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c103160 0x7f550c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 0x7f550c1a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f550c1140f0 con 0x7f550c1007f0 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f550c113f70 con 0x7f550c103160 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5513e74640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f550c114270 con 0x7f550c105b10 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c19e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42842/0 (socket says 172.21.15.79:42842) 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 -- 172.21.15.79:0/2869383338 learned_addr learned my addr 172.21.15.79:0/2869383338 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f551140f640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c103160 0x7f550c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5512411640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 0x7f550c1a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:47.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 -- 172.21.15.79:0/2869383338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 msgr2=0x7f550c1a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 0x7f550c1a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 -- 172.21.15.79:0/2869383338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c103160 msgr2=0x7f550c19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c103160 0x7f550c19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 -- 172.21.15.79:0/2869383338 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c1a63e0 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5512411640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 0x7f550c1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f5511c10640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c19e7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5504005e30 tx=0x7f5504034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504039040 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.319+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55040325c0 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.320+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550c1a6670 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.320+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55040042a0 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.320+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550c1a6b80 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.321+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550c0fdf00 con 0x7f550c1007f0 2024-04-22T06:41:47.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.321+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5504046030 con 0x7f550c1007f0 2024-04-22T06:41:47.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.322+0000 7f550affd640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f54f40a3c30 0x7f54f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:47.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.322+0000 7f551140f640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f54f40a3c30 0x7f54f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:47.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.322+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55040f18f0 con 0x7f550c1007f0 2024-04-22T06:41:47.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.323+0000 7f551140f640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f54f40a3c30 0x7f54f40a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f550c19fcd0 tx=0x7f5500005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:47.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.327+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f550404d060 con 0x7f550c1007f0 2024-04-22T06:41:47.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.543+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f550c02d5c0 con 0x7f550c1007f0 2024-04-22T06:41:47.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.544+0000 7f550affd640 1 -- 172.21.15.79:0/2869383338 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f55040be800 con 0x7f550c1007f0 2024-04-22T06:41:47.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:47.542 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:41:47.543 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 2 2024-04-22T06:41:47.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.545+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f54f40a3c30 msgr2=0x7f54f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:47.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.545+0000 7f5513e74640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f54f40a3c30 0x7f54f40a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f550c19fcd0 tx=0x7f5500005f90 comp rx=0 tx=0).stop 2024-04-22T06:41:47.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 msgr2=0x7f550c19e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:47.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c19e7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5504005e30 tx=0x7f5504034760 comp rx=0 tx=0).stop 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 shutdown_connections 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f54f40a3c30 0x7f54f40a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c103160 0x7f550c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c105b10 0x7f550c1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 --2- 172.21.15.79:0/2869383338 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c1007f0 0x7f550c19e7c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 >> 172.21.15.79:0/2869383338 conn(0x7f550c0fc410 msgr2=0x7f550c103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 shutdown_connections 2024-04-22T06:41:47.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:47.546+0000 7f5513e74640 1 -- 172.21.15.79:0/2869383338 wait complete. 2024-04-22T06:41:47.753 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6832/1714090730,v1:172.21.15.193:6833/1714090730] up:boot 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: fsmap 1 up:standby 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:47 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2869383338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6832/1714090730,v1:172.21.15.193:6833/1714090730] up:boot 2024-04-22T06:41:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: fsmap 1 up:standby 2024-04-22T06:41:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:41:47.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:47.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:47.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:47.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:47.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:47.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:47.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:47 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2869383338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6832/1714090730,v1:172.21.15.193:6833/1714090730] up:boot 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: fsmap 1 up:standby 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:47 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2869383338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:48.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:48 smithi190 ceph-mon[31421]: Deploying daemon mds.k on smithi190 2024-04-22T06:41:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:48 smithi079 ceph-mon[27946]: Deploying daemon mds.k on smithi190 2024-04-22T06:41:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:48 smithi193 ceph-mon[31815]: Deploying daemon mds.k on smithi190 2024-04-22T06:41:49.203 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 systemd[1]: Starting Ceph mds.k for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:49.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:49.602 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 podman[61384]: 2024-04-22 06:41:49.246247991 +0000 UTC m=+0.018117558 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:49.602 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 podman[61384]: 2024-04-22 06:41:49.363598936 +0000 UTC m=+0.135468513 container create 77a8771c3b90284d404df7fa1329be9c793bc49274620cc7a95a78e825f9b723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T06:41:49.602 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 podman[61384]: 2024-04-22 06:41:49.604625131 +0000 UTC m=+0.376494665 container init 77a8771c3b90284d404df7fa1329be9c793bc49274620cc7a95a78e825f9b723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD) 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 -- 172.21.15.79:0/2410138677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 msgr2=0x7fbaf4104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 --2- 172.21.15.79:0/2410138677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf4104de0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009900 tx=0x7fbae802bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 -- 172.21.15.79:0/2410138677 shutdown_connections 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 --2- 172.21.15.79:0/2410138677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4105320 0x7fbaf4107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 --2- 172.21.15.79:0/2410138677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaf4107c50 0x7fbaf410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 --2- 172.21.15.79:0/2410138677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf4104de0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 -- 172.21.15.79:0/2410138677 >> 172.21.15.79:0/2410138677 conn(0x7fbaf40fc3e0 msgr2=0x7fbaf40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:49.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 -- 172.21.15.79:0/2410138677 shutdown_connections 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.666+0000 7fbafab92640 1 -- 172.21.15.79:0/2410138677 wait complete. 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 Processor -- start 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 -- start start 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaf4105320 0x7fbaf4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4107c50 0x7fbaf419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410daa0 con 0x7fbaf41029f0 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410d920 con 0x7fbaf4107c50 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410dc20 con 0x7fbaf4105320 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf41980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42856/0 (socket says 172.21.15.79:42856) 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 -- 172.21.15.79:0/1576257018 learned_addr learned my addr 172.21.15.79:0/1576257018 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf3fff640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaf4105320 0x7fbaf4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf912f640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4107c50 0x7fbaf419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 -- 172.21.15.79:0/1576257018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaf4105320 msgr2=0x7fbaf4198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaf4105320 0x7fbaf4198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 -- 172.21.15.79:0/1576257018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4107c50 msgr2=0x7fbaf419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4107c50 0x7fbaf419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 -- 172.21.15.79:0/1576257018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf419fce0 con 0x7fbaf41029f0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf912f640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4107c50 0x7fbaf419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf892e640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf41980c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbae8036040 tx=0x7fbae80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8043020 con 0x7fbaf41029f0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae80054c0 con 0x7fbaf41029f0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.667+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf419ff70 con 0x7fbaf41029f0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.668+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8003d40 con 0x7fbaf41029f0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.668+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf41a03d0 con 0x7fbaf41029f0 2024-04-22T06:41:49.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.669+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf40fded0 con 0x7fbaf41029f0 2024-04-22T06:41:49.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.670+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbae8046020 con 0x7fbaf41029f0 2024-04-22T06:41:49.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.670+0000 7fbaf1ffb640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fbae40a3bb0 0x7fbae40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.670+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbae80f2430 con 0x7fbaf41029f0 2024-04-22T06:41:49.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.670+0000 7fbaf3fff640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fbae40a3bb0 0x7fbae40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.671+0000 7fbaf3fff640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fbae40a3bb0 0x7fbae40a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbaf41995d0 tx=0x7fbae000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:49.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.675+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbae804d020 con 0x7fbaf41029f0 2024-04-22T06:41:49.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:49 smithi190 ceph-mon[31421]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:49.853 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 podman[61384]: 2024-04-22 06:41:49.607258796 +0000 UTC m=+0.379128333 container start 77a8771c3b90284d404df7fa1329be9c793bc49274620cc7a95a78e825f9b723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T06:41:49.853 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 ceph-mds[61403]: -- 172.21.15.190:0/3118373513 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56476f0d3a40 con 0x56476f0a7000 2024-04-22T06:41:49.853 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k[61398]: starting mds.k at 2024-04-22T06:41:49.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.894+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaf4062b50 con 0x7fbaf41029f0 2024-04-22T06:41:49.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.895+0000 7fbaf1ffb640 1 -- 172.21.15.79:0/1576257018 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fbae80bf2b0 con 0x7fbaf41029f0 2024-04-22T06:41:49.893 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:49.893 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:41:49.894 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 3 2024-04-22T06:41:49.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fbae40a3bb0 msgr2=0x7fbae40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fbae40a3bb0 0x7fbae40a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbaf41995d0 tx=0x7fbae000a2d0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 msgr2=0x7fbaf41980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf41980c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbae8036040 tx=0x7fbae80042f0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 shutdown_connections 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fbae40a3bb0 0x7fbae40a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbaf4107c50 0x7fbaf419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaf4105320 0x7fbaf4198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 --2- 172.21.15.79:0/1576257018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbaf41029f0 0x7fbaf41980c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 >> 172.21.15.79:0/1576257018 conn(0x7fbaf40fc3e0 msgr2=0x7fbaf4103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 shutdown_connections 2024-04-22T06:41:49.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:49.897+0000 7fbafab92640 1 -- 172.21.15.79:0/1576257018 wait complete. 2024-04-22T06:41:49.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:49 smithi079 ceph-mon[27946]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:50.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:49 smithi193 ceph-mon[31815]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:50.202 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 bash[61384]: 77a8771c3b90284d404df7fa1329be9c793bc49274620cc7a95a78e825f9b723 2024-04-22T06:41:50.202 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:41:49 smithi190 systemd[1]: Started Ceph mds.k for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6832/1510033395,v1:172.21.15.190:6833/1510033395] up:boot 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: fsmap 2 up:standby 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1576257018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:50.901 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:50 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6832/1510033395,v1:172.21.15.190:6833/1510033395] up:boot 2024-04-22T06:41:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: fsmap 2 up:standby 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1576257018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:50 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6832/1510033395,v1:172.21.15.190:6833/1510033395] up:boot 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: fsmap 2 up:standby 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1576257018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:50 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:51.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:51 smithi079 ceph-mon[27946]: Deploying daemon mds.j on smithi079 2024-04-22T06:41:51.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:51 smithi079 ceph-mon[27946]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:51.908 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:52.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 -- 172.21.15.79:0/2101196135 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4102980 msgr2=0x7f21b41092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2101196135 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4102980 0x7f21b41092a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009ca0 tx=0x7f21ac02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 -- 172.21.15.79:0/2101196135 shutdown_connections 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2101196135 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4102980 0x7f21b41092a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2101196135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4101fc0 0x7f21b4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2101196135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b4100e70 0x7f21b4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 -- 172.21.15.79:0/2101196135 >> 172.21.15.79:0/2101196135 conn(0x7f21b40fc3e0 msgr2=0x7f21b40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 -- 172.21.15.79:0/2101196135 shutdown_connections 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 -- 172.21.15.79:0/2101196135 wait complete. 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.132+0000 7f21bb66d640 1 Processor -- start 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 -- start start 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4100e70 0x7f21b4069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 0x7f21b419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b4102980 0x7f21b419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f21b410ca40 con 0x7f21b4101fc0 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21b410c8c0 con 0x7f21b4102980 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f21b410cbc0 con 0x7f21b4100e70 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 0x7f21b419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 0x7f21b419cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42878/0 (socket says 172.21.15.79:42878) 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 -- 172.21.15.79:0/2679101185 learned_addr learned my addr 172.21.15.79:0/2679101185 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b9c0a640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b4102980 0x7f21b419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b9409640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4100e70 0x7f21b4069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 -- 172.21.15.79:0/2679101185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4100e70 msgr2=0x7f21b4069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4100e70 0x7f21b4069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 -- 172.21.15.79:0/2679101185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b4102980 msgr2=0x7f21b419f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b4102980 0x7f21b419f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 -- 172.21.15.79:0/2679101185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b419fe40 con 0x7f21b4101fc0 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b9409640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4100e70 0x7f21b4069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21b8c08640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 0x7f21b419cf10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f21a400b450 tx=0x7f21a400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4015020 con 0x7f21b4101fc0 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b41a00d0 con 0x7f21b4101fc0 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21a4011d40 con 0x7f21b4101fc0 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.133+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a400f340 con 0x7f21b4101fc0 2024-04-22T06:41:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.134+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b41a05d0 con 0x7f21b4101fc0 2024-04-22T06:41:52.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.135+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b4108180 con 0x7f21b4101fc0 2024-04-22T06:41:52.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.135+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f21a4018020 con 0x7f21b4101fc0 2024-04-22T06:41:52.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.136+0000 7f21aa7fc640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21a00a3bb0 0x7f21a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.136+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f21a40cfa60 con 0x7f21b4101fc0 2024-04-22T06:41:52.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.136+0000 7f21b9409640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21a00a3bb0 0x7f21a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.137+0000 7f21b9409640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21a00a3bb0 0x7f21a00a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f21b0009b40 tx=0x7f21b0005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:52.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.141+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21a401f020 con 0x7f21b4101fc0 2024-04-22T06:41:52.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:51 smithi190 ceph-mon[31421]: Deploying daemon mds.j on smithi079 2024-04-22T06:41:52.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:51 smithi190 ceph-mon[31421]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:51 smithi193 ceph-mon[31815]: Deploying daemon mds.j on smithi079 2024-04-22T06:41:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:51 smithi193 ceph-mon[31815]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:52.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.390+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21b402d790 con 0x7f21b4101fc0 2024-04-22T06:41:52.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.390+0000 7f21aa7fc640 1 -- 172.21.15.79:0/2679101185 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f21a40cde20 con 0x7f21b4101fc0 2024-04-22T06:41:52.389 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:52.389 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:41:52.389 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 3 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21a00a3bb0 msgr2=0x7f21a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21a00a3bb0 0x7f21a00a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f21b0009b40 tx=0x7f21b0005e50 comp rx=0 tx=0).stop 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 msgr2=0x7f21b419cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 0x7f21b419cf10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f21a400b450 tx=0x7f21a400b910 comp rx=0 tx=0).stop 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 shutdown_connections 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21a00a3bb0 0x7f21a00a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21b4102980 0x7f21b419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21b4101fc0 0x7f21b419cf10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 --2- 172.21.15.79:0/2679101185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21b4100e70 0x7f21b4069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 >> 172.21.15.79:0/2679101185 conn(0x7f21b40fc3e0 msgr2=0x7f21b4105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:52.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 shutdown_connections 2024-04-22T06:41:52.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:52.392+0000 7f21bb66d640 1 -- 172.21.15.79:0/2679101185 wait complete. 2024-04-22T06:41:52.526 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 podman[65605]: 2024-04-22 06:41:52.392671681 +0000 UTC m=+0.017863784 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:52.526 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 podman[65605]: 2024-04-22 06:41:52.528889243 +0000 UTC m=+0.154081370 container create 291f0baadb8bf8e12b0b2abff11033ad3a100ed7fcad460e74a538b013faab72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-22T06:41:52.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:52 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2679101185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:52.797 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 podman[65605]: 2024-04-22 06:41:52.773898721 +0000 UTC m=+0.399090806 container init 291f0baadb8bf8e12b0b2abff11033ad3a100ed7fcad460e74a538b013faab72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T06:41:52.797 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 podman[65605]: 2024-04-22 06:41:52.776713636 +0000 UTC m=+0.401905725 container start 291f0baadb8bf8e12b0b2abff11033ad3a100ed7fcad460e74a538b013faab72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:41:53.105 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:52 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2679101185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:52 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2679101185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:53.240 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 ceph-mds[65627]: -- 172.21.15.79:0/3168401250 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a66b065a40 con 0x55a66b039000 2024-04-22T06:41:53.240 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j[65623]: starting mds.j at 2024-04-22T06:41:53.240 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 bash[65605]: 291f0baadb8bf8e12b0b2abff11033ad3a100ed7fcad460e74a538b013faab72 2024-04-22T06:41:53.240 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:41:52 smithi079 systemd[1]: Started Ceph mds.j for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:53.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:53 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:54.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:53 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:54.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:54.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:53 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:54.403 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 -- 172.21.15.79:0/2488342479 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 msgr2=0x7fcf28101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2488342479 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf28101290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009900 tx=0x7fcf1802bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 -- 172.21.15.79:0/2488342479 shutdown_connections 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2488342479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28101fc0 0x7fcf28102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2488342479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28102980 0x7fcf281092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2488342479 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf28101290 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 -- 172.21.15.79:0/2488342479 >> 172.21.15.79:0/2488342479 conn(0x7fcf280fc3e0 msgr2=0x7fcf280fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:54.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 -- 172.21.15.79:0/2488342479 shutdown_connections 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 -- 172.21.15.79:0/2488342479 wait complete. 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 Processor -- start 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 -- start start 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.511+0000 7fcf2f625640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf281981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28101fc0 0x7fcf281986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28102980 0x7fcf2819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810ca20 con 0x7fcf28100e70 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810c8a0 con 0x7fcf28102980 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2810cba0 con 0x7fcf28101fc0 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf281981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf281981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37386/0 (socket says 172.21.15.79:37386) 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 -- 172.21.15.79:0/2764123802 learned_addr learned my addr 172.21.15.79:0/2764123802 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:54.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 -- 172.21.15.79:0/2764123802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28101fc0 msgr2=0x7fcf281986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2dbc2640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28102980 0x7fcf2819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28101fc0 0x7fcf281986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 -- 172.21.15.79:0/2764123802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28102980 msgr2=0x7fcf2819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28102980 0x7fcf2819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 -- 172.21.15.79:0/2764123802 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2819fdd0 con 0x7fcf28100e70 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2dbc2640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28102980 0x7fcf2819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2d3c1640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf281981b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcf18036040 tx=0x7fcf180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18043020 con 0x7fcf28100e70 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf180054c0 con 0x7fcf28100e70 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf281a0060 con 0x7fcf28100e70 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18003db0 con 0x7fcf28100e70 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.512+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf281a0630 con 0x7fcf28100e70 2024-04-22T06:41:54.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.513+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf281a0830 con 0x7fcf28100e70 2024-04-22T06:41:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.514+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcf18046020 con 0x7fcf28100e70 2024-04-22T06:41:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.515+0000 7fcf1e7fc640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcf140a3bb0 0x7fcf140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.515+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf180f28b0 con 0x7fcf28100e70 2024-04-22T06:41:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.515+0000 7fcf2cbc0640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcf140a3bb0 0x7fcf140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:54.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.515+0000 7fcf2cbc0640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcf140a3bb0 0x7fcf140a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf281996c0 tx=0x7fcf24005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:54.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.518+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf1804d020 con 0x7fcf28100e70 2024-04-22T06:41:54.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.742+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf281a0c00 con 0x7fcf28100e70 2024-04-22T06:41:54.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.742+0000 7fcf1e7fc640 1 -- 172.21.15.79:0/2764123802 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fcf180bf730 con 0x7fcf28100e70 2024-04-22T06:41:54.741 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:54.741 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:41:54.741 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 4 2024-04-22T06:41:54.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.745+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcf140a3bb0 msgr2=0x7fcf140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:54.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.745+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcf140a3bb0 0x7fcf140a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf281996c0 tx=0x7fcf24005c90 comp rx=0 tx=0).stop 2024-04-22T06:41:54.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.745+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 msgr2=0x7fcf281981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:54.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.745+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf281981b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcf18036040 tx=0x7fcf180042f0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 shutdown_connections 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcf140a3bb0 0x7fcf140a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcf28102980 0x7fcf2819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28101fc0 0x7fcf281986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 --2- 172.21.15.79:0/2764123802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcf28100e70 0x7fcf281981b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 >> 172.21.15.79:0/2764123802 conn(0x7fcf280fc3e0 msgr2=0x7fcf280fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 shutdown_connections 2024-04-22T06:41:54.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:54.746+0000 7fcf2f625640 1 -- 172.21.15.79:0/2764123802 wait complete. 2024-04-22T06:41:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:54 smithi193 ceph-mon[31815]: Deploying daemon mds.d on smithi079 2024-04-22T06:41:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:54 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.79:6834/2324822974,v1:172.21.15.79:6835/2324822974] up:boot 2024-04-22T06:41:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:54 smithi193 ceph-mon[31815]: fsmap 3 up:standby 2024-04-22T06:41:54.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:54 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:41:54.981 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:54 smithi079 ceph-mon[27946]: Deploying daemon mds.d on smithi079 2024-04-22T06:41:54.981 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:54 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.79:6834/2324822974,v1:172.21.15.79:6835/2324822974] up:boot 2024-04-22T06:41:54.981 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:54 smithi079 ceph-mon[27946]: fsmap 3 up:standby 2024-04-22T06:41:54.981 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:54 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:41:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:54 smithi190 ceph-mon[31421]: Deploying daemon mds.d on smithi079 2024-04-22T06:41:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:54 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.79:6834/2324822974,v1:172.21.15.79:6835/2324822974] up:boot 2024-04-22T06:41:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:54 smithi190 ceph-mon[31421]: fsmap 3 up:standby 2024-04-22T06:41:55.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:54 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:41:55.240 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 systemd[1]: Starting Ceph mds.d for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:55.630 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 podman[65957]: 2024-04-22 06:41:55.276145802 +0000 UTC m=+0.018087016 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:55.630 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 podman[65957]: 2024-04-22 06:41:55.404345248 +0000 UTC m=+0.146286450 container create 833ef07e27f13a3f06a3e92d217ffa23361b80afb12f332759eaeedaae470cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T06:41:55.630 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 podman[65957]: 2024-04-22 06:41:55.63208617 +0000 UTC m=+0.374027372 container init 833ef07e27f13a3f06a3e92d217ffa23361b80afb12f332759eaeedaae470cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T06:41:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:55 smithi079 ceph-mon[27946]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:55 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2764123802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:55.991 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 podman[65957]: 2024-04-22 06:41:55.634790134 +0000 UTC m=+0.376731340 container start 833ef07e27f13a3f06a3e92d217ffa23361b80afb12f332759eaeedaae470cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:41:55.991 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 ceph-mds[65975]: -- 172.21.15.79:0/374201906 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dead291a40 con 0x55dead265000 2024-04-22T06:41:55.991 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d[65971]: starting mds.d at 2024-04-22T06:41:55.991 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 bash[65957]: 833ef07e27f13a3f06a3e92d217ffa23361b80afb12f332759eaeedaae470cb9 2024-04-22T06:41:55.991 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:41:55 smithi079 systemd[1]: Started Ceph mds.d for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:56.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:55 smithi193 ceph-mon[31815]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:56.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:55 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2764123802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:56.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:55 smithi190 ceph-mon[31421]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:56.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:55 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2764123802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:56.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 -- 172.21.15.79:0/2775530734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 msgr2=0x7f05f410a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/2775530734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f410a470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009cd0 tx=0x7f05ec02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 -- 172.21.15.79:0/2775530734 shutdown_connections 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/2775530734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f410a470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/2775530734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05f4103220 0x7f05f41036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/2775530734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 0x7f05f4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 -- 172.21.15.79:0/2775530734 >> 172.21.15.79:0/2775530734 conn(0x7f05f40fbb80 msgr2=0x7f05f40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 -- 172.21.15.79:0/2775530734 shutdown_connections 2024-04-22T06:41:56.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.874+0000 7f05f8fa4640 1 -- 172.21.15.79:0/2775530734 wait complete. 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 Processor -- start 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 -- start start 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 0x7f05f4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05f4103220 0x7f05f4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f05f410daf0 con 0x7f05f4103be0 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05f410d970 con 0x7f05f4100250 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f8fa4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05f410dc70 con 0x7f05f4103220 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f2d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f2d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37422/0 (socket says 172.21.15.79:37422) 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f2d9d640 1 -- 172.21.15.79:0/945356285 learned_addr learned my addr 172.21.15.79:0/945356285 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05ebfff640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05f4103220 0x7f05f4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:56.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.875+0000 7f05f259c640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 0x7f05f4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f2d9d640 1 -- 172.21.15.79:0/945356285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05f4103220 msgr2=0x7f05f4198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f2d9d640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05f4103220 0x7f05f4198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f2d9d640 1 -- 172.21.15.79:0/945356285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 msgr2=0x7f05f4198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f2d9d640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 0x7f05f4198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f2d9d640 1 -- 172.21.15.79:0/945356285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f419fd20 con 0x7f05f4103be0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f259c640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 0x7f05f4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f2d9d640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f419f6b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009ca0 tx=0x7f05ec0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec041020 con 0x7f05f4103be0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05ec0327c0 con 0x7f05f4103be0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f419ffb0 con 0x7f05f4103be0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec039040 con 0x7f05f4103be0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.876+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f41a0410 con 0x7f05f4103be0 2024-04-22T06:41:56.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.877+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f4109350 con 0x7f05f4103be0 2024-04-22T06:41:56.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.878+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f05ec044020 con 0x7f05f4103be0 2024-04-22T06:41:56.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.879+0000 7f05eb7fe640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f05e00a3bb0 0x7f05e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:56.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.879+0000 7f05f259c640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f05e00a3bb0 0x7f05e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:56.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.879+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f05ec0f1dd0 con 0x7f05f4103be0 2024-04-22T06:41:56.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.879+0000 7f05f259c640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f05e00a3bb0 0x7f05e00a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f05f406a4c0 tx=0x7f05e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:56.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:56.883+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05ec04b020 con 0x7f05f4103be0 2024-04-22T06:41:57.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.101+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05f402d770 con 0x7f05f4103be0 2024-04-22T06:41:57.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.101+0000 7f05eb7fe640 1 -- 172.21.15.79:0/945356285 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f05ec0bec50 con 0x7f05f4103be0 2024-04-22T06:41:57.100 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:57.100 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:41:57.100 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 5 2024-04-22T06:41:57.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.103+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f05e00a3bb0 msgr2=0x7f05e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.103+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f05e00a3bb0 0x7f05e00a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f05f406a4c0 tx=0x7f05e4005f90 comp rx=0 tx=0).stop 2024-04-22T06:41:57.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.103+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 msgr2=0x7f05f419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.103+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f419f6b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009ca0 tx=0x7f05ec0346d0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 shutdown_connections 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f05e00a3bb0 0x7f05e00a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05f4103be0 0x7f05f419f6b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05f4103220 0x7f05f4198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 --2- 172.21.15.79:0/945356285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05f4100250 0x7f05f4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 >> 172.21.15.79:0/945356285 conn(0x7f05f40fbb80 msgr2=0x7f05f4106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 shutdown_connections 2024-04-22T06:41:57.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:57.104+0000 7f05f8fa4640 1 -- 172.21.15.79:0/945356285 wait complete. 2024-04-22T06:41:57.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.79:6836/2658402003,v1:172.21.15.79:6837/2658402003] up:boot 2024-04-22T06:41:57.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: fsmap 4 up:standby 2024-04-22T06:41:57.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:57.113 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:56 smithi079 ceph-mon[27946]: Deploying daemon mds.e on smithi190 2024-04-22T06:41:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.79:6836/2658402003,v1:172.21.15.79:6837/2658402003] up:boot 2024-04-22T06:41:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: fsmap 4 up:standby 2024-04-22T06:41:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:57.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:56 smithi193 ceph-mon[31815]: Deploying daemon mds.e on smithi190 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.79:6836/2658402003,v1:172.21.15.79:6837/2658402003] up:boot 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: fsmap 4 up:standby 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:57.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:56 smithi190 ceph-mon[31421]: Deploying daemon mds.e on smithi190 2024-04-22T06:41:58.185 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:57 smithi190 ceph-mon[31421]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:58.185 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:57 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/945356285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:58.185 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:57 smithi190 systemd[1]: Starting Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:58.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:57 smithi193 ceph-mon[31815]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:58.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:57 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/945356285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:57 smithi079 ceph-mon[27946]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:57 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/945356285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:58.453 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 podman[61707]: 2024-04-22 06:41:58.090222095 +0000 UTC m=+0.018263466 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:58.453 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 podman[61707]: 2024-04-22 06:41:58.207586949 +0000 UTC m=+0.135628310 container create 5330aa7da9cc102c4e28ac41a1489c0f32d1ba1bd658f30a29a453d85ace5731 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T06:41:58.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 podman[61707]: 2024-04-22 06:41:58.460209371 +0000 UTC m=+0.388250732 container init 5330aa7da9cc102c4e28ac41a1489c0f32d1ba1bd658f30a29a453d85ace5731 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:41:58.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 podman[61707]: 2024-04-22 06:41:58.463325617 +0000 UTC m=+0.391366979 container start 5330aa7da9cc102c4e28ac41a1489c0f32d1ba1bd658f30a29a453d85ace5731 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:41:58.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 ceph-mds[61725]: -- 172.21.15.190:0/2440848978 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a58d77a40 con 0x557a58d4b000 2024-04-22T06:41:58.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: starting mds.e at 2024-04-22T06:41:58.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 bash[61707]: 5330aa7da9cc102c4e28ac41a1489c0f32d1ba1bd658f30a29a453d85ace5731 2024-04-22T06:41:58.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:41:58 smithi190 systemd[1]: Started Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:59.114 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:59.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 -- 172.21.15.79:0/3014577259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24102980 msgr2=0x7f7a241092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 --2- 172.21.15.79:0/3014577259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24102980 0x7f7a241092a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009ca0 tx=0x7f7a1c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 -- 172.21.15.79:0/3014577259 shutdown_connections 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 --2- 172.21.15.79:0/3014577259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24102980 0x7f7a241092a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 --2- 172.21.15.79:0/3014577259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a24101fc0 0x7f7a24102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 --2- 172.21.15.79:0/3014577259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24100e70 0x7f7a24101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 -- 172.21.15.79:0/3014577259 >> 172.21.15.79:0/3014577259 conn(0x7f7a240fc3e0 msgr2=0x7f7a240fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 -- 172.21.15.79:0/3014577259 shutdown_connections 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 -- 172.21.15.79:0/3014577259 wait complete. 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 Processor -- start 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 -- start start 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.227+0000 7f7a2b550640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 0x7f7a24069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24101fc0 0x7f7a2419f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a24102980 0x7f7a241a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2410c9a0 con 0x7f7a24100e70 2024-04-22T06:41:59.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2410c820 con 0x7f7a24101fc0 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2410cb20 con 0x7f7a24102980 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 0x7f7a24069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 0x7f7a24069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37442/0 (socket says 172.21.15.79:37442) 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 -- 172.21.15.79:0/1543404122 learned_addr learned my addr 172.21.15.79:0/1543404122 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a28aeb640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24101fc0 0x7f7a2419f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a29aed640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a24102980 0x7f7a241a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 -- 172.21.15.79:0/1543404122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a24102980 msgr2=0x7f7a241a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a24102980 0x7f7a241a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 -- 172.21.15.79:0/1543404122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24101fc0 msgr2=0x7f7a2419f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24101fc0 0x7f7a2419f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 -- 172.21.15.79:0/1543404122 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a241a1f40 con 0x7f7a24100e70 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a28aeb640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24101fc0 0x7f7a2419f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:59.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a292ec640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 0x7f7a24069c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000c920 tx=0x7f7a2000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:59.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20007c70 con 0x7f7a24100e70 2024-04-22T06:41:59.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a20007e10 con 0x7f7a24100e70 2024-04-22T06:41:59.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20018820 con 0x7f7a24100e70 2024-04-22T06:41:59.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a241a21d0 con 0x7f7a24100e70 2024-04-22T06:41:59.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.228+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a241a2720 con 0x7f7a24100e70 2024-04-22T06:41:59.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.229+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a24108120 con 0x7f7a24100e70 2024-04-22T06:41:59.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.230+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7a2000f450 con 0x7f7a24100e70 2024-04-22T06:41:59.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.231+0000 7f7a1a7fc640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7a100a3be0 0x7f7a100a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.231+0000 7f7a28aeb640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7a100a3be0 0x7f7a100a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:59.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.231+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7a200d0120 con 0x7f7a24100e70 2024-04-22T06:41:59.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.231+0000 7f7a28aeb640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7a100a3be0 0x7f7a100a60a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7a14005d00 tx=0x7f7a14005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:59.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.235+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a20014030 con 0x7f7a24100e70 2024-04-22T06:41:59.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.456+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a2402d5c0 con 0x7f7a24100e70 2024-04-22T06:41:59.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.456+0000 7f7a1a7fc640 1 -- 172.21.15.79:0/1543404122 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f7a2001f300 con 0x7f7a24100e70 2024-04-22T06:41:59.456 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:41:59.456 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T06:41:59.456 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 6 2024-04-22T06:41:59.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7a100a3be0 msgr2=0x7f7a100a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7a100a3be0 0x7f7a100a60a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7a14005d00 tx=0x7f7a14005ed0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 msgr2=0x7f7a24069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 0x7f7a24069c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000c920 tx=0x7f7a2000cde0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 shutdown_connections 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7a100a3be0 0x7f7a100a60a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a24101fc0 0x7f7a2419f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a24102980 0x7f7a241a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 --2- 172.21.15.79:0/1543404122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a24100e70 0x7f7a24069c50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 >> 172.21.15.79:0/1543404122 conn(0x7f7a240fc3e0 msgr2=0x7f7a241052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 shutdown_connections 2024-04-22T06:41:59.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:41:59.458+0000 7f7a2b550640 1 -- 172.21.15.79:0/1543404122 wait complete. 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: Deploying daemon mds.f on smithi193 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:boot 2024-04-22T06:41:59.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: fsmap 5 up:standby 2024-04-22T06:41:59.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:41:59.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:41:59 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1543404122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: Deploying daemon mds.f on smithi193 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:boot 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: fsmap 5 up:standby 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:41:59.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:41:59 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1543404122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: Deploying daemon mds.f on smithi193 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:boot 2024-04-22T06:41:59.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: fsmap 5 up:standby 2024-04-22T06:41:59.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:41:59.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:41:59 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1543404122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:00.938 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:00 smithi193 systemd[1]: Starting Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:00.938 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:00 smithi193 podman[62613]: 2024-04-22 06:42:00.842725308 +0000 UTC m=+0.017994815 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:01.189 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:00 smithi193 podman[62613]: 2024-04-22 06:42:00.955129491 +0000 UTC m=+0.130398982 container create 430ad37b311b4b74ef9d0dfeb83c4e68e03f87f133c303d8b1a590477c3718fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0) 2024-04-22T06:42:01.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:01 smithi193 podman[62613]: 2024-04-22 06:42:01.182898723 +0000 UTC m=+0.358168214 container init 430ad37b311b4b74ef9d0dfeb83c4e68e03f87f133c303d8b1a590477c3718fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T06:42:01.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:01 smithi193 podman[62613]: 2024-04-22 06:42:01.186577618 +0000 UTC m=+0.361847121 container start 430ad37b311b4b74ef9d0dfeb83c4e68e03f87f133c303d8b1a590477c3718fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:42:01.469 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:01.486 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mds[62633]: -- 172.21.15.193:0/3403402674 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55858c0b9a40 con 0x55858c08d000 2024-04-22T06:42:01.486 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: starting mds.f at 2024-04-22T06:42:01.487 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:01 smithi193 bash[62613]: 430ad37b311b4b74ef9d0dfeb83c4e68e03f87f133c303d8b1a590477c3718fb 2024-04-22T06:42:01.487 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:42:01 smithi193 systemd[1]: Started Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 -- 172.21.15.79:0/2959419351 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 msgr2=0x7f102410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 --2- 172.21.15.79:0/2959419351 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102410a3e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f101c009ca0 tx=0x7f101c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 -- 172.21.15.79:0/2959419351 shutdown_connections 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 --2- 172.21.15.79:0/2959419351 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102410a3e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 --2- 172.21.15.79:0/2959419351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1024103190 0x7f1024103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 --2- 172.21.15.79:0/2959419351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1024101fa0 0x7f10241023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 -- 172.21.15.79:0/2959419351 >> 172.21.15.79:0/2959419351 conn(0x7f10240fd530 msgr2=0x7f10240ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 -- 172.21.15.79:0/2959419351 shutdown_connections 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.577+0000 7f102a967640 1 -- 172.21.15.79:0/2959419351 wait complete. 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 Processor -- start 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 -- start start 2024-04-22T06:42:01.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1024101fa0 0x7f1024198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1024103190 0x7f1024198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f102410dae0 con 0x7f1024103b50 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f102410d960 con 0x7f1024101fa0 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f102a967640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f102410dc60 con 0x7f1024103190 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37460/0 (socket says 172.21.15.79:37460) 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 -- 172.21.15.79:0/2004319244 learned_addr learned my addr 172.21.15.79:0/2004319244 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f10237fe640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1024103190 0x7f1024198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1023fff640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1024101fa0 0x7f1024198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 -- 172.21.15.79:0/2004319244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1024103190 msgr2=0x7f1024198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1024103190 0x7f1024198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 -- 172.21.15.79:0/2004319244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1024101fa0 msgr2=0x7f1024198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1024101fa0 0x7f1024198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.578+0000 7f1028f04640 1 -- 172.21.15.79:0/2004319244 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102419fd20 con 0x7f1024103b50 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.579+0000 7f1028f04640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102419f6b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f101c02c700 tx=0x7f101c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.579+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c041020 con 0x7f1024103b50 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.579+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f101c034ca0 con 0x7f1024103b50 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.579+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102419ffb0 con 0x7f1024103b50 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.579+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c039040 con 0x7f1024103b50 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.579+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10241a0410 con 0x7f1024103b50 2024-04-22T06:42:01.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.580+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1024109310 con 0x7f1024103b50 2024-04-22T06:42:01.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.581+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f101c044020 con 0x7f1024103b50 2024-04-22T06:42:01.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.581+0000 7f10217fa640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f100c0a3bb0 0x7f100c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:01.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.581+0000 7f1023fff640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f100c0a3bb0 0x7f100c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:01.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.582+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f101c0f2100 con 0x7f1024103b50 2024-04-22T06:42:01.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.582+0000 7f1023fff640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f100c0a3bb0 0x7f100c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1024102ff0 tx=0x7f1014005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:01.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.585+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f101c04b020 con 0x7f1024103b50 2024-04-22T06:42:01.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.802+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f102402d5c0 con 0x7f1024103b50 2024-04-22T06:42:01.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.803+0000 7f10217fa640 1 -- 172.21.15.79:0/2004319244 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f101c0bef80 con 0x7f1024103b50 2024-04-22T06:42:01.802 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:01.802 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T06:42:01.802 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 7 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f100c0a3bb0 msgr2=0x7f100c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f100c0a3bb0 0x7f100c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1024102ff0 tx=0x7f1014005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 msgr2=0x7f102419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102419f6b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f101c02c700 tx=0x7f101c0346d0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 shutdown_connections 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f100c0a3bb0 0x7f100c0a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1024103b50 0x7f102419f6b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1024103190 0x7f1024198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 --2- 172.21.15.79:0/2004319244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1024101fa0 0x7f1024198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 >> 172.21.15.79:0/2004319244 conn(0x7f10240fd530 msgr2=0x7f10241064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 shutdown_connections 2024-04-22T06:42:01.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:01.805+0000 7f102a967640 1 -- 172.21.15.79:0/2004319244 wait complete. 2024-04-22T06:42:01.813 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:01.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:01.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:01.814 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:01 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:01.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:01.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:01 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:01.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:01 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:02 smithi190 ceph-mon[31421]: Deploying daemon mds.c on smithi193 2024-04-22T06:42:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:02 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:boot 2024-04-22T06:42:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:02 smithi190 ceph-mon[31421]: fsmap 6 up:standby 2024-04-22T06:42:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:02 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:42:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:02 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2004319244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:02 smithi079 ceph-mon[27946]: Deploying daemon mds.c on smithi193 2024-04-22T06:42:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:02 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:boot 2024-04-22T06:42:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:02 smithi079 ceph-mon[27946]: fsmap 6 up:standby 2024-04-22T06:42:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:02 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:42:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:02 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2004319244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:03.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:02 smithi193 ceph-mon[31815]: Deploying daemon mds.c on smithi193 2024-04-22T06:42:03.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:02 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:boot 2024-04-22T06:42:03.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:02 smithi193 ceph-mon[31815]: fsmap 6 up:standby 2024-04-22T06:42:03.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:02 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:42:03.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:02 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2004319244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:03.690 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:03 smithi193 systemd[1]: Starting Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:03.816 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:03.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 -- 172.21.15.79:0/15164229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198105370 msgr2=0x7f5198107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:03.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 --2- 172.21.15.79:0/15164229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198105370 0x7f5198107760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f518c009900 tx=0x7f518c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:03.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 -- 172.21.15.79:0/15164229 shutdown_connections 2024-04-22T06:42:03.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 --2- 172.21.15.79:0/15164229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5198107ca0 0x7f519810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:03.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 --2- 172.21.15.79:0/15164229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198105370 0x7f5198107760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:03.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 --2- 172.21.15.79:0/15164229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5198102a40 0x7f5198104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 -- 172.21.15.79:0/15164229 >> 172.21.15.79:0/15164229 conn(0x7f51980fc470 msgr2=0x7f51980fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 -- 172.21.15.79:0/15164229 shutdown_connections 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.928+0000 7f519cd66640 1 -- 172.21.15.79:0/15164229 wait complete. 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 Processor -- start 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 -- start start 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5198102a40 0x7f51981981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5198105370 0x7f5198198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 0x7f519819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:03.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f519810db00 con 0x7f5198107ca0 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f519810d980 con 0x7f5198102a40 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519cd66640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f519810dc80 con 0x7f5198105370 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 0x7f519819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 0x7f519819f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43300/0 (socket says 172.21.15.79:43300) 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 -- 172.21.15.79:0/3400177537 learned_addr learned my addr 172.21.15.79:0/3400177537 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5195d9b640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5198105370 0x7f5198198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f519659c640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5198102a40 0x7f51981981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 -- 172.21.15.79:0/3400177537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5198105370 msgr2=0x7f5198198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5198105370 0x7f5198198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 -- 172.21.15.79:0/3400177537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5198102a40 msgr2=0x7f51981981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5198102a40 0x7f51981981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.929+0000 7f5196d9d640 1 -- 172.21.15.79:0/3400177537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519819fde0 con 0x7f5198107ca0 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.930+0000 7f5196d9d640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 0x7f519819f770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f518800e970 tx=0x7f518800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:03.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.930+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518800ccc0 con 0x7f5198107ca0 2024-04-22T06:42:03.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.930+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51981a00d0 con 0x7f5198107ca0 2024-04-22T06:42:03.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.930+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f518800ce60 con 0x7f5198107ca0 2024-04-22T06:42:03.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.930+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188010430 con 0x7f5198107ca0 2024-04-22T06:42:03.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.930+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51981a05f0 con 0x7f5198107ca0 2024-04-22T06:42:03.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.931+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51980fdf60 con 0x7f5198107ca0 2024-04-22T06:42:03.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.932+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5188004ff0 con 0x7f5198107ca0 2024-04-22T06:42:03.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.932+0000 7f51877fe640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51800a3bb0 0x7f51800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:03.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.932+0000 7f519659c640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51800a3bb0 0x7f51800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:03.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.932+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f51880d0400 con 0x7f5198107ca0 2024-04-22T06:42:03.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.933+0000 7f519659c640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51800a3bb0 0x7f51800a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5190005fd0 tx=0x7f5190009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:03.937+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f518809d280 con 0x7f5198107ca0 2024-04-22T06:42:03.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:03 smithi190 ceph-mon[31421]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:03.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:03 smithi079 ceph-mon[27946]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:04.035 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:03 smithi193 ceph-mon[31815]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:04.035 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:03 smithi193 podman[62936]: 2024-04-22 06:42:03.669901495 +0000 UTC m=+0.017908841 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:04.035 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:03 smithi193 podman[62936]: 2024-04-22 06:42:03.810202048 +0000 UTC m=+0.158209381 container create 98f45dbf9d45b84c73f7b520f13368b1ef1def3fc964449bbdd2bdc4f321682d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:42:04.036 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:04 smithi193 podman[62936]: 2024-04-22 06:42:04.037889072 +0000 UTC m=+0.385896405 container init 98f45dbf9d45b84c73f7b520f13368b1ef1def3fc964449bbdd2bdc4f321682d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:42:04.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.152+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5198062b50 con 0x7f5198107ca0 2024-04-22T06:42:04.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.153+0000 7f51877fe640 1 -- 172.21.15.79:0/3400177537 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f518809cb80 con 0x7f5198107ca0 2024-04-22T06:42:04.151 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:04.151 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T06:42:04.152 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 7 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.154+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51800a3bb0 msgr2=0x7f51800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.154+0000 7f519cd66640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51800a3bb0 0x7f51800a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5190005fd0 tx=0x7f5190009500 comp rx=0 tx=0).stop 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.154+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 msgr2=0x7f519819f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.154+0000 7f519cd66640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 0x7f519819f770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f518800e970 tx=0x7f518800ed70 comp rx=0 tx=0).stop 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 shutdown_connections 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f51800a3bb0 0x7f51800a6070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5198107ca0 0x7f519819f770 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5198105370 0x7f5198198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 --2- 172.21.15.79:0/3400177537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5198102a40 0x7f51981981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 >> 172.21.15.79:0/3400177537 conn(0x7f51980fc470 msgr2=0x7f5198103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:04.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 shutdown_connections 2024-04-22T06:42:04.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:04.155+0000 7f519cd66640 1 -- 172.21.15.79:0/3400177537 wait complete. 2024-04-22T06:42:04.439 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:04 smithi193 podman[62936]: 2024-04-22 06:42:04.040550324 +0000 UTC m=+0.388557665 container start 98f45dbf9d45b84c73f7b520f13368b1ef1def3fc964449bbdd2bdc4f321682d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T06:42:04.440 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mds[62955]: -- 172.21.15.193:0/751228028 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba17e4da40 con 0x55ba17e21000 2024-04-22T06:42:04.440 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: starting mds.c at 2024-04-22T06:42:04.440 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:04 smithi193 bash[62936]: 98f45dbf9d45b84c73f7b520f13368b1ef1def3fc964449bbdd2bdc4f321682d 2024-04-22T06:42:04.440 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:42:04 smithi193 systemd[1]: Started Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:04.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3400177537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:04.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:04.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:04.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:04.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:04.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:04.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:04 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3400177537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:05.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:04 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:05.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3400177537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:05.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:05.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:05.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:05.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:05.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:05.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:04 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:05.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:05 smithi190 ceph-mon[31421]: Deploying daemon mds.h on smithi190 2024-04-22T06:42:05.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:05 smithi190 ceph-mon[31421]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:05.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:05 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6836/678626858,v1:172.21.15.193:6837/678626858] up:boot 2024-04-22T06:42:05.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:05 smithi190 ceph-mon[31421]: fsmap 7 up:standby 2024-04-22T06:42:05.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:05 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:42:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:05 smithi079 ceph-mon[27946]: Deploying daemon mds.h on smithi190 2024-04-22T06:42:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:05 smithi079 ceph-mon[27946]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:05 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6836/678626858,v1:172.21.15.193:6837/678626858] up:boot 2024-04-22T06:42:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:05 smithi079 ceph-mon[27946]: fsmap 7 up:standby 2024-04-22T06:42:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:05 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:42:06.165 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:05 smithi193 ceph-mon[31815]: Deploying daemon mds.h on smithi190 2024-04-22T06:42:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:05 smithi193 ceph-mon[31815]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:05 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6836/678626858,v1:172.21.15.193:6837/678626858] up:boot 2024-04-22T06:42:06.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:05 smithi193 ceph-mon[31815]: fsmap 7 up:standby 2024-04-22T06:42:06.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:05 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:42:06.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 -- 172.21.15.79:0/3274431006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103190 msgr2=0x7f991c103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:06.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 --2- 172.21.15.79:0/3274431006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103190 0x7f991c103610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9910009900 tx=0x7f991002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:06.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 -- 172.21.15.79:0/3274431006 shutdown_connections 2024-04-22T06:42:06.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 --2- 172.21.15.79:0/3274431006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c103b50 0x7f991c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 --2- 172.21.15.79:0/3274431006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103190 0x7f991c103610 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 --2- 172.21.15.79:0/3274431006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 0x7f991c1023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 -- 172.21.15.79:0/3274431006 >> 172.21.15.79:0/3274431006 conn(0x7f991c0fd530 msgr2=0x7f991c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 -- 172.21.15.79:0/3274431006 shutdown_connections 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 -- 172.21.15.79:0/3274431006 wait complete. 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.277+0000 7f992217e640 1 Processor -- start 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 -- start start 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 0x7f991c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c103190 0x7f991c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 0x7f991c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f991c10dae0 con 0x7f991c103b50 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f991c10d960 con 0x7f991c103190 2024-04-22T06:42:06.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f992217e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f991c10dc60 con 0x7f991c101fa0 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 0x7f991c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 0x7f991c074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43324/0 (socket says 172.21.15.79:43324) 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 -- 172.21.15.79:0/58456250 learned_addr learned my addr 172.21.15.79:0/58456250 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991affd640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c103190 0x7f991c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991b7fe640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 0x7f991c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 -- 172.21.15.79:0/58456250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 msgr2=0x7f991c075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 0x7f991c075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 -- 172.21.15.79:0/58456250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c103190 msgr2=0x7f991c074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c103190 0x7f991c074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 -- 172.21.15.79:0/58456250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991c075110 con 0x7f991c103b50 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991b7fe640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 0x7f991c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.278+0000 7f991bfff640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 0x7f991c074880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f991400eb10 tx=0x7f991400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.279+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9914004b20 con 0x7f991c103b50 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.279+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991c1a2270 con 0x7f991c103b50 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.279+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9914004cc0 con 0x7f991c103b50 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.279+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9914010670 con 0x7f991c103b50 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.279+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991c1a2740 con 0x7f991c103b50 2024-04-22T06:42:06.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.280+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991c069230 con 0x7f991c103b50 2024-04-22T06:42:06.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.281+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9914005020 con 0x7f991c103b50 2024-04-22T06:42:06.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.281+0000 7f9918ff9640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f99040a3bb0 0x7f99040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:06.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.281+0000 7f991b7fe640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f99040a3bb0 0x7f99040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:06.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.281+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99140d0600 con 0x7f991c103b50 2024-04-22T06:42:06.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.282+0000 7f991b7fe640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f99040a3bb0 0x7f99040a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f991c102ff0 tx=0x7f990c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:06.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.286+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f991409d480 con 0x7f991c103b50 2024-04-22T06:42:06.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.503+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f991c1092c0 con 0x7f991c103b50 2024-04-22T06:42:06.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.503+0000 7f9918ff9640 1 -- 172.21.15.79:0/58456250 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f991409cd80 con 0x7f991c103b50 2024-04-22T06:42:06.502 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:06.502 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T06:42:06.503 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 8 2024-04-22T06:42:06.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f99040a3bb0 msgr2=0x7f99040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f99040a3bb0 0x7f99040a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f991c102ff0 tx=0x7f990c005d30 comp rx=0 tx=0).stop 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 msgr2=0x7f991c074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 0x7f991c074880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f991400eb10 tx=0x7f991400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 shutdown_connections 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f99040a3bb0 0x7f99040a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f991c103190 0x7f991c074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f991c103b50 0x7f991c074880 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 --2- 172.21.15.79:0/58456250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f991c101fa0 0x7f991c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 >> 172.21.15.79:0/58456250 conn(0x7f991c0fd530 msgr2=0x7f991c106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 shutdown_connections 2024-04-22T06:42:06.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:06.506+0000 7f992217e640 1 -- 172.21.15.79:0/58456250 wait complete. 2024-04-22T06:42:06.703 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:06 smithi190 systemd[1]: Starting Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:06.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:06 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/58456250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:06.989 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:06 smithi190 podman[62012]: 2024-04-22 06:42:06.607619669 +0000 UTC m=+0.018220207 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:06.989 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:06 smithi190 podman[62012]: 2024-04-22 06:42:06.729691884 +0000 UTC m=+0.140292404 container create 443444e919816eb05088a7336453e5d206b4ff4935db23a0bf4fd3a0a904a366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:42:06.989 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:06 smithi190 podman[62012]: 2024-04-22 06:42:06.99105081 +0000 UTC m=+0.401651330 container init 443444e919816eb05088a7336453e5d206b4ff4935db23a0bf4fd3a0a904a366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T06:42:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:06 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/58456250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:06 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/58456250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:07.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:06 smithi190 podman[62012]: 2024-04-22 06:42:06.99368957 +0000 UTC m=+0.404290091 container start 443444e919816eb05088a7336453e5d206b4ff4935db23a0bf4fd3a0a904a366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:42:07.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mds[62031]: -- 172.21.15.190:0/2386139001 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55923427ba40 con 0x55923424f000 2024-04-22T06:42:07.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: starting mds.h at 2024-04-22T06:42:07.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:07 smithi190 bash[62012]: 443444e919816eb05088a7336453e5d206b4ff4935db23a0bf4fd3a0a904a366 2024-04-22T06:42:07.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:42:07 smithi190 systemd[1]: Started Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:07.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:07.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:07.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:07.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:07.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:07.967 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:07.967 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:07 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:07.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:07 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:08.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:07 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:08.516 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:08.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.626+0000 7f81c42ba640 1 -- 172.21.15.79:0/2002985094 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc100e80 msgr2=0x7f81bc1012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:08.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.626+0000 7f81c42ba640 1 --2- 172.21.15.79:0/2002985094 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc100e80 0x7f81bc1012a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009900 tx=0x7f81b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:08.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- 172.21.15.79:0/2002985094 shutdown_connections 2024-04-22T06:42:08.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 --2- 172.21.15.79:0/2002985094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81bc101fd0 0x7f81bc102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 --2- 172.21.15.79:0/2002985094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc102990 0x7f81bc1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 --2- 172.21.15.79:0/2002985094 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc100e80 0x7f81bc1012a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- 172.21.15.79:0/2002985094 >> 172.21.15.79:0/2002985094 conn(0x7f81bc0fc410 msgr2=0x7f81bc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- 172.21.15.79:0/2002985094 shutdown_connections 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- 172.21.15.79:0/2002985094 wait complete. 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 Processor -- start 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- start start 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc100e80 0x7f81bc1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 0x7f81bc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81bc102990 0x7f81bc19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc10db20 con 0x7f81bc101fd0 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc10d9a0 con 0x7f81bc102990 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c42ba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc10dca0 con 0x7f81bc100e80 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c1855640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 0x7f81bc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c1855640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 0x7f81bc1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43342/0 (socket says 172.21.15.79:43342) 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.627+0000 7f81c1855640 1 -- 172.21.15.79:0/1605599852 learned_addr learned my addr 172.21.15.79:0/1605599852 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c2056640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc100e80 0x7f81bc1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:08.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c1855640 1 -- 172.21.15.79:0/1605599852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc100e80 msgr2=0x7f81bc1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c1855640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc100e80 0x7f81bc1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c1855640 1 -- 172.21.15.79:0/1605599852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81bc102990 msgr2=0x7f81bc19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c1855640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81bc102990 0x7f81bc19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c1855640 1 -- 172.21.15.79:0/1605599852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81bc19fdd0 con 0x7f81bc101fd0 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c2056640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc100e80 0x7f81bc1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c1855640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 0x7f81bc1986f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00c970 tx=0x7f81ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac004220 con 0x7f81bc101fd0 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81ac0043c0 con 0x7f81bc101fd0 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81bc1a00c0 con 0x7f81bc101fd0 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00f450 con 0x7f81bc101fd0 2024-04-22T06:42:08.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.628+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81bc1a0610 con 0x7f81bc101fd0 2024-04-22T06:42:08.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.629+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81bc108210 con 0x7f81bc101fd0 2024-04-22T06:42:08.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.630+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f81ac003610 con 0x7f81bc101fd0 2024-04-22T06:42:08.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.631+0000 7f81b37fe640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f81a80a3bb0 0x7f81a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:08.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.631+0000 7f81c2056640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f81a80a3bb0 0x7f81a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:08.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.631+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f81ac0cfe90 con 0x7f81bc101fd0 2024-04-22T06:42:08.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.632+0000 7f81c2056640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f81a80a3bb0 0x7f81a80a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f81b8006010 tx=0x7f81b802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:08.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.636+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81ac014030 con 0x7f81bc101fd0 2024-04-22T06:42:08.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.853+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81bc02d7e0 con 0x7f81bc101fd0 2024-04-22T06:42:08.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.853+0000 7f81b37fe640 1 -- 172.21.15.79:0/1605599852 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f81ac09cd10 con 0x7f81bc101fd0 2024-04-22T06:42:08.852 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:08.852 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T06:42:08.852 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 9 2024-04-22T06:42:08.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.855+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f81a80a3bb0 msgr2=0x7f81a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:08.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f81a80a3bb0 0x7f81a80a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f81b8006010 tx=0x7f81b802c7e0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 msgr2=0x7f81bc1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:08.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 0x7f81bc1986f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00c970 tx=0x7f81ac00ce30 comp rx=0 tx=0).stop 2024-04-22T06:42:08.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 shutdown_connections 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f81a80a3bb0 0x7f81a80a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81bc102990 0x7f81bc19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f81bc101fd0 0x7f81bc1986f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 --2- 172.21.15.79:0/1605599852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81bc100e80 0x7f81bc1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 >> 172.21.15.79:0/1605599852 conn(0x7f81bc0fc410 msgr2=0x7f81bc1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 shutdown_connections 2024-04-22T06:42:08.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:08.856+0000 7f81c42ba640 1 -- 172.21.15.79:0/1605599852 wait complete. 2024-04-22T06:42:08.864 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:08 smithi079 ceph-mon[27946]: Deploying daemon mds.i on smithi193 2024-04-22T06:42:08.864 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:08 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] up:boot 2024-04-22T06:42:08.864 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:08 smithi079 ceph-mon[27946]: fsmap 8 up:standby 2024-04-22T06:42:08.864 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:08 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:42:09.147 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:08 smithi193 ceph-mon[31815]: Deploying daemon mds.i on smithi193 2024-04-22T06:42:09.147 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:08 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] up:boot 2024-04-22T06:42:09.148 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:08 smithi193 ceph-mon[31815]: fsmap 8 up:standby 2024-04-22T06:42:09.148 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:08 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:42:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:08 smithi190 ceph-mon[31421]: Deploying daemon mds.i on smithi193 2024-04-22T06:42:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:08 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] up:boot 2024-04-22T06:42:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:08 smithi190 ceph-mon[31421]: fsmap 8 up:standby 2024-04-22T06:42:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:08 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:42:09.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 systemd[1]: Starting Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:09.832 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:09 smithi193 ceph-mon[31815]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:09.832 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:09 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1605599852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:09.832 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 podman[63244]: 2024-04-22 06:42:09.470777404 +0000 UTC m=+0.018225857 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:09.832 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 podman[63244]: 2024-04-22 06:42:09.590365691 +0000 UTC m=+0.137814189 container create 3bd07a4cc9decb0fd6d11a94fd700138099d527de1e364656eb3a385ed99a888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T06:42:09.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:09 smithi079 ceph-mon[27946]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:09.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:09 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1605599852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:10.189 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 podman[63244]: 2024-04-22 06:42:09.83468183 +0000 UTC m=+0.382130282 container init 3bd07a4cc9decb0fd6d11a94fd700138099d527de1e364656eb3a385ed99a888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:42:10.189 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 podman[63244]: 2024-04-22 06:42:09.837402345 +0000 UTC m=+0.384850805 container start 3bd07a4cc9decb0fd6d11a94fd700138099d527de1e364656eb3a385ed99a888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T06:42:10.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 ceph-mds[63263]: -- 172.21.15.193:0/133944107 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557513b05a40 con 0x557513ad9000 2024-04-22T06:42:10.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: starting mds.i at 2024-04-22T06:42:10.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 bash[63244]: 3bd07a4cc9decb0fd6d11a94fd700138099d527de1e364656eb3a385ed99a888 2024-04-22T06:42:10.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:42:09 smithi193 systemd[1]: Started Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:10.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:09 smithi190 ceph-mon[31421]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:10.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:09 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1605599852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:10.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:10.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 -- 172.21.15.79:0/1180267717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80102990 msgr2=0x7eff801092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:10.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 --2- 172.21.15.79:0/1180267717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80102990 0x7eff801092b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7eff78009ca0 tx=0x7eff7802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:10.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 -- 172.21.15.79:0/1180267717 shutdown_connections 2024-04-22T06:42:10.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 --2- 172.21.15.79:0/1180267717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff80101fd0 0x7eff80102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 --2- 172.21.15.79:0/1180267717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80102990 0x7eff801092b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 --2- 172.21.15.79:0/1180267717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff80100e80 0x7eff801012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 -- 172.21.15.79:0/1180267717 >> 172.21.15.79:0/1180267717 conn(0x7eff800fc410 msgr2=0x7eff800fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 -- 172.21.15.79:0/1180267717 shutdown_connections 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.973+0000 7eff879d5640 1 -- 172.21.15.79:0/1180267717 wait complete. 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 Processor -- start 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 -- start start 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff80100e80 0x7eff80197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 0x7eff80197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff80102990 0x7eff8019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff8010cad0 con 0x7eff80101fd0 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff8010c950 con 0x7eff80102990 2024-04-22T06:42:10.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff879d5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff8010cc50 con 0x7eff80100e80 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff84f70640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 0x7eff80197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff84f70640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 0x7eff80197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43360/0 (socket says 172.21.15.79:43360) 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff84f70640 1 -- 172.21.15.79:0/2675982525 learned_addr learned my addr 172.21.15.79:0/2675982525 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff85f72640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff80102990 0x7eff8019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.974+0000 7eff85771640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff80100e80 0x7eff80197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff84f70640 1 -- 172.21.15.79:0/2675982525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff80100e80 msgr2=0x7eff80197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff84f70640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff80100e80 0x7eff80197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff84f70640 1 -- 172.21.15.79:0/2675982525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff80102990 msgr2=0x7eff8019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff84f70640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff80102990 0x7eff8019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff84f70640 1 -- 172.21.15.79:0/2675982525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8019ed30 con 0x7eff80101fd0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff84f70640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 0x7eff80197650 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7eff7000d920 tx=0x7eff7000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7001b070 con 0x7eff80101fd0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff7000edc0 con 0x7eff80101fd0 2024-04-22T06:42:10.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70014730 con 0x7eff80101fd0 2024-04-22T06:42:10.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8006a6e0 con 0x7eff80101fd0 2024-04-22T06:42:10.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.975+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8006ad00 con 0x7eff80101fd0 2024-04-22T06:42:10.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.976+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff80107c00 con 0x7eff80101fd0 2024-04-22T06:42:10.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.977+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7eff7001e070 con 0x7eff80101fd0 2024-04-22T06:42:10.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.977+0000 7eff767fc640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7eff6c0a3c30 0x7eff6c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:10.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.978+0000 7eff85771640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7eff6c0a3c30 0x7eff6c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:10.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.978+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7eff700d1a30 con 0x7eff80101fd0 2024-04-22T06:42:10.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.978+0000 7eff85771640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7eff6c0a3c30 0x7eff6c0a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00a7f0 tx=0x7eff7c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:10.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:10.981+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff7009e260 con 0x7eff80101fd0 2024-04-22T06:42:11.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.200+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff80108040 con 0x7eff80101fd0 2024-04-22T06:42:11.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.201+0000 7eff767fc640 1 -- 172.21.15.79:0/2675982525 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7eff700a2560 con 0x7eff80101fd0 2024-04-22T06:42:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:11.200 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T06:42:11.200 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 10 2024-04-22T06:42:11.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7eff6c0a3c30 msgr2=0x7eff6c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:11.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7eff6c0a3c30 0x7eff6c0a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00a7f0 tx=0x7eff7c008040 comp rx=0 tx=0).stop 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 msgr2=0x7eff80197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 0x7eff80197650 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7eff7000d920 tx=0x7eff7000dde0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 shutdown_connections 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7eff6c0a3c30 0x7eff6c0a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff80102990 0x7eff8019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff80101fd0 0x7eff80197650 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 --2- 172.21.15.79:0/2675982525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff80100e80 0x7eff80197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 >> 172.21.15.79:0/2675982525 conn(0x7eff800fc410 msgr2=0x7eff80105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 shutdown_connections 2024-04-22T06:42:11.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:11.203+0000 7eff879d5640 1 -- 172.21.15.79:0/2675982525 wait complete. 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: Deploying daemon mds.g on smithi079 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] up:boot 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: fsmap 9 up:standby 2024-04-22T06:42:11.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:10 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: Deploying daemon mds.g on smithi079 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] up:boot 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: fsmap 9 up:standby 2024-04-22T06:42:11.211 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:10 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:42:11.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: Deploying daemon mds.g on smithi079 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] up:boot 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: fsmap 9 up:standby 2024-04-22T06:42:11.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:10 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:42:12.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:12 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2675982525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:12.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:12 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2675982525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:12.240 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 systemd[1]: Starting Ceph mds.g for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:12.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:12 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2675982525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:12.570 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 podman[66561]: 2024-04-22 06:42:12.202445129 +0000 UTC m=+0.018618923 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:12.570 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 podman[66561]: 2024-04-22 06:42:12.303092097 +0000 UTC m=+0.119265891 container create 23d4fa0bb5372a69dc3d50260b42408ca7a8876d0a340bfee03bc6e31f8c593b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T06:42:12.571 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 podman[66561]: 2024-04-22 06:42:12.573030464 +0000 UTC m=+0.389204261 container init 23d4fa0bb5372a69dc3d50260b42408ca7a8876d0a340bfee03bc6e31f8c593b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:42:12.990 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 podman[66561]: 2024-04-22 06:42:12.575874161 +0000 UTC m=+0.392047965 container start 23d4fa0bb5372a69dc3d50260b42408ca7a8876d0a340bfee03bc6e31f8c593b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:42:12.990 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 ceph-mds[66580]: -- 172.21.15.79:0/89634451 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586e09b9a40 con 0x5586e098c800 2024-04-22T06:42:12.990 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g[66576]: starting mds.g at 2024-04-22T06:42:12.990 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 bash[66561]: 23d4fa0bb5372a69dc3d50260b42408ca7a8876d0a340bfee03bc6e31f8c593b 2024-04-22T06:42:12.991 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:42:12 smithi079 systemd[1]: Started Ceph mds.g for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:13.213 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:13.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 -- 172.21.15.79:0/2694466670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00100e70 msgr2=0x7fab00101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:13.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 --2- 172.21.15.79:0/2694466670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00100e70 0x7fab00101290 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009900 tx=0x7faaf002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 -- 172.21.15.79:0/2694466670 shutdown_connections 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 --2- 172.21.15.79:0/2694466670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00101fc0 0x7fab00102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 --2- 172.21.15.79:0/2694466670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab00102980 0x7fab001092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 --2- 172.21.15.79:0/2694466670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00100e70 0x7fab00101290 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 -- 172.21.15.79:0/2694466670 >> 172.21.15.79:0/2694466670 conn(0x7fab000fc3e0 msgr2=0x7fab000fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 -- 172.21.15.79:0/2694466670 shutdown_connections 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.323+0000 7fab04fdb640 1 -- 172.21.15.79:0/2694466670 wait complete. 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 Processor -- start 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 -- start start 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab00100e70 0x7fab001970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 0x7fab001975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00102980 0x7fab0019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:13.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fab0010c9e0 con 0x7fab00101fc0 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fab0010c860 con 0x7fab00102980 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7fab04fdb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab0010cb60 con 0x7fab00100e70 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7faafdd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 0x7fab001975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7faafdd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 0x7fab001975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48872/0 (socket says 172.21.15.79:48872) 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7faafdd9b640 1 -- 172.21.15.79:0/3807649682 learned_addr learned my addr 172.21.15.79:0/3807649682 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7faafe59c640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab00100e70 0x7fab001970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7faafed9d640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00102980 0x7fab0019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.324+0000 7faafdd9b640 1 -- 172.21.15.79:0/3807649682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab00100e70 msgr2=0x7fab001970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faafdd9b640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab00100e70 0x7fab001970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faafdd9b640 1 -- 172.21.15.79:0/3807649682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00102980 msgr2=0x7fab0019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faafdd9b640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00102980 0x7fab0019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faafdd9b640 1 -- 172.21.15.79:0/3807649682 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0019ecd0 con 0x7fab00101fc0 2024-04-22T06:42:13.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faafed9d640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00102980 0x7fab0019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faafdd9b640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 0x7fab001975f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faaf4007920 tx=0x7faaf4007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4019070 con 0x7fab00101fc0 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaf4005020 con 0x7fab00101fc0 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab00069140 con 0x7fab00101fc0 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004a00 con 0x7fab00101fc0 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.325+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab00069640 con 0x7fab00101fc0 2024-04-22T06:42:13.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.326+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab00068de0 con 0x7fab00101fc0 2024-04-22T06:42:13.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.327+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faaf401c070 con 0x7fab00101fc0 2024-04-22T06:42:13.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.327+0000 7faaef7fe640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faae80a3bb0 0x7faae80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:13.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.328+0000 7faafe59c640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faae80a3bb0 0x7faae80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:13.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.328+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faaf40d10a0 con 0x7fab00101fc0 2024-04-22T06:42:13.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.328+0000 7faafe59c640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faae80a3bb0 0x7faae80a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009230 tx=0x7faaf0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:13.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.331+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faaf409df20 con 0x7fab00101fc0 2024-04-22T06:42:13.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.555+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab00107c70 con 0x7fab00101fc0 2024-04-22T06:42:13.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.556+0000 7faaef7fe640 1 -- 172.21.15.79:0/3807649682 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7fab00107c70 con 0x7fab00101fc0 2024-04-22T06:42:13.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:13.556 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T06:42:13.558 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 11 2024-04-22T06:42:13.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faae80a3bb0 msgr2=0x7faae80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:13.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faae80a3bb0 0x7faae80a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009230 tx=0x7faaf0005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 msgr2=0x7fab001975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:13.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 0x7fab001975f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faaf4007920 tx=0x7faaf4007de0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 shutdown_connections 2024-04-22T06:42:13.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7faae80a3bb0 0x7faae80a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab00102980 0x7fab0019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab00101fc0 0x7fab001975f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 --2- 172.21.15.79:0/3807649682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab00100e70 0x7fab001970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:13.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 >> 172.21.15.79:0/3807649682 conn(0x7fab000fc3e0 msgr2=0x7fab001053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:13.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 shutdown_connections 2024-04-22T06:42:13.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:13.558+0000 7fab04fdb640 1 -- 172.21.15.79:0/3807649682 wait complete. 2024-04-22T06:42:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:13.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: Deploying daemon mds.a on smithi079 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.79:6838/1490349330,v1:172.21.15.79:6839/1490349330] up:boot 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: fsmap 10 up:standby 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:42:13.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:13 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3807649682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: Deploying daemon mds.a on smithi079 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.79:6838/1490349330,v1:172.21.15.79:6839/1490349330] up:boot 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: fsmap 10 up:standby 2024-04-22T06:42:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:42:14.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:13 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3807649682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: Deploying daemon mds.a on smithi079 2024-04-22T06:42:14.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.79:6838/1490349330,v1:172.21.15.79:6839/1490349330] up:boot 2024-04-22T06:42:14.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: fsmap 10 up:standby 2024-04-22T06:42:14.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:42:14.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:13 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3807649682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:14.990 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:14 smithi079 systemd[1]: Starting Ceph mds.a for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:15.315 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 podman[66907]: 2024-04-22 06:42:14.953626369 +0000 UTC m=+0.017896758 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:15.315 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 podman[66907]: 2024-04-22 06:42:15.194666832 +0000 UTC m=+0.258937253 container create 4dd988509095d178e2681aeb34539a19d62ba5161abcbf80288088f5b5796292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T06:42:15.568 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 -- 172.21.15.79:0/3567333258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc101fd0 msgr2=0x7f25bc102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 --2- 172.21.15.79:0/3567333258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc101fd0 0x7f25bc102450 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009900 tx=0x7f25ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 -- 172.21.15.79:0/3567333258 shutdown_connections 2024-04-22T06:42:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 --2- 172.21.15.79:0/3567333258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25bc102990 0x7f25bc1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 --2- 172.21.15.79:0/3567333258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc101fd0 0x7f25bc102450 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 --2- 172.21.15.79:0/3567333258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc100e80 0x7f25bc1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 -- 172.21.15.79:0/3567333258 >> 172.21.15.79:0/3567333258 conn(0x7f25bc0fc410 msgr2=0x7f25bc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 -- 172.21.15.79:0/3567333258 shutdown_connections 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.674+0000 7f25c4488640 1 -- 172.21.15.79:0/3567333258 wait complete. 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 Processor -- start 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 -- start start 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 0x7f25bc06b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc101fd0 0x7f25bc19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25bc102990 0x7f25bc19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc10c9c0 con 0x7f25bc100e80 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc10c840 con 0x7f25bc102990 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c4488640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc10cb40 con 0x7f25bc101fd0 2024-04-22T06:42:15.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 0x7f25bc06b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 0x7f25bc06b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48914/0 (socket says 172.21.15.79:48914) 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 -- 172.21.15.79:0/2003115859 learned_addr learned my addr 172.21.15.79:0/2003115859 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2a25640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25bc102990 0x7f25bc19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c1a23640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc101fd0 0x7f25bc19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 -- 172.21.15.79:0/2003115859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc101fd0 msgr2=0x7f25bc19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc101fd0 0x7f25bc19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 -- 172.21.15.79:0/2003115859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25bc102990 msgr2=0x7f25bc19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25bc102990 0x7f25bc19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.675+0000 7f25c2224640 1 -- 172.21.15.79:0/2003115859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc19fd90 con 0x7f25bc100e80 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.676+0000 7f25c2224640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 0x7f25bc06b1f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25b800c920 tx=0x7f25b800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.676+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8007c70 con 0x7f25bc100e80 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.676+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25bc19ffc0 con 0x7f25bc100e80 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.676+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b8007e10 con 0x7f25bc100e80 2024-04-22T06:42:15.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.676+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8018820 con 0x7f25bc100e80 2024-04-22T06:42:15.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.676+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25bc1a0480 con 0x7f25bc100e80 2024-04-22T06:42:15.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.677+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25bc108190 con 0x7f25bc100e80 2024-04-22T06:42:15.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.678+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f25b800f450 con 0x7f25bc100e80 2024-04-22T06:42:15.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.678+0000 7f25b37fe640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f25a80a3c30 0x7f25a80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:15.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.678+0000 7f25c1a23640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f25a80a3c30 0x7f25a80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:15.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.679+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f25b80d0070 con 0x7f25bc100e80 2024-04-22T06:42:15.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.679+0000 7f25c1a23640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f25a80a3c30 0x7f25a80a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f25ac005ab0 tx=0x7f25ac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:15.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.683+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25b8014030 con 0x7f25bc100e80 2024-04-22T06:42:15.726 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 podman[66907]: 2024-04-22 06:42:15.414298812 +0000 UTC m=+0.478569191 container init 4dd988509095d178e2681aeb34539a19d62ba5161abcbf80288088f5b5796292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T06:42:15.727 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 podman[66907]: 2024-04-22 06:42:15.417015488 +0000 UTC m=+0.481285870 container start 4dd988509095d178e2681aeb34539a19d62ba5161abcbf80288088f5b5796292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0) 2024-04-22T06:42:15.727 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mds[66926]: -- 172.21.15.79:0/557239294 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55893da1ba40 con 0x55893d9ee800 2024-04-22T06:42:15.727 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a[66922]: starting mds.a at 2024-04-22T06:42:15.727 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 bash[66907]: 4dd988509095d178e2681aeb34539a19d62ba5161abcbf80288088f5b5796292 2024-04-22T06:42:15.727 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:42:15 smithi079 systemd[1]: Started Ceph mds.a for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:15.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.902+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25bc062bf0 con 0x7f25bc100e80 2024-04-22T06:42:15.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.902+0000 7f25b37fe640 1 -- 172.21.15.79:0/2003115859 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f25b809ce50 con 0x7f25bc100e80 2024-04-22T06:42:15.901 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:15.901 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T06:42:15.901 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 12 2024-04-22T06:42:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f25a80a3c30 msgr2=0x7f25a80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f25a80a3c30 0x7f25a80a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f25ac005ab0 tx=0x7f25ac005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 msgr2=0x7f25bc06b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 0x7f25bc06b1f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25b800c920 tx=0x7f25b800cde0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 shutdown_connections 2024-04-22T06:42:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f25a80a3c30 0x7f25a80a60f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25bc102990 0x7f25bc19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc101fd0 0x7f25bc19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 --2- 172.21.15.79:0/2003115859 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25bc100e80 0x7f25bc06b1f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.904+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 >> 172.21.15.79:0/2003115859 conn(0x7f25bc0fc410 msgr2=0x7f25bc105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.905+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 shutdown_connections 2024-04-22T06:42:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:15.905+0000 7f25c4488640 1 -- 172.21.15.79:0/2003115859 wait complete. 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:15.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:15 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:16.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:15 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:16.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:15 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:16.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:16 smithi190 ceph-mon[31421]: Deploying daemon mds.b on smithi190 2024-04-22T06:42:16.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:16 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.79:6840/3134754161,v1:172.21.15.79:6841/3134754161] up:boot 2024-04-22T06:42:16.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:16 smithi190 ceph-mon[31421]: fsmap 11 up:standby 2024-04-22T06:42:16.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:42:16.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:16 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2003115859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:16 smithi079 ceph-mon[27946]: Deploying daemon mds.b on smithi190 2024-04-22T06:42:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:16 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.79:6840/3134754161,v1:172.21.15.79:6841/3134754161] up:boot 2024-04-22T06:42:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:16 smithi079 ceph-mon[27946]: fsmap 11 up:standby 2024-04-22T06:42:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:42:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:16 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2003115859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:16 smithi193 ceph-mon[31815]: Deploying daemon mds.b on smithi190 2024-04-22T06:42:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:16 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.79:6840/3134754161,v1:172.21.15.79:6841/3134754161] up:boot 2024-04-22T06:42:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:16 smithi193 ceph-mon[31815]: fsmap 11 up:standby 2024-04-22T06:42:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:42:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:16 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2003115859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:17.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:17.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:17 smithi190 ceph-mon[31421]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:17.953 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:17 smithi190 systemd[1]: Starting Ceph mds.b for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:17.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:17 smithi079 ceph-mon[27946]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:18.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 -- 172.21.15.79:0/2200947677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c100ea0 msgr2=0x7f107c1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2200947677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c100ea0 0x7f107c1012c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f106c009900 tx=0x7f106c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:18.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 -- 172.21.15.79:0/2200947677 shutdown_connections 2024-04-22T06:42:18.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2200947677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f107c1029b0 0x7f107c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2200947677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 0x7f107c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2200947677 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c100ea0 0x7f107c1012c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 -- 172.21.15.79:0/2200947677 >> 172.21.15.79:0/2200947677 conn(0x7f107c0fc410 msgr2=0x7f107c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 -- 172.21.15.79:0/2200947677 shutdown_connections 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.028+0000 7f1081c8c640 1 -- 172.21.15.79:0/2200947677 wait complete. 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 Processor -- start 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 -- start start 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f107c100ea0 0x7f107c1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 0x7f107c197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 0x7f107c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f107c10c9f0 con 0x7f107c1029b0 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f107c10c870 con 0x7f107c100ea0 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f1081c8c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f107c10cb70 con 0x7f107c101ff0 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f107bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 0x7f107c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f107bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 0x7f107c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48930/0 (socket says 172.21.15.79:48930) 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f107bfff640 1 -- 172.21.15.79:0/2748596727 learned_addr learned my addr 172.21.15.79:0/2748596727 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f107affd640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 0x7f107c197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.029+0000 7f107b7fe640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f107c100ea0 0x7f107c1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107bfff640 1 -- 172.21.15.79:0/2748596727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 msgr2=0x7f107c197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107bfff640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 0x7f107c197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107bfff640 1 -- 172.21.15.79:0/2748596727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f107c100ea0 msgr2=0x7f107c1970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107bfff640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f107c100ea0 0x7f107c1970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107bfff640 1 -- 172.21.15.79:0/2748596727 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c19ece0 con 0x7f107c1029b0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107affd640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 0x7f107c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f107bfff640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 0x7f107c19e670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f107400eaa0 tx=0x7f107400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107400cdb0 con 0x7f107c1029b0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1074005020 con 0x7f107c1029b0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107c19efd0 con 0x7f107c1029b0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740116d0 con 0x7f107c1029b0 2024-04-22T06:42:18.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.030+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f107c19f440 con 0x7f107c1029b0 2024-04-22T06:42:18.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.031+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107c108200 con 0x7f107c1029b0 2024-04-22T06:42:18.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.032+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f10740051d0 con 0x7f107c1029b0 2024-04-22T06:42:18.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.033+0000 7f1078ff9640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f10680a3bb0 0x7f10680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.033+0000 7f107b7fe640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f10680a3bb0 0x7f10680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.033+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f10740d0210 con 0x7f107c1029b0 2024-04-22T06:42:18.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.033+0000 7f107b7fe640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f10680a3bb0 0x7f10680a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f106c000c00 tx=0x7f106c005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:18.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.037+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1074016030 con 0x7f107c1029b0 2024-04-22T06:42:18.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:17 smithi193 ceph-mon[31815]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:18.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.252+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f107c02d5c0 con 0x7f107c1029b0 2024-04-22T06:42:18.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.253+0000 7f1078ff9640 1 -- 172.21.15.79:0/2748596727 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f107409d090 con 0x7f107c1029b0 2024-04-22T06:42:18.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:18.252 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T06:42:18.252 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 12 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f10680a3bb0 msgr2=0x7f10680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f10680a3bb0 0x7f10680a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f106c000c00 tx=0x7f106c005be0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 msgr2=0x7f107c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 0x7f107c19e670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f107400eaa0 tx=0x7f107400ef60 comp rx=0 tx=0).stop 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 shutdown_connections 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f10680a3bb0 0x7f10680a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f107c1029b0 0x7f107c19e670 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f107c101ff0 0x7f107c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 --2- 172.21.15.79:0/2748596727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f107c100ea0 0x7f107c1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 >> 172.21.15.79:0/2748596727 conn(0x7f107c0fc410 msgr2=0x7f107c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:18.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 shutdown_connections 2024-04-22T06:42:18.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:18.255+0000 7f1081c8c640 1 -- 172.21.15.79:0/2748596727 wait complete. 2024-04-22T06:42:18.346 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 podman[62318]: 2024-04-22 06:42:17.957583781 +0000 UTC m=+0.018039378 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:18.346 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 podman[62318]: 2024-04-22 06:42:18.076844075 +0000 UTC m=+0.137299657 container create bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:42:18.346 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 podman[62318]: 2024-04-22 06:42:18.348016152 +0000 UTC m=+0.408471734 container init bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T06:42:18.703 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 podman[62318]: 2024-04-22 06:42:18.350693377 +0000 UTC m=+0.411148956 container start bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:42:18.703 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mds[62337]: -- 172.21.15.190:0/3488879375 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca321bba40 con 0x55ca3218e800 2024-04-22T06:42:18.703 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: starting mds.b at 2024-04-22T06:42:18.703 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 bash[62318]: bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 2024-04-22T06:42:18.703 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:18 smithi190 systemd[1]: Started Ceph mds.b for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2748596727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:18.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:18 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2748596727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:18 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:19.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2748596727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:18 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:19 smithi193 ceph-mon[31815]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:19 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] up:boot 2024-04-22T06:42:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:19 smithi193 ceph-mon[31815]: fsmap 12 up:standby 2024-04-22T06:42:20.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:19 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:42:20.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:19 smithi190 ceph-mon[31421]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:20.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:19 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] up:boot 2024-04-22T06:42:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:19 smithi190 ceph-mon[31421]: fsmap 12 up:standby 2024-04-22T06:42:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:19 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:42:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:19 smithi079 ceph-mon[27946]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:19 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] up:boot 2024-04-22T06:42:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:19 smithi079 ceph-mon[27946]: fsmap 12 up:standby 2024-04-22T06:42:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:19 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:42:20.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:20.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 -- 172.21.15.79:0/2030934880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4103b40 msgr2=0x7fc2c410a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2030934880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4103b40 0x7fc2c410a3d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009ca0 tx=0x7fc2bc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 -- 172.21.15.79:0/2030934880 shutdown_connections 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2030934880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c4103180 0x7fc2c4103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2030934880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4103b40 0x7fc2c410a3d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2030934880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2c4101f90 0x7fc2c41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 -- 172.21.15.79:0/2030934880 >> 172.21.15.79:0/2030934880 conn(0x7fc2c40fd500 msgr2=0x7fc2c40ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 -- 172.21.15.79:0/2030934880 shutdown_connections 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 -- 172.21.15.79:0/2030934880 wait complete. 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.370+0000 7fc2c881e640 1 Processor -- start 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 -- start start 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 0x7fc2c4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c4103180 0x7fc2c4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2c4103b40 0x7fc2c4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c410db30 con 0x7fc2c4101f90 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c410d9b0 con 0x7fc2c4103180 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c410dcb0 con 0x7fc2c4103b40 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 0x7fc2c4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 0x7fc2c4075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48946/0 (socket says 172.21.15.79:48946) 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 -- 172.21.15.79:0/2298438635 learned_addr learned my addr 172.21.15.79:0/2298438635 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c2d9d640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2c4103b40 0x7fc2c4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c1d9b640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c4103180 0x7fc2c4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 -- 172.21.15.79:0/2298438635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2c4103b40 msgr2=0x7fc2c4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2c4103b40 0x7fc2c4074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 -- 172.21.15.79:0/2298438635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c4103180 msgr2=0x7fc2c4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c4103180 0x7fc2c4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 -- 172.21.15.79:0/2298438635 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c4075110 con 0x7fc2c4101f90 2024-04-22T06:42:20.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c259c640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 0x7fc2c4075d00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc2b400d950 tx=0x7fc2b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4017070 con 0x7fc2c4101f90 2024-04-22T06:42:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b4009c30 con 0x7fc2c4101f90 2024-04-22T06:42:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c41a22d0 con 0x7fc2c4101f90 2024-04-22T06:42:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.371+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b400f620 con 0x7fc2c4101f90 2024-04-22T06:42:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.372+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c41a27d0 con 0x7fc2c4101f90 2024-04-22T06:42:20.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.373+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2c4109330 con 0x7fc2c4101f90 2024-04-22T06:42:20.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.374+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc2b401a070 con 0x7fc2c4101f90 2024-04-22T06:42:20.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.374+0000 7fc2b37fe640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2ac0a3bb0 0x7fc2ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.375+0000 7fc2c1d9b640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2ac0a3bb0 0x7fc2ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.375+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc2b40cf9a0 con 0x7fc2c4101f90 2024-04-22T06:42:20.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.375+0000 7fc2c1d9b640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2ac0a3bb0 0x7fc2ac0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4075a70 tx=0x7fc2b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:20.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.379+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2b4009de0 con 0x7fc2c4101f90 2024-04-22T06:42:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.608+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2c402d5c0 con 0x7fc2c4101f90 2024-04-22T06:42:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.609+0000 7fc2b37fe640 1 -- 172.21.15.79:0/2298438635 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7fc2b409c820 con 0x7fc2c4101f90 2024-04-22T06:42:20.607 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:20.607 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24406,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-22T06:42:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 13 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2ac0a3bb0 msgr2=0x7fc2ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2ac0a3bb0 0x7fc2ac0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4075a70 tx=0x7fc2b8008040 comp rx=0 tx=0).stop 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 msgr2=0x7fc2c4075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 0x7fc2c4075d00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc2b400d950 tx=0x7fc2b400de10 comp rx=0 tx=0).stop 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 shutdown_connections 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2ac0a3bb0 0x7fc2ac0a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c4103180 0x7fc2c4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2c4103b40 0x7fc2c4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 --2- 172.21.15.79:0/2298438635 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2c4101f90 0x7fc2c4075d00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 >> 172.21.15.79:0/2298438635 conn(0x7fc2c40fd500 msgr2=0x7fc2c4106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:20.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 shutdown_connections 2024-04-22T06:42:20.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.611+0000 7fc2c881e640 1 -- 172.21.15.79:0/2298438635 wait complete. 2024-04-22T06:42:20.621 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:20.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 -- 172.21.15.79:0/344458258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0102980 msgr2=0x7f45c01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/344458258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0102980 0x7f45c01092a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009ca0 tx=0x7f45b802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 -- 172.21.15.79:0/344458258 shutdown_connections 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/344458258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45c0101fc0 0x7f45c0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/344458258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0102980 0x7f45c01092a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/344458258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45c0100e70 0x7f45c0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 -- 172.21.15.79:0/344458258 >> 172.21.15.79:0/344458258 conn(0x7f45c00fc3e0 msgr2=0x7f45c00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 -- 172.21.15.79:0/344458258 shutdown_connections 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.722+0000 7f45c6aeb640 1 -- 172.21.15.79:0/344458258 wait complete. 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 Processor -- start 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 -- start start 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45c0100e70 0x7f45c0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 0x7f45c0198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45c0102980 0x7f45c019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f45c010c9c0 con 0x7f45c0101fc0 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f45c010c840 con 0x7f45c0102980 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c6aeb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45c010cb40 con 0x7f45c0100e70 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 0x7f45c0198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 0x7f45c0198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48966/0 (socket says 172.21.15.79:48966) 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 -- 172.21.15.79:0/2664604739 learned_addr learned my addr 172.21.15.79:0/2664604739 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c4887640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45c0100e70 0x7f45c0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45c5088640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45c0102980 0x7f45c019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 -- 172.21.15.79:0/2664604739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45c0100e70 msgr2=0x7f45c0198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45c0100e70 0x7f45c0198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 -- 172.21.15.79:0/2664604739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45c0102980 msgr2=0x7f45c019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45c0102980 0x7f45c019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.723+0000 7f45bffff640 1 -- 172.21.15.79:0/2664604739 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c019fd40 con 0x7f45c0101fc0 2024-04-22T06:42:20.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.724+0000 7f45bffff640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 0x7f45c0198660 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00d770 tx=0x7f45ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:20.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.724+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac017070 con 0x7f45c0101fc0 2024-04-22T06:42:20.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.724+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ac009c30 con 0x7f45c0101fc0 2024-04-22T06:42:20.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.724+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac003cc0 con 0x7f45c0101fc0 2024-04-22T06:42:20.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.724+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c019ffd0 con 0x7f45c0101fc0 2024-04-22T06:42:20.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.724+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c01a04f0 con 0x7f45c0101fc0 2024-04-22T06:42:20.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.725+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c01a0720 con 0x7f45c0101fc0 2024-04-22T06:42:20.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.727+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f45ac003e60 con 0x7f45c0101fc0 2024-04-22T06:42:20.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.727+0000 7f45bdffb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45b00a3bb0 0x7f45b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:20.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.727+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f45ac0cf850 con 0x7f45c0101fc0 2024-04-22T06:42:20.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.727+0000 7f45c4887640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45b00a3bb0 0x7f45b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:20.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.728+0000 7f45c4887640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45b00a3bb0 0x7f45b00a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005fd0 tx=0x7f45b400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:20.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.732+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45ac009de0 con 0x7f45c0101fc0 2024-04-22T06:42:20.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.917+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45c01a0a10 con 0x7f45c0101fc0 2024-04-22T06:42:20.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.918+0000 7f45bdffb640 1 -- 172.21.15.79:0/2664604739 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18454 (secure 0 0 0) 0x7f45ac09c080 con 0x7f45c0101fc0 2024-04-22T06:42:20.916 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":81,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:41:32.712071+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:20.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45b00a3bb0 msgr2=0x7f45b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45b00a3bb0 0x7f45b00a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005fd0 tx=0x7f45b400b390 comp rx=0 tx=0).stop 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 msgr2=0x7f45c0198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 0x7f45c0198660 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00d770 tx=0x7f45ac00dc30 comp rx=0 tx=0).stop 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 shutdown_connections 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45b00a3bb0 0x7f45b00a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45c0102980 0x7f45c019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45c0101fc0 0x7f45c0198660 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 --2- 172.21.15.79:0/2664604739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45c0100e70 0x7f45c0198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 >> 172.21.15.79:0/2664604739 conn(0x7f45c00fc3e0 msgr2=0x7f45c00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 shutdown_connections 2024-04-22T06:42:20.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:20.920+0000 7f45c6aeb640 1 -- 172.21.15.79:0/2664604739 wait complete. 2024-04-22T06:42:20.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:42:20.930 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 -- 172.21.15.79:0/1382430063 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8101fa0 msgr2=0x7f6ef81023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 --2- 172.21.15.79:0/1382430063 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8101fa0 0x7f6ef81023c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009900 tx=0x7f6ee802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 -- 172.21.15.79:0/1382430063 shutdown_connections 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 --2- 172.21.15.79:0/1382430063 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef8103b50 0x7f6ef810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 --2- 172.21.15.79:0/1382430063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ef8103190 0x7f6ef8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 --2- 172.21.15.79:0/1382430063 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8101fa0 0x7f6ef81023c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 -- 172.21.15.79:0/1382430063 >> 172.21.15.79:0/1382430063 conn(0x7f6ef80fd530 msgr2=0x7f6ef80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 -- 172.21.15.79:0/1382430063 shutdown_connections 2024-04-22T06:42:21.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.043+0000 7f6efc92c640 1 -- 172.21.15.79:0/1382430063 wait complete. 2024-04-22T06:42:21.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 Processor -- start 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 -- start start 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef8101fa0 0x7f6ef81981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ef8103190 0x7f6ef8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 0x7f6ef819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef810dae0 con 0x7f6ef8103b50 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef810d960 con 0x7f6ef8101fa0 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.044+0000 7f6efc92c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef810dc60 con 0x7f6ef8103190 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 0x7f6ef819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 0x7f6ef819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48970/0 (socket says 172.21.15.79:48970) 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 -- 172.21.15.79:0/2930815213 learned_addr learned my addr 172.21.15.79:0/2930815213 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:21.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef659c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef8101fa0 0x7f6ef81981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6eeffff640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ef8103190 0x7f6ef8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 -- 172.21.15.79:0/2930815213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ef8103190 msgr2=0x7f6ef8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ef8103190 0x7f6ef8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 -- 172.21.15.79:0/2930815213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef8101fa0 msgr2=0x7f6ef81981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef8101fa0 0x7f6ef81981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 -- 172.21.15.79:0/2930815213 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef80742b0 con 0x7f6ef8103b50 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6ef6d9d640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 0x7f6ef819f790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000eaa0 tx=0x7f6ef000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000cdb0 con 0x7f6ef8103b50 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef80745a0 con 0x7f6ef8103b50 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ef0005020 con 0x7f6ef8103b50 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00116d0 con 0x7f6ef8103b50 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.045+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef8074af0 con 0x7f6ef8103b50 2024-04-22T06:42:21.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.046+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef8075f30 con 0x7f6ef8103b50 2024-04-22T06:42:21.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.048+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6ef00051d0 con 0x7f6ef8103b50 2024-04-22T06:42:21.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.048+0000 7f6eef7fe640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ee40a3bb0 0x7f6ee40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.048+0000 7f6ef659c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ee40a3bb0 0x7f6ee40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.049+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6ef00d0040 con 0x7f6ef8103b50 2024-04-22T06:42:21.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.049+0000 7f6ef659c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ee40a3bb0 0x7f6ee40a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8005ab0 tx=0x7f6ee8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.053+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ef0016030 con 0x7f6ef8103b50 2024-04-22T06:42:21.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.238+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ef81090d0 con 0x7f6ef8103b50 2024-04-22T06:42:21.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.239+0000 7f6eef7fe640 1 -- 172.21.15.79:0/2930815213 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f6ef009c870 con 0x7f6ef8103b50 2024-04-22T06:42:21.237 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:42:21.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ee40a3bb0 msgr2=0x7f6ee40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ee40a3bb0 0x7f6ee40a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8005ab0 tx=0x7f6ee8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 msgr2=0x7f6ef819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 0x7f6ef819f790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000eaa0 tx=0x7f6ef000ef60 comp rx=0 tx=0).stop 2024-04-22T06:42:21.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 shutdown_connections 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6ee40a3bb0 0x7f6ee40a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ef8103b50 0x7f6ef819f790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ef8103190 0x7f6ef8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 --2- 172.21.15.79:0/2930815213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ef8101fa0 0x7f6ef81981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 >> 172.21.15.79:0/2930815213 conn(0x7f6ef80fd530 msgr2=0x7f6ef81065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 shutdown_connections 2024-04-22T06:42:21.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.241+0000 7f6efc92c640 1 -- 172.21.15.79:0/2930815213 wait complete. 2024-04-22T06:42:21.251 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T06:42:21.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 -- 172.21.15.79:0/3177602693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c107c70 msgr2=0x7f3c6c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 --2- 172.21.15.79:0/3177602693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c107c70 0x7f3c6c10a0c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009ca0 tx=0x7f3c6402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 -- 172.21.15.79:0/3177602693 shutdown_connections 2024-04-22T06:42:21.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 --2- 172.21.15.79:0/3177602693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c105340 0x7f3c6c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 --2- 172.21.15.79:0/3177602693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c107c70 0x7f3c6c10a0c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 --2- 172.21.15.79:0/3177602693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102a10 0x7f3c6c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 -- 172.21.15.79:0/3177602693 >> 172.21.15.79:0/3177602693 conn(0x7f3c6c0fc400 msgr2=0x7f3c6c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 -- 172.21.15.79:0/3177602693 shutdown_connections 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 -- 172.21.15.79:0/3177602693 wait complete. 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 Processor -- start 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.354+0000 7f3c7288c640 1 -- start start 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102a10 0x7f3c6c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 0x7f3c6c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c107c70 0x7f3c6c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10db40 con 0x7f3c6c105340 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10d9c0 con 0x7f3c6c107c70 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10dcc0 con 0x7f3c6c102a10 2024-04-22T06:42:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 0x7f3c6c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 0x7f3c6c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48994/0 (socket says 172.21.15.79:48994) 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 -- 172.21.15.79:0/1304011511 learned_addr learned my addr 172.21.15.79:0/1304011511 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6bfff640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102a10 0x7f3c6c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c70e29640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c107c70 0x7f3c6c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 -- 172.21.15.79:0/1304011511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102a10 msgr2=0x7f3c6c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102a10 0x7f3c6c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 -- 172.21.15.79:0/1304011511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c107c70 msgr2=0x7f3c6c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c107c70 0x7f3c6c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 -- 172.21.15.79:0/1304011511 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c19fe30 con 0x7f3c6c105340 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c6b7fe640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 0x7f3c6c198750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000d770 tx=0x7f3c6000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60017070 con 0x7f3c6c105340 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c1a0120 con 0x7f3c6c105340 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c60010460 con 0x7f3c6c105340 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.355+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6000f620 con 0x7f3c6c105340 2024-04-22T06:42:21.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.356+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6c1a0670 con 0x7f3c6c105340 2024-04-22T06:42:21.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.357+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c6c0fdef0 con 0x7f3c6c105340 2024-04-22T06:42:21.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.358+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3c6001a070 con 0x7f3c6c105340 2024-04-22T06:42:21.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.358+0000 7f3c697fa640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c540a3bb0 0x7f3c540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.358+0000 7f3c6bfff640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c540a3bb0 0x7f3c540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.358+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3c600cf8c0 con 0x7f3c6c105340 2024-04-22T06:42:21.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.359+0000 7f3c6bfff640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c540a3bb0 0x7f3c540a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009990 tx=0x7f3c5c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.366+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c6009c740 con 0x7f3c6c105340 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2298438635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:21.544 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2664604739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.544 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:21 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2930815213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:42:21.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.553+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3c6c02d810 con 0x7f3c6c105340 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.553+0000 7f3c697fa640 1 -- 172.21.15.79:0/1304011511 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f3c6009c040 con 0x7f3c6c105340 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout:[ 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout: "poolnum": 1, 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout: "poolname": ".mgr" 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:42:21.552 INFO:teuthology.orchestra.run.smithi079.stdout:] 2024-04-22T06:42:21.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.555+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c540a3bb0 msgr2=0x7f3c540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c540a3bb0 0x7f3c540a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009990 tx=0x7f3c5c009110 comp rx=0 tx=0).stop 2024-04-22T06:42:21.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 msgr2=0x7f3c6c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 0x7f3c6c198750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000d770 tx=0x7f3c6000db70 comp rx=0 tx=0).stop 2024-04-22T06:42:21.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 shutdown_connections 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c540a3bb0 0x7f3c540a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c6c107c70 0x7f3c6c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c6c105340 0x7f3c6c198750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 --2- 172.21.15.79:0/1304011511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102a10 0x7f3c6c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 >> 172.21.15.79:0/1304011511 conn(0x7f3c6c0fc400 msgr2=0x7f3c6c103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 shutdown_connections 2024-04-22T06:42:21.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.556+0000 7f3c7288c640 1 -- 172.21.15.79:0/1304011511 wait complete. 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.564 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.565 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2298438635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.565 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:21.565 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2664604739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.565 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:21 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2930815213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:42:21.566 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T06:42:21.566 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T06:42:21.581 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.581 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2298438635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2664604739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:21.582 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:21 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2930815213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:42:21.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 -- 172.21.15.79:0/3837342386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab4107cf0 msgr2=0x7f6ab410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 --2- 172.21.15.79:0/3837342386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab4107cf0 0x7f6ab410a0e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009ca0 tx=0x7f6aac02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 -- 172.21.15.79:0/3837342386 shutdown_connections 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 --2- 172.21.15.79:0/3837342386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab4107cf0 0x7f6ab410a0e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 --2- 172.21.15.79:0/3837342386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ab41053c0 0x7f6ab41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 --2- 172.21.15.79:0/3837342386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4102a90 0x7f6ab4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 -- 172.21.15.79:0/3837342386 >> 172.21.15.79:0/3837342386 conn(0x7f6ab40fc400 msgr2=0x7f6ab40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 -- 172.21.15.79:0/3837342386 shutdown_connections 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.670+0000 7f6aba975640 1 -- 172.21.15.79:0/3837342386 wait complete. 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 Processor -- start 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 -- start start 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ab4102a90 0x7f6ab4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 0x7f6ab4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4107cf0 0x7f6ab419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab410dad0 con 0x7f6ab41053c0 2024-04-22T06:42:21.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab410d950 con 0x7f6ab4107cf0 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6aba975640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab410dc50 con 0x7f6ab4102a90 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6ab37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 0x7f6ab4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6ab37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 0x7f6ab4198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49020/0 (socket says 172.21.15.79:49020) 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6ab37fe640 1 -- 172.21.15.79:0/286479240 learned_addr learned my addr 172.21.15.79:0/286479240 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.671+0000 7f6ab8f12640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4107cf0 0x7f6ab419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab37fe640 1 -- 172.21.15.79:0/286479240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ab4102a90 msgr2=0x7f6ab4198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab37fe640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ab4102a90 0x7f6ab4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab37fe640 1 -- 172.21.15.79:0/286479240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4107cf0 msgr2=0x7f6ab419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab37fe640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4107cf0 0x7f6ab419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab37fe640 1 -- 172.21.15.79:0/286479240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab419fe20 con 0x7f6ab41053c0 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab8f12640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4107cf0 0x7f6ab419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab37fe640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 0x7f6ab4198740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800d740 tx=0x7f6aa800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8017070 con 0x7f6ab41053c0 2024-04-22T06:42:21.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab41a0110 con 0x7f6ab41053c0 2024-04-22T06:42:21.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aa800de50 con 0x7f6ab41053c0 2024-04-22T06:42:21.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8004ce0 con 0x7f6ab41053c0 2024-04-22T06:42:21.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.672+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab41a0610 con 0x7f6ab41053c0 2024-04-22T06:42:21.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.673+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab40fdef0 con 0x7f6ab41053c0 2024-04-22T06:42:21.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.674+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6aa8004e80 con 0x7f6ab41053c0 2024-04-22T06:42:21.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.674+0000 7f6ab17fa640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6a9c0a3bb0 0x7f6a9c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.675+0000 7f6ab3fff640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6a9c0a3bb0 0x7f6a9c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.675+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6aa80cf9b0 con 0x7f6ab41053c0 2024-04-22T06:42:21.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.675+0000 7f6ab3fff640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6a9c0a3bb0 0x7f6a9c0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4005fd0 tx=0x7f6aa4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.679+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6aa809c830 con 0x7f6ab41053c0 2024-04-22T06:42:21.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:21.889+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f6ab402d5c0 con 0x7f6ab41053c0 2024-04-22T06:42:22.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.644+0000 7f6ab17fa640 1 -- 172.21.15.79:0/286479240 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f6aa809c130 con 0x7f6ab41053c0 2024-04-22T06:42:22.643 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'cephfs_metadata' created 2024-04-22T06:42:22.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6a9c0a3bb0 msgr2=0x7f6a9c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:22.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6a9c0a3bb0 0x7f6a9c0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4005fd0 tx=0x7f6aa4005850 comp rx=0 tx=0).stop 2024-04-22T06:42:22.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 msgr2=0x7f6ab4198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:22.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 0x7f6ab4198740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800d740 tx=0x7f6aa800dc00 comp rx=0 tx=0).stop 2024-04-22T06:42:22.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 shutdown_connections 2024-04-22T06:42:22.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f6a9c0a3bb0 0x7f6a9c0a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ab4107cf0 0x7f6ab419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6ab41053c0 0x7f6ab4198740 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 --2- 172.21.15.79:0/286479240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ab4102a90 0x7f6ab4198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 >> 172.21.15.79:0/286479240 conn(0x7f6ab40fc400 msgr2=0x7f6ab4103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:22.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 shutdown_connections 2024-04-22T06:42:22.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.646+0000 7f6aba975640 1 -- 172.21.15.79:0/286479240 wait complete. 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1304011511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/286479240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.658 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:22.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:22.659 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.660 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T06:42:22.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1304011511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:42:22.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/286479240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:22.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1304011511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:42:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/286479240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:42:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:22.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:22.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1115384822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 msgr2=0x7fc5d810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1115384822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d810a0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009ca0 tx=0x7fc5d002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1115384822 shutdown_connections 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1115384822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d810a0c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1115384822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8105340 0x7fc5d8107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1115384822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d8102a10 0x7fc5d8104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1115384822 >> 172.21.15.79:0/1115384822 conn(0x7fc5d80fc400 msgr2=0x7fc5d80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1115384822 shutdown_connections 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.767+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1115384822 wait complete. 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 Processor -- start 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 -- start start 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d8102a10 0x7fc5d8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8105340 0x7fc5d81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d810dad0 con 0x7fc5d8107c70 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d810d950 con 0x7fc5d8102a10 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5dcf2b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d810dc50 con 0x7fc5d8105340 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5d6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.768+0000 7fc5d6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51514/0 (socket says 172.21.15.79:51514) 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d5d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8105340 0x7fc5d81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:22.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 -- 172.21.15.79:0/1046041963 learned_addr learned my addr 172.21.15.79:0/1046041963 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d659c640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d8102a10 0x7fc5d8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 -- 172.21.15.79:0/1046041963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8105340 msgr2=0x7fc5d81986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8105340 0x7fc5d81986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 -- 172.21.15.79:0/1046041963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d8102a10 msgr2=0x7fc5d8198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d8102a10 0x7fc5d8198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 -- 172.21.15.79:0/1046041963 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d819fd90 con 0x7fc5d8107c70 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5d6d9d640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d819f720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc5d002c700 tx=0x7fc5d00093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0041020 con 0x7fc5d8107c70 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5d0034d60 con 0x7fc5d8107c70 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d81a0020 con 0x7fc5d8107c70 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0039040 con 0x7fc5d8107c70 2024-04-22T06:42:22.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.769+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d81a0510 con 0x7fc5d8107c70 2024-04-22T06:42:22.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.770+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d80fdef0 con 0x7fc5d8107c70 2024-04-22T06:42:22.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.771+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc5d0044020 con 0x7fc5d8107c70 2024-04-22T06:42:22.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.772+0000 7fc5c77fe640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5c00a3bb0 0x7fc5c00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:22.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.772+0000 7fc5d659c640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5c00a3bb0 0x7fc5c00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:22.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.772+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fc5d00f2450 con 0x7fc5d8107c70 2024-04-22T06:42:22.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.772+0000 7fc5d659c640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5c00a3bb0 0x7fc5c00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009b60 tx=0x7fc5c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:22.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.776+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5d004b020 con 0x7fc5d8107c70 2024-04-22T06:42:22.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:22.955+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc5d802d5c0 con 0x7fc5d8107c70 2024-04-22T06:42:23.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.653+0000 7fc5c77fe640 1 -- 172.21.15.79:0/1046041963 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7fc5d00bf160 con 0x7fc5d8107c70 2024-04-22T06:42:23.652 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'cephfs_data' created 2024-04-22T06:42:23.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5c00a3bb0 msgr2=0x7fc5c00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5c00a3bb0 0x7fc5c00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009b60 tx=0x7fc5c8005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:23.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 msgr2=0x7fc5d819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d819f720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc5d002c700 tx=0x7fc5d00093f0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 shutdown_connections 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5c00a3bb0 0x7fc5c00a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5d8107c70 0x7fc5d819f720 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d8105340 0x7fc5d81986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 --2- 172.21.15.79:0/1046041963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5d8102a10 0x7fc5d8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.656+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 >> 172.21.15.79:0/1046041963 conn(0x7fc5d80fc400 msgr2=0x7fc5d81033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:23.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.657+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 shutdown_connections 2024-04-22T06:42:23.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.657+0000 7fc5dcf2b640 1 -- 172.21.15.79:0/1046041963 wait complete. 2024-04-22T06:42:23.679 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T06:42:23.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 -- 172.21.15.79:0/464724988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 msgr2=0x7fec00102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 --2- 172.21.15.79:0/464724988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec00102460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7febf4009900 tx=0x7febf402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 -- 172.21.15.79:0/464724988 shutdown_connections 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 --2- 172.21.15.79:0/464724988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec001029a0 0x7fec001092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 --2- 172.21.15.79:0/464724988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec00102460 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 --2- 172.21.15.79:0/464724988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00100e90 0x7fec001012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.786+0000 7fec0646d640 1 -- 172.21.15.79:0/464724988 >> 172.21.15.79:0/464724988 conn(0x7fec000fc400 msgr2=0x7fec000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 -- 172.21.15.79:0/464724988 shutdown_connections 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 -- 172.21.15.79:0/464724988 wait complete. 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 Processor -- start 2024-04-22T06:42:23.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 -- start start 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00100e90 0x7fec00069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec0019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec001029a0 0x7fec0019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fec0010c9f0 con 0x7fec00101fe0 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec0010c870 con 0x7fec001029a0 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7fec0646d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec0010cb70 con 0x7fec00100e90 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7febff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec0019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7febff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec0019cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51530/0 (socket says 172.21.15.79:51530) 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.787+0000 7febff7fe640 1 -- 172.21.15.79:0/3294707282 learned_addr learned my addr 172.21.15.79:0/3294707282 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febfffff640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00100e90 0x7fec00069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febff7fe640 1 -- 172.21.15.79:0/3294707282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00100e90 msgr2=0x7fec00069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febff7fe640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00100e90 0x7fec00069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febff7fe640 1 -- 172.21.15.79:0/3294707282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec001029a0 msgr2=0x7fec0019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7fec04a0a640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec001029a0 0x7fec0019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febff7fe640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec001029a0 0x7fec0019f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febff7fe640 1 -- 172.21.15.79:0/3294707282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0019fed0 con 0x7fec00101fe0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febff7fe640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec0019cfa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7febf4000c00 tx=0x7febf4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4043020 con 0x7fec00101fe0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec001a0100 con 0x7fec00101fe0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febf4004440 con 0x7fec00101fe0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4003c90 con 0x7fec00101fe0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.788+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec001a0650 con 0x7fec00101fe0 2024-04-22T06:42:23.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.789+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec00108310 con 0x7fec00101fe0 2024-04-22T06:42:23.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.790+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7febf4046020 con 0x7fec00101fe0 2024-04-22T06:42:23.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.791+0000 7febfd7fa640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7febe80a3bb0 0x7febe80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.791+0000 7febfffff640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7febe80a3bb0 0x7febe80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.791+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7febf40f29e0 con 0x7fec00101fe0 2024-04-22T06:42:23.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.792+0000 7febfffff640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7febe80a3bb0 0x7febe80a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7febf0009b60 tx=0x7febf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:23.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:23.795+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7febf404d020 con 0x7fec00101fe0 2024-04-22T06:42:23.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:23 smithi193 ceph-mon[31815]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:23.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:23 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/286479240' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:42:23.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:23 smithi193 ceph-mon[31815]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T06:42:23.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:23 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1046041963' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:42:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:23 smithi190 ceph-mon[31421]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:23 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/286479240' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:42:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:23 smithi190 ceph-mon[31421]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T06:42:23.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:23 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1046041963' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:42:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:23 smithi079 ceph-mon[27946]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:23 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/286479240' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:42:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:23 smithi079 ceph-mon[27946]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T06:42:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:23 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1046041963' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:42:24.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.010+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fec00062b50 con 0x7fec00101fe0 2024-04-22T06:42:24.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.669+0000 7febfd7fa640 1 -- 172.21.15.79:0/3294707282 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7febf40bf550 con 0x7fec00101fe0 2024-04-22T06:42:24.668 INFO:teuthology.orchestra.run.smithi079.stderr:new fs with metadata pool 2 and data pool 3 2024-04-22T06:42:24.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7febe80a3bb0 msgr2=0x7febe80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7febe80a3bb0 0x7febe80a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7febf0009b60 tx=0x7febf0005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 msgr2=0x7fec0019cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec0019cfa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7febf4000c00 tx=0x7febf4004040 comp rx=0 tx=0).stop 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 shutdown_connections 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7febe80a3bb0 0x7febe80a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec001029a0 0x7fec0019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fec00101fe0 0x7fec0019cfa0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 --2- 172.21.15.79:0/3294707282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00100e90 0x7fec00069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 >> 172.21.15.79:0/3294707282 conn(0x7fec000fc400 msgr2=0x7fec001054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 shutdown_connections 2024-04-22T06:42:24.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.672+0000 7fec0646d640 1 -- 172.21.15.79:0/3294707282 wait complete. 2024-04-22T06:42:24.686 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:24.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 -- 172.21.15.79:0/1857526455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988103bf0 msgr2=0x7f698810a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 --2- 172.21.15.79:0/1857526455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988103bf0 0x7f698810a480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6980009cd0 tx=0x7f698002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 -- 172.21.15.79:0/1857526455 shutdown_connections 2024-04-22T06:42:24.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 --2- 172.21.15.79:0/1857526455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 0x7f69881036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 --2- 172.21.15.79:0/1857526455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988103bf0 0x7f698810a480 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 --2- 172.21.15.79:0/1857526455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988100260 0x7f6988100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 -- 172.21.15.79:0/1857526455 >> 172.21.15.79:0/1857526455 conn(0x7f69880fbb90 msgr2=0x7f69880fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 -- 172.21.15.79:0/1857526455 shutdown_connections 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.785+0000 7f698f21e640 1 -- 172.21.15.79:0/1857526455 wait complete. 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 Processor -- start 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 -- start start 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 0x7f6988198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 0x7f69881987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:24.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988103bf0 0x7f698819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f698810dbe0 con 0x7f6988100260 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f698810da60 con 0x7f6988103230 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698f21e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f698810dd60 con 0x7f6988103bf0 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698cfba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 0x7f6988198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698cfba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 0x7f6988198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51538/0 (socket says 172.21.15.79:51538) 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698cfba640 1 -- 172.21.15.79:0/2538310762 learned_addr learned my addr 172.21.15.79:0/2538310762 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.786+0000 7f698d7bb640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988103bf0 0x7f698819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698cfba640 1 -- 172.21.15.79:0/2538310762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988103bf0 msgr2=0x7f698819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698cfba640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988103bf0 0x7f698819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f6987fff640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 0x7f69881987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698cfba640 1 -- 172.21.15.79:0/2538310762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 msgr2=0x7f69881987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698cfba640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 0x7f69881987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698cfba640 1 -- 172.21.15.79:0/2538310762 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698819fe80 con 0x7f6988100260 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698d7bb640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988103bf0 0x7f698819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f6987fff640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 0x7f69881987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698cfba640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 0x7f6988198260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f697c00b450 tx=0x7f697c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c015020 con 0x7f6988100260 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697c011d40 con 0x7f6988100260 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69881a0170 con 0x7f6988100260 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c00f340 con 0x7f6988100260 2024-04-22T06:42:24.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.787+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69881a06f0 con 0x7f6988100260 2024-04-22T06:42:24.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.788+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69881093e0 con 0x7f6988100260 2024-04-22T06:42:24.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.790+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f697c018020 con 0x7f6988100260 2024-04-22T06:42:24.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.790+0000 7f6985ffb640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69780a3bb0 0x7f69780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:24.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.791+0000 7f6987fff640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69780a3bb0 0x7f69780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:24.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.791+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f697c0d0410 con 0x7f6988100260 2024-04-22T06:42:24.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.791+0000 7f6987fff640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69780a3bb0 0x7f69780a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6974006fd0 tx=0x7f6974008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:24.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.794+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f697c01f020 con 0x7f6988100260 2024-04-22T06:42:24.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:24 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1046041963' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:42:24.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:24 smithi193 ceph-mon[31815]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T06:42:24.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:24 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3294707282' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:42:24.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:24 smithi193 ceph-mon[31815]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T06:42:24.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:24 smithi193 ceph-mon[31815]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:42:24.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:24 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1046041963' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:42:24.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:24 smithi190 ceph-mon[31421]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T06:42:24.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:24 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3294707282' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:42:24.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:24 smithi190 ceph-mon[31421]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T06:42:24.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:24 smithi190 ceph-mon[31421]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:42:24.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.974+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6988062b50 con 0x7f6988100260 2024-04-22T06:42:24.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.976+0000 7f6985ffb640 1 -- 172.21.15.79:0/2538310762 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22230 (secure 0 0 0) 0x7f697c00d040 con 0x7f6988100260 2024-04-22T06:42:24.974 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:24.975 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":84,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:42:24.011988+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:24.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69780a3bb0 msgr2=0x7f69780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69780a3bb0 0x7f69780a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6974006fd0 tx=0x7f6974008040 comp rx=0 tx=0).stop 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 msgr2=0x7f6988198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 0x7f6988198260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f697c00b450 tx=0x7f697c00b910 comp rx=0 tx=0).stop 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 shutdown_connections 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69780a3bb0 0x7f69780a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6988103230 0x7f69881987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6988103bf0 0x7f698819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 --2- 172.21.15.79:0/2538310762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6988100260 0x7f6988198260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 >> 172.21.15.79:0/2538310762 conn(0x7f69880fbb90 msgr2=0x7f69881065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 shutdown_connections 2024-04-22T06:42:24.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:24.978+0000 7f698f21e640 1 -- 172.21.15.79:0/2538310762 wait complete. 2024-04-22T06:42:24.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:24 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1046041963' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:42:24.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:24 smithi079 ceph-mon[27946]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T06:42:24.986 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:24 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3294707282' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:42:24.987 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:24 smithi079 ceph-mon[27946]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T06:42:24.987 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:24 smithi079 ceph-mon[27946]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:42:24.988 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 -- 172.21.15.79:0/532933113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500102a20 msgr2=0x7f4500104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 --2- 172.21.15.79:0/532933113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500102a20 0x7f4500104e10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009900 tx=0x7f44f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 -- 172.21.15.79:0/532933113 shutdown_connections 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 --2- 172.21.15.79:0/532933113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500107c80 0x7f450010a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 --2- 172.21.15.79:0/532933113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4500105350 0x7f4500107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 --2- 172.21.15.79:0/532933113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500102a20 0x7f4500104e10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 -- 172.21.15.79:0/532933113 >> 172.21.15.79:0/532933113 conn(0x7f45000fc410 msgr2=0x7f45000fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 -- 172.21.15.79:0/532933113 shutdown_connections 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.087+0000 7f4504ab2640 1 -- 172.21.15.79:0/532933113 wait complete. 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 Processor -- start 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 -- start start 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4500102a20 0x7f4500198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500105350 0x7f4500198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 0x7f450019f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f450010dbc0 con 0x7f4500107c80 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f450010da40 con 0x7f4500105350 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f4504ab2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f450010dd40 con 0x7f4500102a20 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 0x7f450019f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 0x7f450019f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51554/0 (socket says 172.21.15.79:51554) 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 -- 172.21.15.79:0/2149016598 learned_addr learned my addr 172.21.15.79:0/2149016598 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:25.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fdd9b640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500105350 0x7f4500198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fe59c640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4500102a20 0x7f4500198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 -- 172.21.15.79:0/2149016598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4500102a20 msgr2=0x7f4500198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4500102a20 0x7f4500198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 -- 172.21.15.79:0/2149016598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500105350 msgr2=0x7f4500198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500105350 0x7f4500198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fed9d640 1 -- 172.21.15.79:0/2149016598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450019ff40 con 0x7f4500107c80 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.088+0000 7f44fdd9b640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500105350 0x7f4500198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.089+0000 7f44fed9d640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 0x7f450019f8d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f44f800b970 tx=0x7f44f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.089+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8019070 con 0x7f4500107c80 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.089+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44f8014440 con 0x7f4500107c80 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.089+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45001a0230 con 0x7f4500107c80 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.089+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80075d0 con 0x7f4500107c80 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.089+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45001a07b0 con 0x7f4500107c80 2024-04-22T06:42:25.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.090+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45000fdf00 con 0x7f4500107c80 2024-04-22T06:42:25.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.091+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f44f801c070 con 0x7f4500107c80 2024-04-22T06:42:25.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.092+0000 7f44ef7fe640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44e80a3bb0 0x7f44e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.092+0000 7f44fe59c640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44e80a3bb0 0x7f44e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.092+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f44f80cfe20 con 0x7f4500107c80 2024-04-22T06:42:25.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.092+0000 7f44fe59c640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44e80a3bb0 0x7f44e80a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f44f00096c0 tx=0x7f44f0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.096+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44f809c950 con 0x7f4500107c80 2024-04-22T06:42:25.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.278+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4500062b50 con 0x7f4500107c80 2024-04-22T06:42:25.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.279+0000 7f44ef7fe640 1 -- 172.21.15.79:0/2149016598 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22217 (secure 0 0 0) 0x7f44f809c250 con 0x7f4500107c80 2024-04-22T06:42:25.278 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:25.278 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":85,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:42:25.177009+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:25.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44e80a3bb0 msgr2=0x7f44e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44e80a3bb0 0x7f44e80a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f44f00096c0 tx=0x7f44f0000bc0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 msgr2=0x7f450019f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 0x7f450019f8d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f44f800b970 tx=0x7f44f800be30 comp rx=0 tx=0).stop 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 shutdown_connections 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44e80a3bb0 0x7f44e80a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4500105350 0x7f4500198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4500107c80 0x7f450019f8d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 --2- 172.21.15.79:0/2149016598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4500102a20 0x7f4500198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 >> 172.21.15.79:0/2149016598 conn(0x7f45000fc410 msgr2=0x7f4500103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 shutdown_connections 2024-04-22T06:42:25.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.281+0000 7f4504ab2640 1 -- 172.21.15.79:0/2149016598 wait complete. 2024-04-22T06:42:25.291 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 -- 172.21.15.79:0/3995192155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d41053a0 msgr2=0x7f63d4107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 --2- 172.21.15.79:0/3995192155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d41053a0 0x7f63d4107790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009900 tx=0x7f63c002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 -- 172.21.15.79:0/3995192155 shutdown_connections 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 --2- 172.21.15.79:0/3995192155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d4107cd0 0x7f63d410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 --2- 172.21.15.79:0/3995192155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d41053a0 0x7f63d4107790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 --2- 172.21.15.79:0/3995192155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d4102a70 0x7f63d4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.393+0000 7f63db081640 1 -- 172.21.15.79:0/3995192155 >> 172.21.15.79:0/3995192155 conn(0x7f63d40fc400 msgr2=0x7f63d40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 -- 172.21.15.79:0/3995192155 shutdown_connections 2024-04-22T06:42:25.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 -- 172.21.15.79:0/3995192155 wait complete. 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 Processor -- start 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 -- start start 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 0x7f63d419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d41053a0 0x7f63d419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d4107cd0 0x7f63d41a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f63d410db10 con 0x7f63d4102a70 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63d410d990 con 0x7f63d41053a0 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63db081640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63d410dc90 con 0x7f63d4107cd0 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63d8e1d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 0x7f63d419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63d8e1d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 0x7f63d419a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51574/0 (socket says 172.21.15.79:51574) 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63d8e1d640 1 -- 172.21.15.79:0/998583242 learned_addr learned my addr 172.21.15.79:0/998583242 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63d3fff640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d41053a0 0x7f63d419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63d8e1d640 1 -- 172.21.15.79:0/998583242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d4107cd0 msgr2=0x7f63d41a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.394+0000 7f63d8e1d640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d4107cd0 0x7f63d41a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d8e1d640 1 -- 172.21.15.79:0/998583242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d41053a0 msgr2=0x7f63d419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d8e1d640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d41053a0 0x7f63d419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d8e1d640 1 -- 172.21.15.79:0/998583242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d41a20b0 con 0x7f63d4102a70 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d3fff640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d41053a0 0x7f63d419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d8e1d640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 0x7f63d419a490 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f63c800d950 tx=0x7f63c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8017030 con 0x7f63d4102a70 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63c8004020 con 0x7f63d4102a70 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d41a23a0 con 0x7f63d4102a70 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c80047a0 con 0x7f63d4102a70 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.395+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d41a28c0 con 0x7f63d4102a70 2024-04-22T06:42:25.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.396+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d41097c0 con 0x7f63d4102a70 2024-04-22T06:42:25.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.397+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f63c8009a30 con 0x7f63d4102a70 2024-04-22T06:42:25.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.397+0000 7f63d1ffb640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63c40a3bb0 0x7f63c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.398+0000 7f63d3fff640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63c40a3bb0 0x7f63c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.398+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f63c80cfe50 con 0x7f63d4102a70 2024-04-22T06:42:25.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.398+0000 7f63d3fff640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63c40a3bb0 0x7f63c40a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f63c0006010 tx=0x7f63c002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.402+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63c809c980 con 0x7f63d4102a70 2024-04-22T06:42:25.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.619+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f63d402d5c0 con 0x7f63d4102a70 2024-04-22T06:42:25.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.679+0000 7f63d1ffb640 1 -- 172.21.15.79:0/998583242 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f63c809c280 con 0x7f63d4102a70 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63c40a3bb0 msgr2=0x7f63c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63c40a3bb0 0x7f63c40a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f63c0006010 tx=0x7f63c002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 msgr2=0x7f63d419a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 0x7f63d419a490 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f63c800d950 tx=0x7f63c800de10 comp rx=0 tx=0).stop 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 shutdown_connections 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63c40a3bb0 0x7f63c40a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d41053a0 0x7f63d419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63d4107cd0 0x7f63d41a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 --2- 172.21.15.79:0/998583242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63d4102a70 0x7f63d419a490 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 >> 172.21.15.79:0/998583242 conn(0x7f63d40fc400 msgr2=0x7f63d4102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:25.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 shutdown_connections 2024-04-22T06:42:25.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.681+0000 7f63db081640 1 -- 172.21.15.79:0/998583242 wait complete. 2024-04-22T06:42:25.692 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T06:42:25.692 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-22T06:42:25.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 -- 172.21.15.79:0/256242766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac101fa0 msgr2=0x7f80ac1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 --2- 172.21.15.79:0/256242766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac101fa0 0x7f80ac1023c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f809c009900 tx=0x7f809c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:25.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 -- 172.21.15.79:0/256242766 shutdown_connections 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 --2- 172.21.15.79:0/256242766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80ac103b50 0x7f80ac10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 --2- 172.21.15.79:0/256242766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80ac103190 0x7f80ac103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 --2- 172.21.15.79:0/256242766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac101fa0 0x7f80ac1023c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 -- 172.21.15.79:0/256242766 >> 172.21.15.79:0/256242766 conn(0x7f80ac0fd4f0 msgr2=0x7f80ac0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 -- 172.21.15.79:0/256242766 shutdown_connections 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.795+0000 7f80b1a32640 1 -- 172.21.15.79:0/256242766 wait complete. 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 Processor -- start 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 -- start start 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80ac101fa0 0x7f80ac1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 0x7f80ac1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80ac103b50 0x7f80ac19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac10db20 con 0x7f80ac103190 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac10d9a0 con 0x7f80ac101fa0 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b1a32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac10dca0 con 0x7f80ac103b50 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 0x7f80ac1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 0x7f80ac1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51590/0 (socket says 172.21.15.79:51590) 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 -- 172.21.15.79:0/2020153586 learned_addr learned my addr 172.21.15.79:0/2020153586 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80aaffd640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80ac101fa0 0x7f80ac1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80ab7fe640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80ac103b50 0x7f80ac19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 -- 172.21.15.79:0/2020153586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80ac103b50 msgr2=0x7f80ac19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80ac103b50 0x7f80ac19f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 -- 172.21.15.79:0/2020153586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80ac101fa0 msgr2=0x7f80ac1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80ac101fa0 0x7f80ac1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 -- 172.21.15.79:0/2020153586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ac19fed0 con 0x7f80ac103190 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80a3fff640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 0x7f80ac1987f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f809400b4a0 tx=0x7f809400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.796+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094014070 con 0x7f80ac103190 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.797+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8094010ac0 con 0x7f80ac103190 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.797+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ac1a01c0 con 0x7f80ac103190 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.797+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094017c30 con 0x7f80ac103190 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.797+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ac1a0740 con 0x7f80ac103190 2024-04-22T06:42:25.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.798+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80ac109380 con 0x7f80ac103190 2024-04-22T06:42:25.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.799+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8094017070 con 0x7f80ac103190 2024-04-22T06:42:25.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.799+0000 7f80b0a30640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f80980a3bb0 0x7f80980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.799+0000 7f80aaffd640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f80980a3bb0 0x7f80980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.799+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f80940cf700 con 0x7f80ac103190 2024-04-22T06:42:25.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.800+0000 7f80aaffd640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f80980a3bb0 0x7f80980a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f809c006010 tx=0x7f809c02c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:25.804+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f809409c230 con 0x7f80ac103190 2024-04-22T06:42:25.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: pgmap v288: 97 pgs: 15 creating+peering, 6 active+clean, 76 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:25.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3294707282' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:42:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T06:42:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T06:42:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2538310762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:42:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2149016598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:25 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/998583242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: pgmap v288: 97 pgs: 15 creating+peering, 6 active+clean, 76 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3294707282' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2538310762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2149016598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:25 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/998583242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: pgmap v288: 97 pgs: 15 creating+peering, 6 active+clean, 76 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3294707282' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2538310762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2149016598' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:25 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/998583242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:42:26.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.024+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f80ac02d810 con 0x7f80ac103190 2024-04-22T06:42:26.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.689+0000 7f80b0a30640 1 -- 172.21.15.79:0/2020153586 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f809409bb30 con 0x7f80ac103190 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f80980a3bb0 msgr2=0x7f80980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f80980a3bb0 0x7f80980a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f809c006010 tx=0x7f809c02c800 comp rx=0 tx=0).stop 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 msgr2=0x7f80ac1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 0x7f80ac1987f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f809400b4a0 tx=0x7f809400b960 comp rx=0 tx=0).stop 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 shutdown_connections 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f80980a3bb0 0x7f80980a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80ac103b50 0x7f80ac19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80ac103190 0x7f80ac1987f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 --2- 172.21.15.79:0/2020153586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80ac101fa0 0x7f80ac1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 >> 172.21.15.79:0/2020153586 conn(0x7f80ac0fd4f0 msgr2=0x7f80ac106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 shutdown_connections 2024-04-22T06:42:26.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.691+0000 7f80b1a32640 1 -- 172.21.15.79:0/2020153586 wait complete. 2024-04-22T06:42:26.705 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-22T06:42:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 -- 172.21.15.79:0/694627653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660100e90 msgr2=0x7fa6601012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 --2- 172.21.15.79:0/694627653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660100e90 0x7fa6601012b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa654009900 tx=0x7fa65402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 -- 172.21.15.79:0/694627653 shutdown_connections 2024-04-22T06:42:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 --2- 172.21.15.79:0/694627653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6601029a0 0x7fa6601092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 --2- 172.21.15.79:0/694627653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa660101fe0 0x7fa660102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 --2- 172.21.15.79:0/694627653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660100e90 0x7fa6601012b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 -- 172.21.15.79:0/694627653 >> 172.21.15.79:0/694627653 conn(0x7fa6600fc400 msgr2=0x7fa6600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 -- 172.21.15.79:0/694627653 shutdown_connections 2024-04-22T06:42:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 -- 172.21.15.79:0/694627653 wait complete. 2024-04-22T06:42:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.809+0000 7fa6650c1640 1 Processor -- start 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 -- start start 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa660100e90 0x7fa660198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 0x7fa660198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6601029a0 0x7fa66019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa66010cab0 con 0x7fa660101fe0 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa66010c930 con 0x7fa660100e90 2024-04-22T06:42:26.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa66010cc30 con 0x7fa6601029a0 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 0x7fa660198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 0x7fa660198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51614/0 (socket says 172.21.15.79:51614) 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 -- 172.21.15.79:0/2184018376 learned_addr learned my addr 172.21.15.79:0/2184018376 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65f59e640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6601029a0 0x7fa66019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65ed9d640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa660100e90 0x7fa660198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 -- 172.21.15.79:0/2184018376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6601029a0 msgr2=0x7fa66019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6601029a0 0x7fa66019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 -- 172.21.15.79:0/2184018376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa660100e90 msgr2=0x7fa660198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa660100e90 0x7fa660198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 -- 172.21.15.79:0/2184018376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66019fe50 con 0x7fa660101fe0 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65f59e640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6601029a0 0x7fa66019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa65e59c640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 0x7fa660198770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b450 tx=0x7fa65000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650015020 con 0x7fa660101fe0 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6601a0140 con 0x7fa660101fe0 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa650011d40 con 0x7fa660101fe0 2024-04-22T06:42:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65000f340 con 0x7fa660101fe0 2024-04-22T06:42:26.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.810+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6601a0710 con 0x7fa660101fe0 2024-04-22T06:42:26.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.811+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6601a09d0 con 0x7fa660101fe0 2024-04-22T06:42:26.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.813+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa650018020 con 0x7fa660101fe0 2024-04-22T06:42:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.813+0000 7fa647fff640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa6480a3bb0 0x7fa6480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.813+0000 7fa65ed9d640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa6480a3bb0 0x7fa6480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.813+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa6500d0410 con 0x7fa660101fe0 2024-04-22T06:42:26.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.814+0000 7fa65ed9d640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa6480a3bb0 0x7fa6480a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa65402c840 tx=0x7fa654000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:26.817+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa65001f020 con 0x7fa660101fe0 2024-04-22T06:42:26.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:26 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/998583242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:42:26.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:26 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] up:active 2024-04-22T06:42:26.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:26 smithi193 ceph-mon[31815]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:42:26.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:26 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2020153586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T06:42:26.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:26 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/998583242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:42:26.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:26 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] up:active 2024-04-22T06:42:26.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:26 smithi190 ceph-mon[31421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:42:26.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:26 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2020153586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T06:42:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:26 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/998583242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:42:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:26 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] up:active 2024-04-22T06:42:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:26 smithi079 ceph-mon[27946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:42:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:26 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2020153586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T06:42:27.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.035+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fa6601a0ba0 con 0x7fa660101fe0 2024-04-22T06:42:27.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.697+0000 7fa647fff640 1 -- 172.21.15.79:0/2184018376 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7fa65000d040 con 0x7fa660101fe0 2024-04-22T06:42:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.699+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa6480a3bb0 msgr2=0x7fa6480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.699+0000 7fa6650c1640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa6480a3bb0 0x7fa6480a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa65402c840 tx=0x7fa654000bc0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 msgr2=0x7fa660198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 0x7fa660198770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b450 tx=0x7fa65000b910 comp rx=0 tx=0).stop 2024-04-22T06:42:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 shutdown_connections 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa6480a3bb0 0x7fa6480a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6601029a0 0x7fa66019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa660101fe0 0x7fa660198770 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 --2- 172.21.15.79:0/2184018376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa660100e90 0x7fa660198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 >> 172.21.15.79:0/2184018376 conn(0x7fa6600fc400 msgr2=0x7fa66006a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 shutdown_connections 2024-04-22T06:42:27.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.700+0000 7fa6650c1640 1 -- 172.21.15.79:0/2184018376 wait complete. 2024-04-22T06:42:27.714 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 -- 172.21.15.79:0/1241349911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102a00 msgr2=0x7fe558109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/1241349911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102a00 0x7fe558109320 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe554009ca0 tx=0x7fe55402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 -- 172.21.15.79:0/1241349911 shutdown_connections 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/1241349911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102a00 0x7fe558109320 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/1241349911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe558102040 0x7fe5581024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/1241349911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe558100e60 0x7fe558101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 -- 172.21.15.79:0/1241349911 >> 172.21.15.79:0/1241349911 conn(0x7fe5580fc440 msgr2=0x7fe5580fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 -- 172.21.15.79:0/1241349911 shutdown_connections 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 -- 172.21.15.79:0/1241349911 wait complete. 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 Processor -- start 2024-04-22T06:42:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.818+0000 7fe55f8c1640 1 -- start start 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe558100e60 0x7fe558198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 0x7fe558198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe558102a00 0x7fe55819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe55810ca60 con 0x7fe558102040 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe55810c8e0 con 0x7fe558102a00 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe55810cbe0 con 0x7fe558100e60 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 0x7fe558198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 0x7fe558198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51642/0 (socket says 172.21.15.79:51642) 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 -- 172.21.15.79:0/896874557 learned_addr learned my addr 172.21.15.79:0/896874557 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55d65d640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe558100e60 0x7fe558198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55de5e640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe558102a00 0x7fe55819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 -- 172.21.15.79:0/896874557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe558100e60 msgr2=0x7fe558198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe558100e60 0x7fe558198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 -- 172.21.15.79:0/896874557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe558102a00 msgr2=0x7fe55819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe558102a00 0x7fe55819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 -- 172.21.15.79:0/896874557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55819fe20 con 0x7fe558102040 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55ce5c640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 0x7fe558198740 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe54800b4a0 tx=0x7fe54800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548014070 con 0x7fe558102040 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe548010ac0 con 0x7fe558102040 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5581a0110 con 0x7fe558102040 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548017c30 con 0x7fe558102040 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.819+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5581a0610 con 0x7fe558102040 2024-04-22T06:42:27.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.820+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5581a08d0 con 0x7fe558102040 2024-04-22T06:42:27.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.821+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe548017070 con 0x7fe558102040 2024-04-22T06:42:27.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.822+0000 7fe54e7fc640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe5440a3bb0 0x7fe5440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.822+0000 7fe55d65d640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe5440a3bb0 0x7fe5440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.822+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe5480cf5a0 con 0x7fe558102040 2024-04-22T06:42:27.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.822+0000 7fe55d65d640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe5440a3bb0 0x7fe5440a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe550005fd0 tx=0x7fe55000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:27.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:27.826+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe54809ba80 con 0x7fe558102040 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: pgmap v290: 97 pgs: 15 creating+peering, 38 active+clean, 44 unknown; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2020153586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:42:27.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2184018376' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: pgmap v290: 97 pgs: 15 creating+peering, 38 active+clean, 44 unknown; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2020153586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:42:27.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2184018376' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:42:28.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.043+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fe5581a0aa0 con 0x7fe558102040 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: pgmap v290: 97 pgs: 15 creating+peering, 38 active+clean, 44 unknown; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2020153586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-22T06:42:28.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:42:28.191 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2184018376' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:42:28.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.709+0000 7fe54e7fc640 1 -- 172.21.15.79:0/896874557 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fe54809fd80 con 0x7fe558102040 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe5440a3bb0 msgr2=0x7fe5440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe5440a3bb0 0x7fe5440a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe550005fd0 tx=0x7fe55000b410 comp rx=0 tx=0).stop 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 msgr2=0x7fe558198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 0x7fe558198740 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe54800b4a0 tx=0x7fe54800b960 comp rx=0 tx=0).stop 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 shutdown_connections 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe5440a3bb0 0x7fe5440a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe558102a00 0x7fe55819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe558102040 0x7fe558198740 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 --2- 172.21.15.79:0/896874557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe558100e60 0x7fe558198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 >> 172.21.15.79:0/896874557 conn(0x7fe5580fc440 msgr2=0x7fe5580fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:28.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 shutdown_connections 2024-04-22T06:42:28.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.712+0000 7fe55f8c1640 1 -- 172.21.15.79:0/896874557 wait complete. 2024-04-22T06:42:28.725 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-22T06:42:28.725 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 -- 172.21.15.79:0/1168124303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c100e60 msgr2=0x7f073c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 --2- 172.21.15.79:0/1168124303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c100e60 0x7f073c101280 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f072c009900 tx=0x7f072c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 -- 172.21.15.79:0/1168124303 shutdown_connections 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 --2- 172.21.15.79:0/1168124303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c102a00 0x7f073c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 --2- 172.21.15.79:0/1168124303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f073c102040 0x7f073c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 --2- 172.21.15.79:0/1168124303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c100e60 0x7f073c101280 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 -- 172.21.15.79:0/1168124303 >> 172.21.15.79:0/1168124303 conn(0x7f073c0fc440 msgr2=0x7f073c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 -- 172.21.15.79:0/1168124303 shutdown_connections 2024-04-22T06:42:28.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.832+0000 7f07421bd640 1 -- 172.21.15.79:0/1168124303 wait complete. 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 Processor -- start 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 -- start start 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f073c100e60 0x7f073c10b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 0x7f073c109ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c102a00 0x7f073c10a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f073c10db70 con 0x7f073c102040 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f073c10d9f0 con 0x7f073c102a00 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07421bd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f073c10dcf0 con 0x7f073c100e60 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 0x7f073c109ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 0x7f073c109ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51652/0 (socket says 172.21.15.79:51652) 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 -- 172.21.15.79:0/3289334470 learned_addr learned my addr 172.21.15.79:0/3289334470 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f073bfff640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c102a00 0x7f073c10a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f073b7fe640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f073c100e60 0x7f073c10b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 -- 172.21.15.79:0/3289334470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f073c100e60 msgr2=0x7f073c10b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f073c100e60 0x7f073c10b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 -- 172.21.15.79:0/3289334470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c102a00 msgr2=0x7f073c10a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c102a00 0x7f073c10a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 -- 172.21.15.79:0/3289334470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c10ac70 con 0x7f073c102040 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f0733fff640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 0x7f073c109ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f072400b4a0 tx=0x7f072400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.833+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724014070 con 0x7f073c102040 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.834+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073c1a23d0 con 0x7f073c102040 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.834+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0724010ac0 con 0x7f073c102040 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.834+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724017c30 con 0x7f073c102040 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.834+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073c1a29a0 con 0x7f073c102040 2024-04-22T06:42:28.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.835+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f073c108280 con 0x7f073c102040 2024-04-22T06:42:28.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.836+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0724017070 con 0x7f073c102040 2024-04-22T06:42:28.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.836+0000 7f07397fa640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f07280a3bb0 0x7f07280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.836+0000 7f073b7fe640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f07280a3bb0 0x7f07280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.836+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f07240cf9f0 con 0x7f073c102040 2024-04-22T06:42:28.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.837+0000 7f073b7fe640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f07280a3bb0 0x7f07280a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f072c006010 tx=0x7f072c005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:28.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:28.841+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f072409c520 con 0x7f073c102040 2024-04-22T06:42:28.952 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: 2024-04-22T06:42:28.748+0000 7f2935ae5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: 2024-04-22T06:42:28.748+0000 7f2935ae5700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: 2024-04-22T06:42:28.748+0000 7f2935ae5700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2184018376' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] up:active 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:42:28.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-22T06:42:28.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-22T06:42:28.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:42:28.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/896874557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2184018376' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] up:active 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-22T06:42:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:42:28.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/896874557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:42:29.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.046+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f073c062390 con 0x7f07280a3bb0 2024-04-22T06:42:29.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.071+0000 7f07397fa640 1 -- 172.21.15.79:0/3289334470 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f073c062390 con 0x7f07280a3bb0 2024-04-22T06:42:29.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f07280a3bb0 msgr2=0x7f07280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f07280a3bb0 0x7f07280a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f072c006010 tx=0x7f072c005be0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 msgr2=0x7f073c109ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 0x7f073c109ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f072400b4a0 tx=0x7f072400b960 comp rx=0 tx=0).stop 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 shutdown_connections 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f07280a3bb0 0x7f07280a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073c102a00 0x7f073c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f073c102040 0x7f073c109ea0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 --2- 172.21.15.79:0/3289334470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f073c100e60 0x7f073c10b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 >> 172.21.15.79:0/3289334470 conn(0x7f073c0fc440 msgr2=0x7f073c105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 shutdown_connections 2024-04-22T06:42:29.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.073+0000 7f07421bd640 1 -- 172.21.15.79:0/3289334470 wait complete. 2024-04-22T06:42:29.083 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2184018376' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] up:active 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:42:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/896874557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:42:29.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 -- 172.21.15.79:0/4104884767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 msgr2=0x7f0a8c101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4104884767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c101640 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009900 tx=0x7f0a8002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 -- 172.21.15.79:0/4104884767 shutdown_connections 2024-04-22T06:42:29.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4104884767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a8c101b80 0x7f0a8c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4104884767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c101640 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4104884767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c100880 0x7f0a8c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 -- 172.21.15.79:0/4104884767 >> 172.21.15.79:0/4104884767 conn(0x7f0a8c0fc400 msgr2=0x7f0a8c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 -- 172.21.15.79:0/4104884767 shutdown_connections 2024-04-22T06:42:29.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.188+0000 7f0a92f15640 1 -- 172.21.15.79:0/4104884767 wait complete. 2024-04-22T06:42:29.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 Processor -- start 2024-04-22T06:42:29.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 -- start start 2024-04-22T06:42:29.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c100880 0x7f0a8c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a8c101b80 0x7f0a8c1a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c10fd10 con 0x7f0a8c1011e0 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c10fb90 con 0x7f0a8c101b80 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a92f15640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c10fe90 con 0x7f0a8c100880 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c19a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51674/0 (socket says 172.21.15.79:51674) 2024-04-22T06:42:29.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 -- 172.21.15.79:0/4027079524 learned_addr learned my addr 172.21.15.79:0/4027079524 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:29.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a914b2640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a8c101b80 0x7f0a8c1a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a90cb1640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c100880 0x7f0a8c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 -- 172.21.15.79:0/4027079524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c100880 msgr2=0x7f0a8c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c100880 0x7f0a8c19a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 -- 172.21.15.79:0/4027079524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a8c101b80 msgr2=0x7f0a8c1a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a8c101b80 0x7f0a8c1a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 -- 172.21.15.79:0/4027079524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c1a2090 con 0x7f0a8c1011e0 2024-04-22T06:42:29.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.189+0000 7f0a8bfff640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c19a9b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0a80006010 tx=0x7f0a800045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.190+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80043020 con 0x7f0a8c1011e0 2024-04-22T06:42:29.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.190+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a8002dd20 con 0x7f0a8c1011e0 2024-04-22T06:42:29.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.190+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c1a2320 con 0x7f0a8c1011e0 2024-04-22T06:42:29.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.190+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003b20 con 0x7f0a8c1011e0 2024-04-22T06:42:29.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.190+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1a2940 con 0x7f0a8c1011e0 2024-04-22T06:42:29.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.191+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c10b860 con 0x7f0a8c1011e0 2024-04-22T06:42:29.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.192+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0a80046020 con 0x7f0a8c1011e0 2024-04-22T06:42:29.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.192+0000 7f0a89ffb640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a740a3c30 0x7f0a740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.192+0000 7f0a90cb1640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a740a3c30 0x7f0a740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.192+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0a800f2430 con 0x7f0a8c1011e0 2024-04-22T06:42:29.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.193+0000 7f0a90cb1640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a740a3c30 0x7f0a740a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00a810 tx=0x7f0a7c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.197+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a8004d020 con 0x7f0a8c1011e0 2024-04-22T06:42:29.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.405+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0a8c062420 con 0x7f0a740a3c30 2024-04-22T06:42:29.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.428+0000 7f0a89ffb640 1 -- 172.21.15.79:0/4027079524 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0a8c062420 con 0x7f0a740a3c30 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a740a3c30 msgr2=0x7f0a740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a740a3c30 0x7f0a740a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00a810 tx=0x7f0a7c008040 comp rx=0 tx=0).stop 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 msgr2=0x7f0a8c19a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c19a9b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0a80006010 tx=0x7f0a800045e0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 shutdown_connections 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0a740a3c30 0x7f0a740a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a8c101b80 0x7f0a8c1a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0a8c1011e0 0x7f0a8c19a9b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 --2- 172.21.15.79:0/4027079524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c100880 0x7f0a8c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 >> 172.21.15.79:0/4027079524 conn(0x7f0a8c0fc400 msgr2=0x7f0a8c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 shutdown_connections 2024-04-22T06:42:29.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.430+0000 7f0a92f15640 1 -- 172.21.15.79:0/4027079524 wait complete. 2024-04-22T06:42:29.439 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:42:29.050+0000 7f5274f8c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:29.439 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:42:29.050+0000 7f5274f8c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:29.439 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:42:29.050+0000 7f5274f8c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:29.439 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:42:29.050+0000 7f5274f8c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:29.439 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:42:29.050+0000 7f5274f8c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:29.440 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.543+0000 7fd1f9104640 1 -- 172.21.15.79:0/2829532250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4100e90 msgr2=0x7fd1f41012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 --2- 172.21.15.79:0/2829532250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4100e90 0x7fd1f41012b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc009900 tx=0x7fd1dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 -- 172.21.15.79:0/2829532250 shutdown_connections 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 --2- 172.21.15.79:0/2829532250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1f4101fe0 0x7fd1f4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 --2- 172.21.15.79:0/2829532250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1f41029a0 0x7fd1f41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 --2- 172.21.15.79:0/2829532250 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4100e90 0x7fd1f41012b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 -- 172.21.15.79:0/2829532250 >> 172.21.15.79:0/2829532250 conn(0x7fd1f40fc400 msgr2=0x7fd1f40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 -- 172.21.15.79:0/2829532250 shutdown_connections 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 -- 172.21.15.79:0/2829532250 wait complete. 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 Processor -- start 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.544+0000 7fd1f9104640 1 -- start start 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1f4100e90 0x7fd1f406b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 0x7fd1f419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1f41029a0 0x7fd1f419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f410ca30 con 0x7fd1f4101fe0 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f410c8b0 con 0x7fd1f41029a0 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f410cbb0 con 0x7fd1f4100e90 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 0x7fd1f419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 0x7fd1f419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51702/0 (socket says 172.21.15.79:51702) 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 -- 172.21.15.79:0/286152318 learned_addr learned my addr 172.21.15.79:0/286152318 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f359e640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1f41029a0 0x7fd1f419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f2d9d640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1f4100e90 0x7fd1f406b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 -- 172.21.15.79:0/286152318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1f4100e90 msgr2=0x7fd1f406b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1f4100e90 0x7fd1f406b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 -- 172.21.15.79:0/286152318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1f41029a0 msgr2=0x7fd1f419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1f41029a0 0x7fd1f419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 -- 172.21.15.79:0/286152318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f419fe00 con 0x7fd1f4101fe0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f259c640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 0x7fd1f419cf00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd1e400e9c0 tx=0x7fd1e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e400cd80 con 0x7fd1f4101fe0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f41a00f0 con 0x7fd1f4101fe0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1e40035f0 con 0x7fd1f4101fe0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4010690 con 0x7fd1f4101fe0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.545+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f41a0640 con 0x7fd1f4101fe0 2024-04-22T06:42:29.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.546+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f4108220 con 0x7fd1f4101fe0 2024-04-22T06:42:29.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.547+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd1e40037a0 con 0x7fd1f4101fe0 2024-04-22T06:42:29.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.548+0000 7fd1ebfff640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1e00a3bb0 0x7fd1e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.548+0000 7fd1f2d9d640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1e00a3bb0 0x7fd1e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.548+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd1e40d0610 con 0x7fd1f4101fe0 2024-04-22T06:42:29.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.548+0000 7fd1f2d9d640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1e00a3bb0 0x7fd1e00a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc000c00 tx=0x7fd1dc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.552+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd1e409d140 con 0x7fd1f4101fe0 2024-04-22T06:42:29.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.761+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd1f4062390 con 0x7fd1e00a3bb0 2024-04-22T06:42:29.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.767+0000 7fd1ebfff640 1 -- 172.21.15.79:0/286152318 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd1f4062390 con 0x7fd1e00a3bb0 2024-04-22T06:42:29.766 INFO:teuthology.orchestra.run.smithi079.stdout:/volumes/qa/sv_0/2ff83c91-54bc-4eb6-ae2e-0c53e2448cda 2024-04-22T06:42:29.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1e00a3bb0 msgr2=0x7fd1e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1e00a3bb0 0x7fd1e00a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc000c00 tx=0x7fd1dc02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 msgr2=0x7fd1f419cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 0x7fd1f419cf00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd1e400e9c0 tx=0x7fd1e400ee80 comp rx=0 tx=0).stop 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 shutdown_connections 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1e00a3bb0 0x7fd1e00a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1f41029a0 0x7fd1f419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1f4101fe0 0x7fd1f419cf00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 --2- 172.21.15.79:0/286152318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1f4100e90 0x7fd1f406b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 >> 172.21.15.79:0/286152318 conn(0x7fd1f40fc400 msgr2=0x7fd1f41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 shutdown_connections 2024-04-22T06:42:29.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.770+0000 7fd1f9104640 1 -- 172.21.15.79:0/286152318 wait complete. 2024-04-22T06:42:29.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-22T06:42:29.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 -- 172.21.15.79:0/2424754655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c105360 msgr2=0x7f3c5c107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 --2- 172.21.15.79:0/2424754655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c105360 0x7f3c5c107750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009900 tx=0x7f3c4402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 -- 172.21.15.79:0/2424754655 shutdown_connections 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 --2- 172.21.15.79:0/2424754655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c5c107c90 0x7f3c5c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 --2- 172.21.15.79:0/2424754655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c105360 0x7f3c5c107750 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 --2- 172.21.15.79:0/2424754655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c5c102a30 0x7f3c5c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 -- 172.21.15.79:0/2424754655 >> 172.21.15.79:0/2424754655 conn(0x7f3c5c0fc420 msgr2=0x7f3c5c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 -- 172.21.15.79:0/2424754655 shutdown_connections 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 -- 172.21.15.79:0/2424754655 wait complete. 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.884+0000 7f3c61806640 1 Processor -- start 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 -- start start 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c5c102a30 0x7f3c5c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c5c105360 0x7f3c5c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 0x7f3c5c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c10daf0 con 0x7f3c5c107c90 2024-04-22T06:42:29.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c10d970 con 0x7f3c5c102a30 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c61806640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c10dc70 con 0x7f3c5c105360 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 0x7f3c5c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 0x7f3c5c19f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51726/0 (socket says 172.21.15.79:51726) 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 -- 172.21.15.79:0/2294297805 learned_addr learned my addr 172.21.15.79:0/2294297805 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5affd640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c5c102a30 0x7f3c5c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5a7fc640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c5c105360 0x7f3c5c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 -- 172.21.15.79:0/2294297805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c5c105360 msgr2=0x7f3c5c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c5c105360 0x7f3c5c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 -- 172.21.15.79:0/2294297805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c5c102a30 msgr2=0x7f3c5c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c5c102a30 0x7f3c5c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 -- 172.21.15.79:0/2294297805 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c19fe50 con 0x7f3c5c107c90 2024-04-22T06:42:29.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c5b7fe640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 0x7f3c5c19f7e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400eaa0 tx=0x7f3c5400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.885+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5400cdb0 con 0x7f3c5c107c90 2024-04-22T06:42:29.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.886+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c1a0140 con 0x7f3c5c107c90 2024-04-22T06:42:29.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.886+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c54005020 con 0x7f3c5c107c90 2024-04-22T06:42:29.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.886+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540116d0 con 0x7f3c5c107c90 2024-04-22T06:42:29.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.886+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5c1a0660 con 0x7f3c5c107c90 2024-04-22T06:42:29.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.887+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5c0fdf10 con 0x7f3c5c107c90 2024-04-22T06:42:29.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.888+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3c540051d0 con 0x7f3c5c107c90 2024-04-22T06:42:29.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.888+0000 7f3c60804640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c480a3bb0 0x7f3c480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.888+0000 7f3c5affd640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c480a3bb0 0x7f3c480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.888+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3c540d03e0 con 0x7f3c5c107c90 2024-04-22T06:42:29.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.889+0000 7f3c5affd640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c480a3bb0 0x7f3c480a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c005d00 tx=0x7f3c4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:29.893+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c54016030 con 0x7f3c5c107c90 2024-04-22T06:42:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-mon[27946]: pgmap v291: 97 pgs: 8 creating+peering, 89 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s wr, 10 op/s 2024-04-22T06:42:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/896874557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:42:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] up:active 2024-04-22T06:42:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-22T06:42:29.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:29 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:42:30.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.096+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3c5c062390 con 0x7f3c480a3bb0 2024-04-22T06:42:30.102 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:29 smithi193 ceph-mon[31815]: pgmap v291: 97 pgs: 8 creating+peering, 89 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s wr, 10 op/s 2024-04-22T06:42:30.102 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:29 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/896874557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:42:30.102 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:29 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] up:active 2024-04-22T06:42:30.102 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:29 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-22T06:42:30.102 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:29 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:42:30.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.121+0000 7f3c60804640 1 -- 172.21.15.79:0/2294297805 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3c5c062390 con 0x7f3c480a3bb0 2024-04-22T06:42:30.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c480a3bb0 msgr2=0x7f3c480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c480a3bb0 0x7f3c480a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c005d00 tx=0x7f3c4c00b410 comp rx=0 tx=0).stop 2024-04-22T06:42:30.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 msgr2=0x7f3c5c19f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 0x7f3c5c19f7e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400eaa0 tx=0x7f3c5400ef60 comp rx=0 tx=0).stop 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 shutdown_connections 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3c480a3bb0 0x7f3c480a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3c5c107c90 0x7f3c5c19f7e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c5c105360 0x7f3c5c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 --2- 172.21.15.79:0/2294297805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c5c102a30 0x7f3c5c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 >> 172.21.15.79:0/2294297805 conn(0x7f3c5c0fc420 msgr2=0x7f3c5c1048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 shutdown_connections 2024-04-22T06:42:30.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.123+0000 7f3c61806640 1 -- 172.21.15.79:0/2294297805 wait complete. 2024-04-22T06:42:30.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:29 smithi190 ceph-mon[31421]: pgmap v291: 97 pgs: 8 creating+peering, 89 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s wr, 10 op/s 2024-04-22T06:42:30.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:29 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/896874557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:42:30.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:29 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] up:active 2024-04-22T06:42:30.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:29 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-22T06:42:30.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:29 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:42:30.132 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 -- 172.21.15.79:0/1821774678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50107c90 msgr2=0x7f9a5010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 --2- 172.21.15.79:0/1821774678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50107c90 0x7f9a5010a0e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009ca0 tx=0x7f9a4802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 -- 172.21.15.79:0/1821774678 shutdown_connections 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 --2- 172.21.15.79:0/1821774678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a50105360 0x7f9a50107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 --2- 172.21.15.79:0/1821774678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50107c90 0x7f9a5010a0e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 --2- 172.21.15.79:0/1821774678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a50102a30 0x7f9a50104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 -- 172.21.15.79:0/1821774678 >> 172.21.15.79:0/1821774678 conn(0x7f9a500fc440 msgr2=0x7f9a500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 -- 172.21.15.79:0/1821774678 shutdown_connections 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.236+0000 7f9a56567640 1 -- 172.21.15.79:0/1821774678 wait complete. 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 Processor -- start 2024-04-22T06:42:30.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 -- start start 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 0x7f9a50198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a50105360 0x7f9a501986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a50107c90 0x7f9a5019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5010daf0 con 0x7f9a50102a30 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5010d970 con 0x7f9a50107c90 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a56567640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5010dc70 con 0x7f9a50105360 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 0x7f9a50198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 0x7f9a50198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51752/0 (socket says 172.21.15.79:51752) 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 -- 172.21.15.79:0/1533461655 learned_addr learned my addr 172.21.15.79:0/1533461655 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4f7fe640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a50105360 0x7f9a501986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a54b04640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a50107c90 0x7f9a5019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 -- 172.21.15.79:0/1533461655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a50105360 msgr2=0x7f9a501986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a50105360 0x7f9a501986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 -- 172.21.15.79:0/1533461655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a50107c90 msgr2=0x7f9a5019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a50107c90 0x7f9a5019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.237+0000 7f9a4ffff640 1 -- 172.21.15.79:0/1533461655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5019fd90 con 0x7f9a50102a30 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.238+0000 7f9a4ffff640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 0x7f9a50198170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000d770 tx=0x7f9a4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.238+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40017070 con 0x7f9a50102a30 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.238+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a400054d0 con 0x7f9a50102a30 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.238+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a501a0080 con 0x7f9a50102a30 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.238+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40004c30 con 0x7f9a50102a30 2024-04-22T06:42:30.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.238+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a501a0580 con 0x7f9a50102a30 2024-04-22T06:42:30.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.239+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a500fdf30 con 0x7f9a50102a30 2024-04-22T06:42:30.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.240+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9a40004e10 con 0x7f9a50102a30 2024-04-22T06:42:30.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.240+0000 7f9a4d7fa640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380a3c30 0x7f9a380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.241+0000 7f9a4f7fe640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380a3c30 0x7f9a380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.241+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9a400cfd60 con 0x7f9a50102a30 2024-04-22T06:42:30.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.241+0000 7f9a4f7fe640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380a3c30 0x7f9a380a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a50199680 tx=0x7f9a4400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.245+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a4009c1a0 con 0x7f9a50102a30 2024-04-22T06:42:30.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.448+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f9a50062420 con 0x7f9a380a3c30 2024-04-22T06:42:30.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.452+0000 7f9a4d7fa640 1 -- 172.21.15.79:0/1533461655 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9a50062420 con 0x7f9a380a3c30 2024-04-22T06:42:30.451 INFO:teuthology.orchestra.run.smithi079.stdout:/volumes/qa/sv_1/c65b5f2f-2526-4052-a234-373065fd4b48 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380a3c30 msgr2=0x7f9a380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380a3c30 0x7f9a380a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a50199680 tx=0x7f9a4400b390 comp rx=0 tx=0).stop 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 msgr2=0x7f9a50198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 0x7f9a50198170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000d770 tx=0x7f9a4000dc30 comp rx=0 tx=0).stop 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 shutdown_connections 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a380a3c30 0x7f9a380a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a50107c90 0x7f9a5019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a50105360 0x7f9a501986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 --2- 172.21.15.79:0/1533461655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a50102a30 0x7f9a50198170 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 >> 172.21.15.79:0/1533461655 conn(0x7f9a500fc440 msgr2=0x7f9a50103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 shutdown_connections 2024-04-22T06:42:30.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.454+0000 7f9a56567640 1 -- 172.21.15.79:0/1533461655 wait complete. 2024-04-22T06:42:30.464 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 -- 172.21.15.79:0/2846517757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 msgr2=0x7fe998100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 --2- 172.21.15.79:0/2846517757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe998100670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 -- 172.21.15.79:0/2846517757 shutdown_connections 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 --2- 172.21.15.79:0/2846517757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9981032c0 0x7fe998103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 --2- 172.21.15.79:0/2846517757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe998103c80 0x7fe99810a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 --2- 172.21.15.79:0/2846517757 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe998100670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 -- 172.21.15.79:0/2846517757 >> 172.21.15.79:0/2846517757 conn(0x7fe9980fbb80 msgr2=0x7fe9980fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 -- 172.21.15.79:0/2846517757 shutdown_connections 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 -- 172.21.15.79:0/2846517757 wait complete. 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 Processor -- start 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.562+0000 7fe99e477640 1 -- start start 2024-04-22T06:42:30.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe9981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9981032c0 0x7fe998198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe998103c80 0x7fe99819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810daf0 con 0x7fe998100250 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810d970 con 0x7fe998103c80 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810dc70 con 0x7fe9981032c0 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe9981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe9981981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51774/0 (socket says 172.21.15.79:51774) 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 -- 172.21.15.79:0/764107072 learned_addr learned my addr 172.21.15.79:0/764107072 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe9977fe640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9981032c0 0x7fe998198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99ca14640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe998103c80 0x7fe99819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 -- 172.21.15.79:0/764107072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9981032c0 msgr2=0x7fe998198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9981032c0 0x7fe998198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 -- 172.21.15.79:0/764107072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe998103c80 msgr2=0x7fe99819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe998103c80 0x7fe99819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 -- 172.21.15.79:0/764107072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99819fdf0 con 0x7fe998100250 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe997fff640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe9981981d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe98c000c00 tx=0x7fe98c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c043020 con 0x7fe998100250 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9981a0080 con 0x7fe998100250 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe98c004a80 con 0x7fe998100250 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c003cd0 con 0x7fe998100250 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.563+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9981a0620 con 0x7fe998100250 2024-04-22T06:42:30.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.564+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe998109470 con 0x7fe998100250 2024-04-22T06:42:30.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.565+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe98c046020 con 0x7fe998100250 2024-04-22T06:42:30.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.566+0000 7fe9957fa640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe9800a3bb0 0x7fe9800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.566+0000 7fe9977fe640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe9800a3bb0 0x7fe9800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.566+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe98c0f2ba0 con 0x7fe998100250 2024-04-22T06:42:30.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.566+0000 7fe9977fe640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe9800a3bb0 0x7fe9800a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe9981996e0 tx=0x7fe98800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.570+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe98c04d020 con 0x7fe998100250 2024-04-22T06:42:30.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.725+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe98c0bb640 con 0x7fe998100250 2024-04-22T06:42:30.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.790+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe99802d5c0 con 0x7fe998100250 2024-04-22T06:42:30.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.791+0000 7fe9957fa640 1 -- 172.21.15.79:0/764107072 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fe98c0bf6d0 con 0x7fe998100250 2024-04-22T06:42:30.790 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:30.790 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:42:28.714055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24436,24440,24467]},"id":1}]} 2024-04-22T06:42:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:42:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe9800a3bb0 msgr2=0x7fe9800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe9800a3bb0 0x7fe9800a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe9981996e0 tx=0x7fe98800b410 comp rx=0 tx=0).stop 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 msgr2=0x7fe9981981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe9981981d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe98c000c00 tx=0x7fe98c004270 comp rx=0 tx=0).stop 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 shutdown_connections 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe9800a3bb0 0x7fe9800a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe998103c80 0x7fe99819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9981032c0 0x7fe998198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 --2- 172.21.15.79:0/764107072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe998100250 0x7fe9981981d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 >> 172.21.15.79:0/764107072 conn(0x7fe9980fbb80 msgr2=0x7fe998106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 shutdown_connections 2024-04-22T06:42:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.793+0000 7fe99e477640 1 -- 172.21.15.79:0/764107072 wait complete. 2024-04-22T06:42:30.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 -- 172.21.15.79:0/3155446423 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c102a70 msgr2=0x7ff65c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3155446423 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c102a70 0x7ff65c104e60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff650009900 tx=0x7ff65002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 -- 172.21.15.79:0/3155446423 shutdown_connections 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3155446423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff65c1053a0 0x7ff65c107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3155446423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff65c107cd0 0x7ff65c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3155446423 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c102a70 0x7ff65c104e60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.902+0000 7ff661ddb640 1 -- 172.21.15.79:0/3155446423 >> 172.21.15.79:0/3155446423 conn(0x7ff65c0fc3e0 msgr2=0x7ff65c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 -- 172.21.15.79:0/3155446423 shutdown_connections 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 -- 172.21.15.79:0/3155446423 wait complete. 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 Processor -- start 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 -- start start 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff65c102a70 0x7ff65c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff65c1053a0 0x7ff65c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 0x7ff65c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c10daa0 con 0x7ff65c107cd0 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c10d920 con 0x7ff65c102a70 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.903+0000 7ff661ddb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c10dc20 con 0x7ff65c1053a0 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 0x7ff65c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 0x7ff65c19f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51800/0 (socket says 172.21.15.79:51800) 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 -- 172.21.15.79:0/3700933668 learned_addr learned my addr 172.21.15.79:0/3700933668 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65b7fe640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff65c102a70 0x7ff65c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65affd640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff65c1053a0 0x7ff65c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 -- 172.21.15.79:0/3700933668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff65c1053a0 msgr2=0x7ff65c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff65c1053a0 0x7ff65c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 -- 172.21.15.79:0/3700933668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff65c102a70 msgr2=0x7ff65c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff65c102a70 0x7ff65c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 -- 172.21.15.79:0/3700933668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c19fe30 con 0x7ff65c107cd0 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff65bfff640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 0x7ff65c19f7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff654009fd0 tx=0x7ff65400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654017070 con 0x7ff65c107cd0 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff654003e20 con 0x7ff65c107cd0 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c1a0120 con 0x7ff65c107cd0 2024-04-22T06:42:30.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654010460 con 0x7ff65c107cd0 2024-04-22T06:42:30.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.904+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c1a0620 con 0x7ff65c107cd0 2024-04-22T06:42:30.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.905+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c0fded0 con 0x7ff65c107cd0 2024-04-22T06:42:30.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.906+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff654003970 con 0x7ff65c107cd0 2024-04-22T06:42:30.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.907+0000 7ff658ff9640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff6440a3c40 0x7ff6440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.907+0000 7ff65b7fe640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff6440a3c40 0x7ff6440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.907+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff6540d00f0 con 0x7ff65c107cd0 2024-04-22T06:42:30.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.907+0000 7ff65b7fe640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff6440a3c40 0x7ff6440a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff650009230 tx=0x7ff650005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:30.911+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff65409c4d0 con 0x7ff65c107cd0 2024-04-22T06:42:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:30 smithi079 ceph-mon[27946]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:30 smithi079 ceph-mon[27946]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:30 smithi079 ceph-mon[27946]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:30 smithi190 ceph-mon[31421]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:30 smithi190 ceph-mon[31421]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:30 smithi190 ceph-mon[31421]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.091+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff65c062b50 con 0x7ff65c107cd0 2024-04-22T06:42:31.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.093+0000 7ff658ff9640 1 -- 172.21.15.79:0/3700933668 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22217 (secure 0 0 0) 0x7ff6540a07d0 con 0x7ff65c107cd0 2024-04-22T06:42:31.092 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:31.092 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":85,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:42:25.177009+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff6440a3c40 msgr2=0x7ff6440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff6440a3c40 0x7ff6440a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff650009230 tx=0x7ff650005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 msgr2=0x7ff65c19f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 0x7ff65c19f7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff654009fd0 tx=0x7ff65400b260 comp rx=0 tx=0).stop 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 shutdown_connections 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff6440a3c40 0x7ff6440a6100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff65c107cd0 0x7ff65c19f7c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff65c1053a0 0x7ff65c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 --2- 172.21.15.79:0/3700933668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff65c102a70 0x7ff65c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 >> 172.21.15.79:0/3700933668 conn(0x7ff65c0fc3e0 msgr2=0x7ff65c1035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 shutdown_connections 2024-04-22T06:42:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.095+0000 7ff661ddb640 1 -- 172.21.15.79:0/3700933668 wait complete. 2024-04-22T06:42:31.105 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T06:42:31.105 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:42:31.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:30 smithi193 ceph-mon[31815]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:30 smithi193 ceph-mon[31815]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:30 smithi193 ceph-mon[31815]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 -- 172.21.15.79:0/1607534148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 msgr2=0x7f75e8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 --2- 172.21.15.79:0/1607534148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e8103610 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009900 tx=0x7f75dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 -- 172.21.15.79:0/1607534148 shutdown_connections 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 --2- 172.21.15.79:0/1607534148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 0x7f75e810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 --2- 172.21.15.79:0/1607534148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e8103610 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 --2- 172.21.15.79:0/1607534148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75e8101fa0 0x7f75e81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 -- 172.21.15.79:0/1607534148 >> 172.21.15.79:0/1607534148 conn(0x7f75e80fd530 msgr2=0x7f75e80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 -- 172.21.15.79:0/1607534148 shutdown_connections 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.204+0000 7f75eefab640 1 -- 172.21.15.79:0/1607534148 wait complete. 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 Processor -- start 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 -- start start 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75e8101fa0 0x7f75e8198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 0x7f75e819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f75e810dae0 con 0x7f75e8103190 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f75e810d960 con 0x7f75e8101fa0 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75e810dc60 con 0x7f75e8103b50 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51820/0 (socket says 172.21.15.79:51820) 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 -- 172.21.15.79:0/3280065857 learned_addr learned my addr 172.21.15.79:0/3280065857 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75ecd47640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75e8101fa0 0x7f75e8198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75ed548640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 0x7f75e819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 -- 172.21.15.79:0/3280065857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 msgr2=0x7f75e819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 0x7f75e819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 -- 172.21.15.79:0/3280065857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75e8101fa0 msgr2=0x7f75e8198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75e8101fa0 0x7f75e8198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 -- 172.21.15.79:0/3280065857 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e819fd80 con 0x7f75e8103190 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75ed548640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 0x7f75e819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e7fff640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e81986a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f75dc006010 tx=0x7f75dc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc043020 con 0x7f75e8103190 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.205+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e81a0010 con 0x7f75e8103190 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.206+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75dc004440 con 0x7f75e8103190 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.206+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc003bf0 con 0x7f75e8103190 2024-04-22T06:42:31.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.206+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e81a0560 con 0x7f75e8103190 2024-04-22T06:42:31.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.207+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e81092c0 con 0x7f75e8103190 2024-04-22T06:42:31.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.208+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f75dc046020 con 0x7f75e8103190 2024-04-22T06:42:31.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.209+0000 7f75e5ffb640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75d00a3c40 0x7f75d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.209+0000 7f75ecd47640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75d00a3c40 0x7f75d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.209+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f75dc0f2680 con 0x7f75e8103190 2024-04-22T06:42:31.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.209+0000 7f75ecd47640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75d00a3c40 0x7f75d00a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75e8102ff0 tx=0x7f75d800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.213+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75dc04d020 con 0x7f75e8103190 2024-04-22T06:42:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.391+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f75e8062390 con 0x7f75d00a3c40 2024-04-22T06:42:31.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.399+0000 7f75e5ffb640 1 -- 172.21.15.79:0/3280065857 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258939 (secure 0 0 0) 0x7f75e8062390 con 0x7f75d00a3c40 2024-04-22T06:42:31.400 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:31.401 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75d00a3c40 msgr2=0x7f75d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75d00a3c40 0x7f75d00a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75e8102ff0 tx=0x7f75d800b390 comp rx=0 tx=0).stop 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 msgr2=0x7f75e81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e81986a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f75dc006010 tx=0x7f75dc004040 comp rx=0 tx=0).stop 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 shutdown_connections 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f75d00a3c40 0x7f75d00a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75e8103b50 0x7f75e819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75e8103190 0x7f75e81986a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 --2- 172.21.15.79:0/3280065857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75e8101fa0 0x7f75e8198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 >> 172.21.15.79:0/3280065857 conn(0x7f75e80fd530 msgr2=0x7f75e8106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 shutdown_connections 2024-04-22T06:42:31.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.402+0000 7f75eefab640 1 -- 172.21.15.79:0/3280065857 wait complete. 2024-04-22T06:42:31.411 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-22T06:42:30.645256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614087,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":153,"num_read_kb":150,"num_write":284,"num_write_kb":2989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":243,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376912,"kb_used_data":11056,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564912,"statfs":{"total":1151940427776,"available":1151554469888,"internally_reserved":0,"allocated":11321344,"data_stored":7065389,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23719,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":17,"num_read_kb":31,"num_write":103,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.990426"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103478+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:42:26.103478+0000","last_peered":"2024-04-22T06:42:26.103478+0000","last_clean":"2024-04-22T06:42:26.103478+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:42:26.103478+0000","last_undegraded":"2024-04-22T06:42:26.103478+0000","last_fullsized":"2024-04-22T06:42:26.103478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102783+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:42:26.102783+0000","last_peered":"2024-04-22T06:42:26.102783+0000","last_clean":"2024-04-22T06:42:26.102783+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:42:26.102783+0000","last_undegraded":"2024-04-22T06:42:26.102783+0000","last_fullsized":"2024-04-22T06:42:26.102783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102587+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:42:26.102587+0000","last_peered":"2024-04-22T06:42:26.102587+0000","last_clean":"2024-04-22T06:42:26.102587+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:42:26.102587+0000","last_undegraded":"2024-04-22T06:42:26.102587+0000","last_fullsized":"2024-04-22T06:42:26.102587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104210+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:42:26.104210+0000","last_peered":"2024-04-22T06:42:26.104210+0000","last_clean":"2024-04-22T06:42:26.104210+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:42:26.104210+0000","last_undegraded":"2024-04-22T06:42:26.104210+0000","last_fullsized":"2024-04-22T06:42:26.104210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103189+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:42:26.103189+0000","last_peered":"2024-04-22T06:42:26.103189+0000","last_clean":"2024-04-22T06:42:26.103189+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:42:26.103189+0000","last_undegraded":"2024-04-22T06:42:26.103189+0000","last_fullsized":"2024-04-22T06:42:26.103189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102710+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:42:26.102710+0000","last_peered":"2024-04-22T06:42:26.102710+0000","last_clean":"2024-04-22T06:42:26.102710+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:42:26.102710+0000","last_undegraded":"2024-04-22T06:42:26.102710+0000","last_fullsized":"2024-04-22T06:42:26.102710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103001+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:42:26.103001+0000","last_peered":"2024-04-22T06:42:26.103001+0000","last_clean":"2024-04-22T06:42:26.103001+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:42:26.103001+0000","last_undegraded":"2024-04-22T06:42:26.103001+0000","last_fullsized":"2024-04-22T06:42:26.103001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.475700+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:42:25.475700+0000","last_peered":"2024-04-22T06:42:25.475700+0000","last_clean":"2024-04-22T06:42:25.475700+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:42:25.475700+0000","last_undegraded":"2024-04-22T06:42:25.475700+0000","last_fullsized":"2024-04-22T06:42:25.475700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186479+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:42:25.186479+0000","last_peered":"2024-04-22T06:42:25.186479+0000","last_clean":"2024-04-22T06:42:25.186479+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:42:25.186479+0000","last_undegraded":"2024-04-22T06:42:25.186479+0000","last_fullsized":"2024-04-22T06:42:25.186479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103747+0000","last_change":"2024-04-22T06:42:24.201022+0000","last_active":"2024-04-22T06:42:26.103747+0000","last_peered":"2024-04-22T06:42:26.103747+0000","last_clean":"2024-04-22T06:42:26.103747+0000","last_became_active":"2024-04-22T06:42:24.200646+0000","last_became_peered":"2024-04-22T06:42:24.200646+0000","last_unstale":"2024-04-22T06:42:26.103747+0000","last_undegraded":"2024-04-22T06:42:26.103747+0000","last_fullsized":"2024-04-22T06:42:26.103747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:23.106777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103732+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:42:26.103732+0000","last_peered":"2024-04-22T06:42:26.103732+0000","last_clean":"2024-04-22T06:42:26.103732+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:42:26.103732+0000","last_undegraded":"2024-04-22T06:42:26.103732+0000","last_fullsized":"2024-04-22T06:42:26.103732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103273+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:42:26.103273+0000","last_peered":"2024-04-22T06:42:26.103273+0000","last_clean":"2024-04-22T06:42:26.103273+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:42:26.103273+0000","last_undegraded":"2024-04-22T06:42:26.103273+0000","last_fullsized":"2024-04-22T06:42:26.103273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183807+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:42:25.183807+0000","last_peered":"2024-04-22T06:42:25.183807+0000","last_clean":"2024-04-22T06:42:25.183807+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:42:25.183807+0000","last_undegraded":"2024-04-22T06:42:25.183807+0000","last_fullsized":"2024-04-22T06:42:25.183807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102513+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:42:26.102513+0000","last_peered":"2024-04-22T06:42:26.102513+0000","last_clean":"2024-04-22T06:42:26.102513+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:42:26.102513+0000","last_undegraded":"2024-04-22T06:42:26.102513+0000","last_fullsized":"2024-04-22T06:42:26.102513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102529+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:42:26.102529+0000","last_peered":"2024-04-22T06:42:26.102529+0000","last_clean":"2024-04-22T06:42:26.102529+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:42:26.102529+0000","last_undegraded":"2024-04-22T06:42:26.102529+0000","last_fullsized":"2024-04-22T06:42:26.102529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186035+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:42:25.186035+0000","last_peered":"2024-04-22T06:42:25.186035+0000","last_clean":"2024-04-22T06:42:25.186035+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:42:25.186035+0000","last_undegraded":"2024-04-22T06:42:25.186035+0000","last_fullsized":"2024-04-22T06:42:25.186035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103684+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:42:26.103684+0000","last_peered":"2024-04-22T06:42:26.103684+0000","last_clean":"2024-04-22T06:42:26.103684+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:42:26.103684+0000","last_undegraded":"2024-04-22T06:42:26.103684+0000","last_fullsized":"2024-04-22T06:42:26.103684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102269+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:42:26.102269+0000","last_peered":"2024-04-22T06:42:26.102269+0000","last_clean":"2024-04-22T06:42:26.102269+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:42:26.102269+0000","last_undegraded":"2024-04-22T06:42:26.102269+0000","last_fullsized":"2024-04-22T06:42:26.102269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185280+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:42:25.185280+0000","last_peered":"2024-04-22T06:42:25.185280+0000","last_clean":"2024-04-22T06:42:25.185280+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:42:25.185280+0000","last_undegraded":"2024-04-22T06:42:25.185280+0000","last_fullsized":"2024-04-22T06:42:25.185280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.713699+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:42:27.713699+0000","last_peered":"2024-04-22T06:42:27.713699+0000","last_clean":"2024-04-22T06:42:27.713699+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:42:27.713699+0000","last_undegraded":"2024-04-22T06:42:27.713699+0000","last_fullsized":"2024-04-22T06:42:27.713699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476137+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:42:25.476137+0000","last_peered":"2024-04-22T06:42:25.476137+0000","last_clean":"2024-04-22T06:42:25.476137+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:42:25.476137+0000","last_undegraded":"2024-04-22T06:42:25.476137+0000","last_fullsized":"2024-04-22T06:42:25.476137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476172+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:42:25.476172+0000","last_peered":"2024-04-22T06:42:25.476172+0000","last_clean":"2024-04-22T06:42:25.476172+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:42:25.476172+0000","last_undegraded":"2024-04-22T06:42:25.476172+0000","last_fullsized":"2024-04-22T06:42:25.476172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102793+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:42:26.102793+0000","last_peered":"2024-04-22T06:42:26.102793+0000","last_clean":"2024-04-22T06:42:26.102793+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:42:26.102793+0000","last_undegraded":"2024-04-22T06:42:26.102793+0000","last_fullsized":"2024-04-22T06:42:26.102793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.711405+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:42:27.711405+0000","last_peered":"2024-04-22T06:42:27.711405+0000","last_clean":"2024-04-22T06:42:27.711405+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:42:27.711405+0000","last_undegraded":"2024-04-22T06:42:27.711405+0000","last_fullsized":"2024-04-22T06:42:27.711405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186067+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:42:25.186067+0000","last_peered":"2024-04-22T06:42:25.186067+0000","last_clean":"2024-04-22T06:42:25.186067+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:42:25.186067+0000","last_undegraded":"2024-04-22T06:42:25.186067+0000","last_fullsized":"2024-04-22T06:42:25.186067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103755+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:42:26.103755+0000","last_peered":"2024-04-22T06:42:26.103755+0000","last_clean":"2024-04-22T06:42:26.103755+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:42:26.103755+0000","last_undegraded":"2024-04-22T06:42:26.103755+0000","last_fullsized":"2024-04-22T06:42:26.103755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103869+0000","last_change":"2024-04-22T06:42:24.196811+0000","last_active":"2024-04-22T06:42:26.103869+0000","last_peered":"2024-04-22T06:42:26.103869+0000","last_clean":"2024-04-22T06:42:26.103869+0000","last_became_active":"2024-04-22T06:42:24.196488+0000","last_became_peered":"2024-04-22T06:42:24.196488+0000","last_unstale":"2024-04-22T06:42:26.103869+0000","last_undegraded":"2024-04-22T06:42:26.103869+0000","last_fullsized":"2024-04-22T06:42:26.103869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:15.794562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.724773+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:42:29.724773+0000","last_peered":"2024-04-22T06:42:29.724773+0000","last_clean":"2024-04-22T06:42:29.724773+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:42:29.724773+0000","last_undegraded":"2024-04-22T06:42:29.724773+0000","last_fullsized":"2024-04-22T06:42:29.724773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102857+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:42:26.102857+0000","last_peered":"2024-04-22T06:42:26.102857+0000","last_clean":"2024-04-22T06:42:26.102857+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:42:26.102857+0000","last_undegraded":"2024-04-22T06:42:26.102857+0000","last_fullsized":"2024-04-22T06:42:26.102857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.711942+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:42:27.711942+0000","last_peered":"2024-04-22T06:42:27.711942+0000","last_clean":"2024-04-22T06:42:27.711942+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:42:27.711942+0000","last_undegraded":"2024-04-22T06:42:27.711942+0000","last_fullsized":"2024-04-22T06:42:27.711942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103199+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:42:26.103199+0000","last_peered":"2024-04-22T06:42:26.103199+0000","last_clean":"2024-04-22T06:42:26.103199+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:42:26.103199+0000","last_undegraded":"2024-04-22T06:42:26.103199+0000","last_fullsized":"2024-04-22T06:42:26.103199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183518+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:42:25.183518+0000","last_peered":"2024-04-22T06:42:25.183518+0000","last_clean":"2024-04-22T06:42:25.183518+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:42:25.183518+0000","last_undegraded":"2024-04-22T06:42:25.183518+0000","last_fullsized":"2024-04-22T06:42:25.183518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.475797+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:42:25.475797+0000","last_peered":"2024-04-22T06:42:25.475797+0000","last_clean":"2024-04-22T06:42:25.475797+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:42:25.475797+0000","last_undegraded":"2024-04-22T06:42:25.475797+0000","last_fullsized":"2024-04-22T06:42:25.475797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102534+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:42:26.102534+0000","last_peered":"2024-04-22T06:42:26.102534+0000","last_clean":"2024-04-22T06:42:26.102534+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:42:26.102534+0000","last_undegraded":"2024-04-22T06:42:26.102534+0000","last_fullsized":"2024-04-22T06:42:26.102534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102557+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:42:26.102557+0000","last_peered":"2024-04-22T06:42:26.102557+0000","last_clean":"2024-04-22T06:42:26.102557+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:42:26.102557+0000","last_undegraded":"2024-04-22T06:42:26.102557+0000","last_fullsized":"2024-04-22T06:42:26.102557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:28.721559+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:42:28.721559+0000","last_peered":"2024-04-22T06:42:28.721559+0000","last_clean":"2024-04-22T06:42:28.721559+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:42:28.721559+0000","last_undegraded":"2024-04-22T06:42:28.721559+0000","last_fullsized":"2024-04-22T06:42:28.721559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186283+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:42:25.186283+0000","last_peered":"2024-04-22T06:42:25.186283+0000","last_clean":"2024-04-22T06:42:25.186283+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:42:25.186283+0000","last_undegraded":"2024-04-22T06:42:25.186283+0000","last_fullsized":"2024-04-22T06:42:25.186283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104250+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:42:26.104250+0000","last_peered":"2024-04-22T06:42:26.104250+0000","last_clean":"2024-04-22T06:42:26.104250+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:42:26.104250+0000","last_undegraded":"2024-04-22T06:42:26.104250+0000","last_fullsized":"2024-04-22T06:42:26.104250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102586+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:42:26.102586+0000","last_peered":"2024-04-22T06:42:26.102586+0000","last_clean":"2024-04-22T06:42:26.102586+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:42:26.102586+0000","last_undegraded":"2024-04-22T06:42:26.102586+0000","last_fullsized":"2024-04-22T06:42:26.102586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705481+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:42:26.705481+0000","last_peered":"2024-04-22T06:42:26.705481+0000","last_clean":"2024-04-22T06:42:26.705481+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:42:26.705481+0000","last_undegraded":"2024-04-22T06:42:26.705481+0000","last_fullsized":"2024-04-22T06:42:26.705481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186337+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:42:25.186337+0000","last_peered":"2024-04-22T06:42:25.186337+0000","last_clean":"2024-04-22T06:42:25.186337+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:42:25.186337+0000","last_undegraded":"2024-04-22T06:42:25.186337+0000","last_fullsized":"2024-04-22T06:42:25.186337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'15","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.729040+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:42:29.729040+0000","last_peered":"2024-04-22T06:42:29.729040+0000","last_clean":"2024-04-22T06:42:29.729040+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:42:29.729040+0000","last_undegraded":"2024-04-22T06:42:29.729040+0000","last_fullsized":"2024-04-22T06:42:29.729040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":23,"num_write":16,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103704+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:42:26.103704+0000","last_peered":"2024-04-22T06:42:26.103704+0000","last_clean":"2024-04-22T06:42:26.103704+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:42:26.103704+0000","last_undegraded":"2024-04-22T06:42:26.103704+0000","last_fullsized":"2024-04-22T06:42:26.103704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185081+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:42:25.185081+0000","last_peered":"2024-04-22T06:42:25.185081+0000","last_clean":"2024-04-22T06:42:25.185081+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:42:25.185081+0000","last_undegraded":"2024-04-22T06:42:25.185081+0000","last_fullsized":"2024-04-22T06:42:25.185081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183848+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:42:25.183848+0000","last_peered":"2024-04-22T06:42:25.183848+0000","last_clean":"2024-04-22T06:42:25.183848+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:42:25.183848+0000","last_undegraded":"2024-04-22T06:42:25.183848+0000","last_fullsized":"2024-04-22T06:42:25.183848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705937+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:42:26.705937+0000","last_peered":"2024-04-22T06:42:26.705937+0000","last_clean":"2024-04-22T06:42:26.705937+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:42:26.705937+0000","last_undegraded":"2024-04-22T06:42:26.705937+0000","last_fullsized":"2024-04-22T06:42:26.705937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103006+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:42:26.103006+0000","last_peered":"2024-04-22T06:42:26.103006+0000","last_clean":"2024-04-22T06:42:26.103006+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:42:26.103006+0000","last_undegraded":"2024-04-22T06:42:26.103006+0000","last_fullsized":"2024-04-22T06:42:26.103006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104277+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:42:26.104277+0000","last_peered":"2024-04-22T06:42:26.104277+0000","last_clean":"2024-04-22T06:42:26.104277+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:42:26.104277+0000","last_undegraded":"2024-04-22T06:42:26.104277+0000","last_fullsized":"2024-04-22T06:42:26.104277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103268+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:42:26.103268+0000","last_peered":"2024-04-22T06:42:26.103268+0000","last_clean":"2024-04-22T06:42:26.103268+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:42:26.103268+0000","last_undegraded":"2024-04-22T06:42:26.103268+0000","last_fullsized":"2024-04-22T06:42:26.103268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.713508+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:42:27.713508+0000","last_peered":"2024-04-22T06:42:27.713508+0000","last_clean":"2024-04-22T06:42:27.713508+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:42:27.713508+0000","last_undegraded":"2024-04-22T06:42:27.713508+0000","last_fullsized":"2024-04-22T06:42:27.713508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103762+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:42:26.103762+0000","last_peered":"2024-04-22T06:42:26.103762+0000","last_clean":"2024-04-22T06:42:26.103762+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:42:26.103762+0000","last_undegraded":"2024-04-22T06:42:26.103762+0000","last_fullsized":"2024-04-22T06:42:26.103762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103886+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:42:26.103886+0000","last_peered":"2024-04-22T06:42:26.103886+0000","last_clean":"2024-04-22T06:42:26.103886+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:42:26.103886+0000","last_undegraded":"2024-04-22T06:42:26.103886+0000","last_fullsized":"2024-04-22T06:42:26.103886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102935+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:42:26.102935+0000","last_peered":"2024-04-22T06:42:26.102935+0000","last_clean":"2024-04-22T06:42:26.102935+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:42:26.102935+0000","last_undegraded":"2024-04-22T06:42:26.102935+0000","last_fullsized":"2024-04-22T06:42:26.102935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102337+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:42:26.102337+0000","last_peered":"2024-04-22T06:42:26.102337+0000","last_clean":"2024-04-22T06:42:26.102337+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:42:26.102337+0000","last_undegraded":"2024-04-22T06:42:26.102337+0000","last_fullsized":"2024-04-22T06:42:26.102337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104021+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:42:26.104021+0000","last_peered":"2024-04-22T06:42:26.104021+0000","last_clean":"2024-04-22T06:42:26.104021+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:42:26.104021+0000","last_undegraded":"2024-04-22T06:42:26.104021+0000","last_fullsized":"2024-04-22T06:42:26.104021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103215+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:42:26.103215+0000","last_peered":"2024-04-22T06:42:26.103215+0000","last_clean":"2024-04-22T06:42:26.103215+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:42:26.103215+0000","last_undegraded":"2024-04-22T06:42:26.103215+0000","last_fullsized":"2024-04-22T06:42:26.103215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103283+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:42:26.103283+0000","last_peered":"2024-04-22T06:42:26.103283+0000","last_clean":"2024-04-22T06:42:26.103283+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:42:26.103283+0000","last_undegraded":"2024-04-22T06:42:26.103283+0000","last_fullsized":"2024-04-22T06:42:26.103283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102667+0000","last_change":"2024-04-22T06:42:24.195082+0000","last_active":"2024-04-22T06:42:26.102667+0000","last_peered":"2024-04-22T06:42:26.102667+0000","last_clean":"2024-04-22T06:42:26.102667+0000","last_became_active":"2024-04-22T06:42:24.192569+0000","last_became_peered":"2024-04-22T06:42:24.192569+0000","last_unstale":"2024-04-22T06:42:26.102667+0000","last_undegraded":"2024-04-22T06:42:26.102667+0000","last_fullsized":"2024-04-22T06:42:26.102667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:07:44.979385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183427+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:42:25.183427+0000","last_peered":"2024-04-22T06:42:25.183427+0000","last_clean":"2024-04-22T06:42:25.183427+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:42:25.183427+0000","last_undegraded":"2024-04-22T06:42:25.183427+0000","last_fullsized":"2024-04-22T06:42:25.183427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476051+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:42:25.476051+0000","last_peered":"2024-04-22T06:42:25.476051+0000","last_clean":"2024-04-22T06:42:25.476051+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:42:25.476051+0000","last_undegraded":"2024-04-22T06:42:25.476051+0000","last_fullsized":"2024-04-22T06:42:25.476051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103368+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:42:26.103368+0000","last_peered":"2024-04-22T06:42:26.103368+0000","last_clean":"2024-04-22T06:42:26.103368+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:42:26.103368+0000","last_undegraded":"2024-04-22T06:42:26.103368+0000","last_fullsized":"2024-04-22T06:42:26.103368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102464+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:42:26.102464+0000","last_peered":"2024-04-22T06:42:26.102464+0000","last_clean":"2024-04-22T06:42:26.102464+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:42:26.102464+0000","last_undegraded":"2024-04-22T06:42:26.102464+0000","last_fullsized":"2024-04-22T06:42:26.102464+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183854+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:42:25.183854+0000","last_peered":"2024-04-22T06:42:25.183854+0000","last_clean":"2024-04-22T06:42:25.183854+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:42:25.183854+0000","last_undegraded":"2024-04-22T06:42:25.183854+0000","last_fullsized":"2024-04-22T06:42:25.183854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102715+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:42:26.102715+0000","last_peered":"2024-04-22T06:42:26.102715+0000","last_clean":"2024-04-22T06:42:26.102715+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:42:26.102715+0000","last_undegraded":"2024-04-22T06:42:26.102715+0000","last_fullsized":"2024-04-22T06:42:26.102715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103582+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:42:26.103582+0000","last_peered":"2024-04-22T06:42:26.103582+0000","last_clean":"2024-04-22T06:42:26.103582+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:42:26.103582+0000","last_undegraded":"2024-04-22T06:42:26.103582+0000","last_fullsized":"2024-04-22T06:42:26.103582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183851+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:42:25.183851+0000","last_peered":"2024-04-22T06:42:25.183851+0000","last_clean":"2024-04-22T06:42:25.183851+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:42:25.183851+0000","last_undegraded":"2024-04-22T06:42:25.183851+0000","last_fullsized":"2024-04-22T06:42:25.183851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.713193+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:42:27.713193+0000","last_peered":"2024-04-22T06:42:27.713193+0000","last_clean":"2024-04-22T06:42:27.713193+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:42:27.713193+0000","last_undegraded":"2024-04-22T06:42:27.713193+0000","last_fullsized":"2024-04-22T06:42:27.713193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476069+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:42:25.476069+0000","last_peered":"2024-04-22T06:42:25.476069+0000","last_clean":"2024-04-22T06:42:25.476069+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:42:25.476069+0000","last_undegraded":"2024-04-22T06:42:25.476069+0000","last_fullsized":"2024-04-22T06:42:25.476069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104177+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:42:26.104177+0000","last_peered":"2024-04-22T06:42:26.104177+0000","last_clean":"2024-04-22T06:42:26.104177+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:42:26.104177+0000","last_undegraded":"2024-04-22T06:42:26.104177+0000","last_fullsized":"2024-04-22T06:42:26.104177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102811+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:42:26.102811+0000","last_peered":"2024-04-22T06:42:26.102811+0000","last_clean":"2024-04-22T06:42:26.102811+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:42:26.102811+0000","last_undegraded":"2024-04-22T06:42:26.102811+0000","last_fullsized":"2024-04-22T06:42:26.102811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705611+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:42:26.705611+0000","last_peered":"2024-04-22T06:42:26.705611+0000","last_clean":"2024-04-22T06:42:26.705611+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:42:26.705611+0000","last_undegraded":"2024-04-22T06:42:26.705611+0000","last_fullsized":"2024-04-22T06:42:26.705611+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102986+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:42:26.102986+0000","last_peered":"2024-04-22T06:42:26.102986+0000","last_clean":"2024-04-22T06:42:26.102986+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:42:26.102986+0000","last_undegraded":"2024-04-22T06:42:26.102986+0000","last_fullsized":"2024-04-22T06:42:26.102986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705472+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:42:26.705472+0000","last_peered":"2024-04-22T06:42:26.705472+0000","last_clean":"2024-04-22T06:42:26.705472+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:42:26.705472+0000","last_undegraded":"2024-04-22T06:42:26.705472+0000","last_fullsized":"2024-04-22T06:42:26.705472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102864+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:42:26.102864+0000","last_peered":"2024-04-22T06:42:26.102864+0000","last_clean":"2024-04-22T06:42:26.102864+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:42:26.102864+0000","last_undegraded":"2024-04-22T06:42:26.102864+0000","last_fullsized":"2024-04-22T06:42:26.102864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.184868+0000","last_change":"2024-04-22T06:42:23.681287+0000","last_active":"2024-04-22T06:42:25.184868+0000","last_peered":"2024-04-22T06:42:25.184868+0000","last_clean":"2024-04-22T06:42:25.184868+0000","last_became_active":"2024-04-22T06:42:23.680939+0000","last_became_peered":"2024-04-22T06:42:23.680939+0000","last_unstale":"2024-04-22T06:42:25.184868+0000","last_undegraded":"2024-04-22T06:42:25.184868+0000","last_fullsized":"2024-04-22T06:42:25.184868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:56.903734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476068+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:42:25.476068+0000","last_peered":"2024-04-22T06:42:25.476068+0000","last_clean":"2024-04-22T06:42:25.476068+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:42:25.476068+0000","last_undegraded":"2024-04-22T06:42:25.476068+0000","last_fullsized":"2024-04-22T06:42:25.476068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705624+0000","last_change":"2024-04-22T06:42:23.673602+0000","last_active":"2024-04-22T06:42:26.705624+0000","last_peered":"2024-04-22T06:42:26.705624+0000","last_clean":"2024-04-22T06:42:26.705624+0000","last_became_active":"2024-04-22T06:42:23.673281+0000","last_became_peered":"2024-04-22T06:42:23.673281+0000","last_unstale":"2024-04-22T06:42:26.705624+0000","last_undegraded":"2024-04-22T06:42:26.705624+0000","last_fullsized":"2024-04-22T06:42:26.705624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:09.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103436+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:42:26.103436+0000","last_peered":"2024-04-22T06:42:26.103436+0000","last_clean":"2024-04-22T06:42:26.103436+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:42:26.103436+0000","last_undegraded":"2024-04-22T06:42:26.103436+0000","last_fullsized":"2024-04-22T06:42:26.103436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185159+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:42:25.185159+0000","last_peered":"2024-04-22T06:42:25.185159+0000","last_clean":"2024-04-22T06:42:25.185159+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:42:25.185159+0000","last_undegraded":"2024-04-22T06:42:25.185159+0000","last_fullsized":"2024-04-22T06:42:25.185159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103816+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:42:26.103816+0000","last_peered":"2024-04-22T06:42:26.103816+0000","last_clean":"2024-04-22T06:42:26.103816+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:42:26.103816+0000","last_undegraded":"2024-04-22T06:42:26.103816+0000","last_fullsized":"2024-04-22T06:42:26.103816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103499+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:42:26.103499+0000","last_peered":"2024-04-22T06:42:26.103499+0000","last_clean":"2024-04-22T06:42:26.103499+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:42:26.103499+0000","last_undegraded":"2024-04-22T06:42:26.103499+0000","last_fullsized":"2024-04-22T06:42:26.103499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103995+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:42:26.103995+0000","last_peered":"2024-04-22T06:42:26.103995+0000","last_clean":"2024-04-22T06:42:26.103995+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:42:26.103995+0000","last_undegraded":"2024-04-22T06:42:26.103995+0000","last_fullsized":"2024-04-22T06:42:26.103995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183764+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:42:25.183764+0000","last_peered":"2024-04-22T06:42:25.183764+0000","last_clean":"2024-04-22T06:42:25.183764+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:42:25.183764+0000","last_undegraded":"2024-04-22T06:42:25.183764+0000","last_fullsized":"2024-04-22T06:42:25.183764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103826+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:42:26.103826+0000","last_peered":"2024-04-22T06:42:26.103826+0000","last_clean":"2024-04-22T06:42:26.103826+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:42:26.103826+0000","last_undegraded":"2024-04-22T06:42:26.103826+0000","last_fullsized":"2024-04-22T06:42:26.103826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103044+0000","last_change":"2024-04-22T06:42:24.199973+0000","last_active":"2024-04-22T06:42:26.103044+0000","last_peered":"2024-04-22T06:42:26.103044+0000","last_clean":"2024-04-22T06:42:26.103044+0000","last_became_active":"2024-04-22T06:42:24.199562+0000","last_became_peered":"2024-04-22T06:42:24.199562+0000","last_unstale":"2024-04-22T06:42:26.103044+0000","last_undegraded":"2024-04-22T06:42:26.103044+0000","last_fullsized":"2024-04-22T06:42:26.103044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:34.232346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.184986+0000","last_change":"2024-04-22T06:42:24.194984+0000","last_active":"2024-04-22T06:42:25.184986+0000","last_peered":"2024-04-22T06:42:25.184986+0000","last_clean":"2024-04-22T06:42:25.184986+0000","last_became_active":"2024-04-22T06:42:24.194674+0000","last_became_peered":"2024-04-22T06:42:24.194674+0000","last_unstale":"2024-04-22T06:42:25.184986+0000","last_undegraded":"2024-04-22T06:42:25.184986+0000","last_fullsized":"2024-04-22T06:42:25.184986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:54:52.355697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.184858+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:42:25.184858+0000","last_peered":"2024-04-22T06:42:25.184858+0000","last_clean":"2024-04-22T06:42:25.184858+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:42:25.184858+0000","last_undegraded":"2024-04-22T06:42:25.184858+0000","last_fullsized":"2024-04-22T06:42:25.184858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102269+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:42:26.102269+0000","last_peered":"2024-04-22T06:42:26.102269+0000","last_clean":"2024-04-22T06:42:26.102269+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:42:26.102269+0000","last_undegraded":"2024-04-22T06:42:26.102269+0000","last_fullsized":"2024-04-22T06:42:26.102269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183679+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:42:25.183679+0000","last_peered":"2024-04-22T06:42:25.183679+0000","last_clean":"2024-04-22T06:42:25.183679+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:42:25.183679+0000","last_undegraded":"2024-04-22T06:42:25.183679+0000","last_fullsized":"2024-04-22T06:42:25.183679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103413+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:42:26.103413+0000","last_peered":"2024-04-22T06:42:26.103413+0000","last_clean":"2024-04-22T06:42:26.103413+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:42:26.103413+0000","last_undegraded":"2024-04-22T06:42:26.103413+0000","last_fullsized":"2024-04-22T06:42:26.103413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185366+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:42:25.185366+0000","last_peered":"2024-04-22T06:42:25.185366+0000","last_clean":"2024-04-22T06:42:25.185366+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:42:25.185366+0000","last_undegraded":"2024-04-22T06:42:25.185366+0000","last_fullsized":"2024-04-22T06:42:25.185366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476055+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:42:25.476055+0000","last_peered":"2024-04-22T06:42:25.476055+0000","last_clean":"2024-04-22T06:42:25.476055+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:42:25.476055+0000","last_undegraded":"2024-04-22T06:42:25.476055+0000","last_fullsized":"2024-04-22T06:42:25.476055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102628+0000","last_change":"2024-04-22T06:42:24.183528+0000","last_active":"2024-04-22T06:42:26.102628+0000","last_peered":"2024-04-22T06:42:26.102628+0000","last_clean":"2024-04-22T06:42:26.102628+0000","last_became_active":"2024-04-22T06:42:24.183002+0000","last_became_peered":"2024-04-22T06:42:24.183002+0000","last_unstale":"2024-04-22T06:42:26.102628+0000","last_undegraded":"2024-04-22T06:42:26.102628+0000","last_fullsized":"2024-04-22T06:42:26.102628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:47.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.475834+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:42:25.475834+0000","last_peered":"2024-04-22T06:42:25.475834+0000","last_clean":"2024-04-22T06:42:25.475834+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:42:25.475834+0000","last_undegraded":"2024-04-22T06:42:25.475834+0000","last_fullsized":"2024-04-22T06:42:25.475834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186418+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:42:25.186418+0000","last_peered":"2024-04-22T06:42:25.186418+0000","last_clean":"2024-04-22T06:42:25.186418+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:42:25.186418+0000","last_undegraded":"2024-04-22T06:42:25.186418+0000","last_fullsized":"2024-04-22T06:42:25.186418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183852+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:42:25.183852+0000","last_peered":"2024-04-22T06:42:25.183852+0000","last_clean":"2024-04-22T06:42:25.183852+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:42:25.183852+0000","last_undegraded":"2024-04-22T06:42:25.183852+0000","last_fullsized":"2024-04-22T06:42:25.183852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102720+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:42:26.102720+0000","last_peered":"2024-04-22T06:42:26.102720+0000","last_clean":"2024-04-22T06:42:26.102720+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:42:26.102720+0000","last_undegraded":"2024-04-22T06:42:26.102720+0000","last_fullsized":"2024-04-22T06:42:26.102720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":23602,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":17,"num_read_kb":31,"num_write":102,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":31911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1051870,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":1029471,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":438922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":1030756,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":22864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:42:31.416 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:42:31.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- 172.21.15.79:0/1263461918 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608100e90 msgr2=0x7f96081012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- 172.21.15.79:0/1263461918 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608100e90 0x7f96081012b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009900 tx=0x7f95f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- 172.21.15.79:0/1263461918 shutdown_connections 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- 172.21.15.79:0/1263461918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96081029a0 0x7f96081092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- 172.21.15.79:0/1263461918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9608101fe0 0x7f9608102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- 172.21.15.79:0/1263461918 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608100e90 0x7f96081012b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- 172.21.15.79:0/1263461918 >> 172.21.15.79:0/1263461918 conn(0x7f96080fc420 msgr2=0x7f96080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- 172.21.15.79:0/1263461918 shutdown_connections 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- 172.21.15.79:0/1263461918 wait complete. 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 Processor -- start 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- start start 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9608100e90 0x7f960806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 0x7f960819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96081029a0 0x7f960819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f960810c9c0 con 0x7f9608101fe0 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f960810c840 con 0x7f9608100e90 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960ff15640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f960810cb40 con 0x7f96081029a0 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.551+0000 7f960d4b0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 0x7f960819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 0x7f960819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51844/0 (socket says 172.21.15.79:51844) 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 -- 172.21.15.79:0/2309600268 learned_addr learned my addr 172.21.15.79:0/2309600268 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 -- 172.21.15.79:0/2309600268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96081029a0 msgr2=0x7f960819f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96081029a0 0x7f960819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960dcb1640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9608100e90 0x7f960806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 -- 172.21.15.79:0/2309600268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9608100e90 msgr2=0x7f960806b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9608100e90 0x7f960806b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 -- 172.21.15.79:0/2309600268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960819fdc0 con 0x7f9608101fe0 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960dcb1640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9608100e90 0x7f960806b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:31.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960d4b0640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 0x7f960819ce90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f960400d950 tx=0x7f960400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604017020 con 0x7f9608101fe0 2024-04-22T06:42:31.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96081a00b0 con 0x7f9608101fe0 2024-04-22T06:42:31.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9604009c30 con 0x7f9608101fe0 2024-04-22T06:42:31.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960400f6b0 con 0x7f9608101fe0 2024-04-22T06:42:31.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.552+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96081a0600 con 0x7f9608101fe0 2024-04-22T06:42:31.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.553+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9608108220 con 0x7f9608101fe0 2024-04-22T06:42:31.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.554+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f960401a020 con 0x7f9608101fe0 2024-04-22T06:42:31.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.555+0000 7f95feffd640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f95f40a3c40 0x7f95f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.555+0000 7f960dcb1640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f95f40a3c40 0x7f95f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.555+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f96040d0160 con 0x7f9608101fe0 2024-04-22T06:42:31.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.555+0000 7f960dcb1640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f95f40a3c40 0x7f95f40a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f95f80096c0 tx=0x7f95f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.559+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9604009de0 con 0x7f9608101fe0 2024-04-22T06:42:31.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.739+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9608062390 con 0x7f95f40a3c40 2024-04-22T06:42:31.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.746+0000 7f95feffd640 1 -- 172.21.15.79:0/2309600268 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258939 (secure 0 0 0) 0x7f9608062390 con 0x7f95f40a3c40 2024-04-22T06:42:31.747 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:31.748 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:42:31.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f95f40a3c40 msgr2=0x7f95f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f95f40a3c40 0x7f95f40a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f95f80096c0 tx=0x7f95f8000bc0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 msgr2=0x7f960819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 0x7f960819ce90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f960400d950 tx=0x7f960400de10 comp rx=0 tx=0).stop 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 shutdown_connections 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f95f40a3c40 0x7f95f40a6100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96081029a0 0x7f960819f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9608101fe0 0x7f960819ce90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 --2- 172.21.15.79:0/2309600268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9608100e90 0x7f960806b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 >> 172.21.15.79:0/2309600268 conn(0x7f96080fc420 msgr2=0x7f96081053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 shutdown_connections 2024-04-22T06:42:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:31.749+0000 7f960ff15640 1 -- 172.21.15.79:0/2309600268 wait complete. 2024-04-22T06:42:31.759 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-22T06:42:30.645256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614087,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":153,"num_read_kb":150,"num_write":284,"num_write_kb":2989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":243,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376912,"kb_used_data":11056,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564912,"statfs":{"total":1151940427776,"available":1151554469888,"internally_reserved":0,"allocated":11321344,"data_stored":7065389,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23719,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":17,"num_read_kb":31,"num_write":103,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.990426"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103478+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:42:26.103478+0000","last_peered":"2024-04-22T06:42:26.103478+0000","last_clean":"2024-04-22T06:42:26.103478+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:42:26.103478+0000","last_undegraded":"2024-04-22T06:42:26.103478+0000","last_fullsized":"2024-04-22T06:42:26.103478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102783+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:42:26.102783+0000","last_peered":"2024-04-22T06:42:26.102783+0000","last_clean":"2024-04-22T06:42:26.102783+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:42:26.102783+0000","last_undegraded":"2024-04-22T06:42:26.102783+0000","last_fullsized":"2024-04-22T06:42:26.102783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102587+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:42:26.102587+0000","last_peered":"2024-04-22T06:42:26.102587+0000","last_clean":"2024-04-22T06:42:26.102587+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:42:26.102587+0000","last_undegraded":"2024-04-22T06:42:26.102587+0000","last_fullsized":"2024-04-22T06:42:26.102587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104210+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:42:26.104210+0000","last_peered":"2024-04-22T06:42:26.104210+0000","last_clean":"2024-04-22T06:42:26.104210+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:42:26.104210+0000","last_undegraded":"2024-04-22T06:42:26.104210+0000","last_fullsized":"2024-04-22T06:42:26.104210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103189+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:42:26.103189+0000","last_peered":"2024-04-22T06:42:26.103189+0000","last_clean":"2024-04-22T06:42:26.103189+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:42:26.103189+0000","last_undegraded":"2024-04-22T06:42:26.103189+0000","last_fullsized":"2024-04-22T06:42:26.103189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102710+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:42:26.102710+0000","last_peered":"2024-04-22T06:42:26.102710+0000","last_clean":"2024-04-22T06:42:26.102710+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:42:26.102710+0000","last_undegraded":"2024-04-22T06:42:26.102710+0000","last_fullsized":"2024-04-22T06:42:26.102710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103001+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:42:26.103001+0000","last_peered":"2024-04-22T06:42:26.103001+0000","last_clean":"2024-04-22T06:42:26.103001+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:42:26.103001+0000","last_undegraded":"2024-04-22T06:42:26.103001+0000","last_fullsized":"2024-04-22T06:42:26.103001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.475700+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:42:25.475700+0000","last_peered":"2024-04-22T06:42:25.475700+0000","last_clean":"2024-04-22T06:42:25.475700+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:42:25.475700+0000","last_undegraded":"2024-04-22T06:42:25.475700+0000","last_fullsized":"2024-04-22T06:42:25.475700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186479+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:42:25.186479+0000","last_peered":"2024-04-22T06:42:25.186479+0000","last_clean":"2024-04-22T06:42:25.186479+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:42:25.186479+0000","last_undegraded":"2024-04-22T06:42:25.186479+0000","last_fullsized":"2024-04-22T06:42:25.186479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103747+0000","last_change":"2024-04-22T06:42:24.201022+0000","last_active":"2024-04-22T06:42:26.103747+0000","last_peered":"2024-04-22T06:42:26.103747+0000","last_clean":"2024-04-22T06:42:26.103747+0000","last_became_active":"2024-04-22T06:42:24.200646+0000","last_became_peered":"2024-04-22T06:42:24.200646+0000","last_unstale":"2024-04-22T06:42:26.103747+0000","last_undegraded":"2024-04-22T06:42:26.103747+0000","last_fullsized":"2024-04-22T06:42:26.103747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:23.106777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103732+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:42:26.103732+0000","last_peered":"2024-04-22T06:42:26.103732+0000","last_clean":"2024-04-22T06:42:26.103732+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:42:26.103732+0000","last_undegraded":"2024-04-22T06:42:26.103732+0000","last_fullsized":"2024-04-22T06:42:26.103732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103273+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:42:26.103273+0000","last_peered":"2024-04-22T06:42:26.103273+0000","last_clean":"2024-04-22T06:42:26.103273+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:42:26.103273+0000","last_undegraded":"2024-04-22T06:42:26.103273+0000","last_fullsized":"2024-04-22T06:42:26.103273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183807+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:42:25.183807+0000","last_peered":"2024-04-22T06:42:25.183807+0000","last_clean":"2024-04-22T06:42:25.183807+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:42:25.183807+0000","last_undegraded":"2024-04-22T06:42:25.183807+0000","last_fullsized":"2024-04-22T06:42:25.183807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102513+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:42:26.102513+0000","last_peered":"2024-04-22T06:42:26.102513+0000","last_clean":"2024-04-22T06:42:26.102513+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:42:26.102513+0000","last_undegraded":"2024-04-22T06:42:26.102513+0000","last_fullsized":"2024-04-22T06:42:26.102513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102529+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:42:26.102529+0000","last_peered":"2024-04-22T06:42:26.102529+0000","last_clean":"2024-04-22T06:42:26.102529+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:42:26.102529+0000","last_undegraded":"2024-04-22T06:42:26.102529+0000","last_fullsized":"2024-04-22T06:42:26.102529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186035+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:42:25.186035+0000","last_peered":"2024-04-22T06:42:25.186035+0000","last_clean":"2024-04-22T06:42:25.186035+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:42:25.186035+0000","last_undegraded":"2024-04-22T06:42:25.186035+0000","last_fullsized":"2024-04-22T06:42:25.186035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103684+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:42:26.103684+0000","last_peered":"2024-04-22T06:42:26.103684+0000","last_clean":"2024-04-22T06:42:26.103684+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:42:26.103684+0000","last_undegraded":"2024-04-22T06:42:26.103684+0000","last_fullsized":"2024-04-22T06:42:26.103684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102269+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:42:26.102269+0000","last_peered":"2024-04-22T06:42:26.102269+0000","last_clean":"2024-04-22T06:42:26.102269+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:42:26.102269+0000","last_undegraded":"2024-04-22T06:42:26.102269+0000","last_fullsized":"2024-04-22T06:42:26.102269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185280+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:42:25.185280+0000","last_peered":"2024-04-22T06:42:25.185280+0000","last_clean":"2024-04-22T06:42:25.185280+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:42:25.185280+0000","last_undegraded":"2024-04-22T06:42:25.185280+0000","last_fullsized":"2024-04-22T06:42:25.185280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.713699+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:42:27.713699+0000","last_peered":"2024-04-22T06:42:27.713699+0000","last_clean":"2024-04-22T06:42:27.713699+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:42:27.713699+0000","last_undegraded":"2024-04-22T06:42:27.713699+0000","last_fullsized":"2024-04-22T06:42:27.713699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476137+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:42:25.476137+0000","last_peered":"2024-04-22T06:42:25.476137+0000","last_clean":"2024-04-22T06:42:25.476137+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:42:25.476137+0000","last_undegraded":"2024-04-22T06:42:25.476137+0000","last_fullsized":"2024-04-22T06:42:25.476137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476172+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:42:25.476172+0000","last_peered":"2024-04-22T06:42:25.476172+0000","last_clean":"2024-04-22T06:42:25.476172+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:42:25.476172+0000","last_undegraded":"2024-04-22T06:42:25.476172+0000","last_fullsized":"2024-04-22T06:42:25.476172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102793+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:42:26.102793+0000","last_peered":"2024-04-22T06:42:26.102793+0000","last_clean":"2024-04-22T06:42:26.102793+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:42:26.102793+0000","last_undegraded":"2024-04-22T06:42:26.102793+0000","last_fullsized":"2024-04-22T06:42:26.102793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.711405+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:42:27.711405+0000","last_peered":"2024-04-22T06:42:27.711405+0000","last_clean":"2024-04-22T06:42:27.711405+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:42:27.711405+0000","last_undegraded":"2024-04-22T06:42:27.711405+0000","last_fullsized":"2024-04-22T06:42:27.711405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186067+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:42:25.186067+0000","last_peered":"2024-04-22T06:42:25.186067+0000","last_clean":"2024-04-22T06:42:25.186067+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:42:25.186067+0000","last_undegraded":"2024-04-22T06:42:25.186067+0000","last_fullsized":"2024-04-22T06:42:25.186067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103755+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:42:26.103755+0000","last_peered":"2024-04-22T06:42:26.103755+0000","last_clean":"2024-04-22T06:42:26.103755+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:42:26.103755+0000","last_undegraded":"2024-04-22T06:42:26.103755+0000","last_fullsized":"2024-04-22T06:42:26.103755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103869+0000","last_change":"2024-04-22T06:42:24.196811+0000","last_active":"2024-04-22T06:42:26.103869+0000","last_peered":"2024-04-22T06:42:26.103869+0000","last_clean":"2024-04-22T06:42:26.103869+0000","last_became_active":"2024-04-22T06:42:24.196488+0000","last_became_peered":"2024-04-22T06:42:24.196488+0000","last_unstale":"2024-04-22T06:42:26.103869+0000","last_undegraded":"2024-04-22T06:42:26.103869+0000","last_fullsized":"2024-04-22T06:42:26.103869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:17:15.794562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.724773+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:42:29.724773+0000","last_peered":"2024-04-22T06:42:29.724773+0000","last_clean":"2024-04-22T06:42:29.724773+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:42:29.724773+0000","last_undegraded":"2024-04-22T06:42:29.724773+0000","last_fullsized":"2024-04-22T06:42:29.724773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102857+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:42:26.102857+0000","last_peered":"2024-04-22T06:42:26.102857+0000","last_clean":"2024-04-22T06:42:26.102857+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:42:26.102857+0000","last_undegraded":"2024-04-22T06:42:26.102857+0000","last_fullsized":"2024-04-22T06:42:26.102857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.711942+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:42:27.711942+0000","last_peered":"2024-04-22T06:42:27.711942+0000","last_clean":"2024-04-22T06:42:27.711942+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:42:27.711942+0000","last_undegraded":"2024-04-22T06:42:27.711942+0000","last_fullsized":"2024-04-22T06:42:27.711942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103199+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:42:26.103199+0000","last_peered":"2024-04-22T06:42:26.103199+0000","last_clean":"2024-04-22T06:42:26.103199+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:42:26.103199+0000","last_undegraded":"2024-04-22T06:42:26.103199+0000","last_fullsized":"2024-04-22T06:42:26.103199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183518+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:42:25.183518+0000","last_peered":"2024-04-22T06:42:25.183518+0000","last_clean":"2024-04-22T06:42:25.183518+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:42:25.183518+0000","last_undegraded":"2024-04-22T06:42:25.183518+0000","last_fullsized":"2024-04-22T06:42:25.183518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.475797+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:42:25.475797+0000","last_peered":"2024-04-22T06:42:25.475797+0000","last_clean":"2024-04-22T06:42:25.475797+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:42:25.475797+0000","last_undegraded":"2024-04-22T06:42:25.475797+0000","last_fullsized":"2024-04-22T06:42:25.475797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102534+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:42:26.102534+0000","last_peered":"2024-04-22T06:42:26.102534+0000","last_clean":"2024-04-22T06:42:26.102534+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:42:26.102534+0000","last_undegraded":"2024-04-22T06:42:26.102534+0000","last_fullsized":"2024-04-22T06:42:26.102534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102557+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:42:26.102557+0000","last_peered":"2024-04-22T06:42:26.102557+0000","last_clean":"2024-04-22T06:42:26.102557+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:42:26.102557+0000","last_undegraded":"2024-04-22T06:42:26.102557+0000","last_fullsized":"2024-04-22T06:42:26.102557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:28.721559+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:42:28.721559+0000","last_peered":"2024-04-22T06:42:28.721559+0000","last_clean":"2024-04-22T06:42:28.721559+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:42:28.721559+0000","last_undegraded":"2024-04-22T06:42:28.721559+0000","last_fullsized":"2024-04-22T06:42:28.721559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186283+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:42:25.186283+0000","last_peered":"2024-04-22T06:42:25.186283+0000","last_clean":"2024-04-22T06:42:25.186283+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:42:25.186283+0000","last_undegraded":"2024-04-22T06:42:25.186283+0000","last_fullsized":"2024-04-22T06:42:25.186283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104250+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:42:26.104250+0000","last_peered":"2024-04-22T06:42:26.104250+0000","last_clean":"2024-04-22T06:42:26.104250+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:42:26.104250+0000","last_undegraded":"2024-04-22T06:42:26.104250+0000","last_fullsized":"2024-04-22T06:42:26.104250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102586+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:42:26.102586+0000","last_peered":"2024-04-22T06:42:26.102586+0000","last_clean":"2024-04-22T06:42:26.102586+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:42:26.102586+0000","last_undegraded":"2024-04-22T06:42:26.102586+0000","last_fullsized":"2024-04-22T06:42:26.102586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705481+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:42:26.705481+0000","last_peered":"2024-04-22T06:42:26.705481+0000","last_clean":"2024-04-22T06:42:26.705481+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:42:26.705481+0000","last_undegraded":"2024-04-22T06:42:26.705481+0000","last_fullsized":"2024-04-22T06:42:26.705481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186337+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:42:25.186337+0000","last_peered":"2024-04-22T06:42:25.186337+0000","last_clean":"2024-04-22T06:42:25.186337+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:42:25.186337+0000","last_undegraded":"2024-04-22T06:42:25.186337+0000","last_fullsized":"2024-04-22T06:42:25.186337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'15","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.729040+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:42:29.729040+0000","last_peered":"2024-04-22T06:42:29.729040+0000","last_clean":"2024-04-22T06:42:29.729040+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:42:29.729040+0000","last_undegraded":"2024-04-22T06:42:29.729040+0000","last_fullsized":"2024-04-22T06:42:29.729040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":23,"num_write":16,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103704+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:42:26.103704+0000","last_peered":"2024-04-22T06:42:26.103704+0000","last_clean":"2024-04-22T06:42:26.103704+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:42:26.103704+0000","last_undegraded":"2024-04-22T06:42:26.103704+0000","last_fullsized":"2024-04-22T06:42:26.103704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185081+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:42:25.185081+0000","last_peered":"2024-04-22T06:42:25.185081+0000","last_clean":"2024-04-22T06:42:25.185081+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:42:25.185081+0000","last_undegraded":"2024-04-22T06:42:25.185081+0000","last_fullsized":"2024-04-22T06:42:25.185081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183848+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:42:25.183848+0000","last_peered":"2024-04-22T06:42:25.183848+0000","last_clean":"2024-04-22T06:42:25.183848+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:42:25.183848+0000","last_undegraded":"2024-04-22T06:42:25.183848+0000","last_fullsized":"2024-04-22T06:42:25.183848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705937+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:42:26.705937+0000","last_peered":"2024-04-22T06:42:26.705937+0000","last_clean":"2024-04-22T06:42:26.705937+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:42:26.705937+0000","last_undegraded":"2024-04-22T06:42:26.705937+0000","last_fullsized":"2024-04-22T06:42:26.705937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103006+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:42:26.103006+0000","last_peered":"2024-04-22T06:42:26.103006+0000","last_clean":"2024-04-22T06:42:26.103006+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:42:26.103006+0000","last_undegraded":"2024-04-22T06:42:26.103006+0000","last_fullsized":"2024-04-22T06:42:26.103006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104277+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:42:26.104277+0000","last_peered":"2024-04-22T06:42:26.104277+0000","last_clean":"2024-04-22T06:42:26.104277+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:42:26.104277+0000","last_undegraded":"2024-04-22T06:42:26.104277+0000","last_fullsized":"2024-04-22T06:42:26.104277+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103268+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:42:26.103268+0000","last_peered":"2024-04-22T06:42:26.103268+0000","last_clean":"2024-04-22T06:42:26.103268+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:42:26.103268+0000","last_undegraded":"2024-04-22T06:42:26.103268+0000","last_fullsized":"2024-04-22T06:42:26.103268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.713508+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:42:27.713508+0000","last_peered":"2024-04-22T06:42:27.713508+0000","last_clean":"2024-04-22T06:42:27.713508+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:42:27.713508+0000","last_undegraded":"2024-04-22T06:42:27.713508+0000","last_fullsized":"2024-04-22T06:42:27.713508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103762+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:42:26.103762+0000","last_peered":"2024-04-22T06:42:26.103762+0000","last_clean":"2024-04-22T06:42:26.103762+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:42:26.103762+0000","last_undegraded":"2024-04-22T06:42:26.103762+0000","last_fullsized":"2024-04-22T06:42:26.103762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103886+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:42:26.103886+0000","last_peered":"2024-04-22T06:42:26.103886+0000","last_clean":"2024-04-22T06:42:26.103886+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:42:26.103886+0000","last_undegraded":"2024-04-22T06:42:26.103886+0000","last_fullsized":"2024-04-22T06:42:26.103886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102935+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:42:26.102935+0000","last_peered":"2024-04-22T06:42:26.102935+0000","last_clean":"2024-04-22T06:42:26.102935+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:42:26.102935+0000","last_undegraded":"2024-04-22T06:42:26.102935+0000","last_fullsized":"2024-04-22T06:42:26.102935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102337+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:42:26.102337+0000","last_peered":"2024-04-22T06:42:26.102337+0000","last_clean":"2024-04-22T06:42:26.102337+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:42:26.102337+0000","last_undegraded":"2024-04-22T06:42:26.102337+0000","last_fullsized":"2024-04-22T06:42:26.102337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104021+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:42:26.104021+0000","last_peered":"2024-04-22T06:42:26.104021+0000","last_clean":"2024-04-22T06:42:26.104021+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:42:26.104021+0000","last_undegraded":"2024-04-22T06:42:26.104021+0000","last_fullsized":"2024-04-22T06:42:26.104021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103215+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:42:26.103215+0000","last_peered":"2024-04-22T06:42:26.103215+0000","last_clean":"2024-04-22T06:42:26.103215+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:42:26.103215+0000","last_undegraded":"2024-04-22T06:42:26.103215+0000","last_fullsized":"2024-04-22T06:42:26.103215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103283+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:42:26.103283+0000","last_peered":"2024-04-22T06:42:26.103283+0000","last_clean":"2024-04-22T06:42:26.103283+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:42:26.103283+0000","last_undegraded":"2024-04-22T06:42:26.103283+0000","last_fullsized":"2024-04-22T06:42:26.103283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102667+0000","last_change":"2024-04-22T06:42:24.195082+0000","last_active":"2024-04-22T06:42:26.102667+0000","last_peered":"2024-04-22T06:42:26.102667+0000","last_clean":"2024-04-22T06:42:26.102667+0000","last_became_active":"2024-04-22T06:42:24.192569+0000","last_became_peered":"2024-04-22T06:42:24.192569+0000","last_unstale":"2024-04-22T06:42:26.102667+0000","last_undegraded":"2024-04-22T06:42:26.102667+0000","last_fullsized":"2024-04-22T06:42:26.102667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:07:44.979385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183427+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:42:25.183427+0000","last_peered":"2024-04-22T06:42:25.183427+0000","last_clean":"2024-04-22T06:42:25.183427+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:42:25.183427+0000","last_undegraded":"2024-04-22T06:42:25.183427+0000","last_fullsized":"2024-04-22T06:42:25.183427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476051+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:42:25.476051+0000","last_peered":"2024-04-22T06:42:25.476051+0000","last_clean":"2024-04-22T06:42:25.476051+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:42:25.476051+0000","last_undegraded":"2024-04-22T06:42:25.476051+0000","last_fullsized":"2024-04-22T06:42:25.476051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103368+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:42:26.103368+0000","last_peered":"2024-04-22T06:42:26.103368+0000","last_clean":"2024-04-22T06:42:26.103368+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:42:26.103368+0000","last_undegraded":"2024-04-22T06:42:26.103368+0000","last_fullsized":"2024-04-22T06:42:26.103368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102464+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:42:26.102464+0000","last_peered":"2024-04-22T06:42:26.102464+0000","last_clean":"2024-04-22T06:42:26.102464+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:42:26.102464+0000","last_undegraded":"2024-04-22T06:42:26.102464+0000","last_fullsized":"2024-04-22T06:42:26.102464+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183854+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:42:25.183854+0000","last_peered":"2024-04-22T06:42:25.183854+0000","last_clean":"2024-04-22T06:42:25.183854+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:42:25.183854+0000","last_undegraded":"2024-04-22T06:42:25.183854+0000","last_fullsized":"2024-04-22T06:42:25.183854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102715+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:42:26.102715+0000","last_peered":"2024-04-22T06:42:26.102715+0000","last_clean":"2024-04-22T06:42:26.102715+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:42:26.102715+0000","last_undegraded":"2024-04-22T06:42:26.102715+0000","last_fullsized":"2024-04-22T06:42:26.102715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103582+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:42:26.103582+0000","last_peered":"2024-04-22T06:42:26.103582+0000","last_clean":"2024-04-22T06:42:26.103582+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:42:26.103582+0000","last_undegraded":"2024-04-22T06:42:26.103582+0000","last_fullsized":"2024-04-22T06:42:26.103582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183851+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:42:25.183851+0000","last_peered":"2024-04-22T06:42:25.183851+0000","last_clean":"2024-04-22T06:42:25.183851+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:42:25.183851+0000","last_undegraded":"2024-04-22T06:42:25.183851+0000","last_fullsized":"2024-04-22T06:42:25.183851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:27.713193+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:42:27.713193+0000","last_peered":"2024-04-22T06:42:27.713193+0000","last_clean":"2024-04-22T06:42:27.713193+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:42:27.713193+0000","last_undegraded":"2024-04-22T06:42:27.713193+0000","last_fullsized":"2024-04-22T06:42:27.713193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476069+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:42:25.476069+0000","last_peered":"2024-04-22T06:42:25.476069+0000","last_clean":"2024-04-22T06:42:25.476069+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:42:25.476069+0000","last_undegraded":"2024-04-22T06:42:25.476069+0000","last_fullsized":"2024-04-22T06:42:25.476069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.104177+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:42:26.104177+0000","last_peered":"2024-04-22T06:42:26.104177+0000","last_clean":"2024-04-22T06:42:26.104177+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:42:26.104177+0000","last_undegraded":"2024-04-22T06:42:26.104177+0000","last_fullsized":"2024-04-22T06:42:26.104177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102811+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:42:26.102811+0000","last_peered":"2024-04-22T06:42:26.102811+0000","last_clean":"2024-04-22T06:42:26.102811+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:42:26.102811+0000","last_undegraded":"2024-04-22T06:42:26.102811+0000","last_fullsized":"2024-04-22T06:42:26.102811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705611+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:42:26.705611+0000","last_peered":"2024-04-22T06:42:26.705611+0000","last_clean":"2024-04-22T06:42:26.705611+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:42:26.705611+0000","last_undegraded":"2024-04-22T06:42:26.705611+0000","last_fullsized":"2024-04-22T06:42:26.705611+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102986+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:42:26.102986+0000","last_peered":"2024-04-22T06:42:26.102986+0000","last_clean":"2024-04-22T06:42:26.102986+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:42:26.102986+0000","last_undegraded":"2024-04-22T06:42:26.102986+0000","last_fullsized":"2024-04-22T06:42:26.102986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705472+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:42:26.705472+0000","last_peered":"2024-04-22T06:42:26.705472+0000","last_clean":"2024-04-22T06:42:26.705472+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:42:26.705472+0000","last_undegraded":"2024-04-22T06:42:26.705472+0000","last_fullsized":"2024-04-22T06:42:26.705472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102864+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:42:26.102864+0000","last_peered":"2024-04-22T06:42:26.102864+0000","last_clean":"2024-04-22T06:42:26.102864+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:42:26.102864+0000","last_undegraded":"2024-04-22T06:42:26.102864+0000","last_fullsized":"2024-04-22T06:42:26.102864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.184868+0000","last_change":"2024-04-22T06:42:23.681287+0000","last_active":"2024-04-22T06:42:25.184868+0000","last_peered":"2024-04-22T06:42:25.184868+0000","last_clean":"2024-04-22T06:42:25.184868+0000","last_became_active":"2024-04-22T06:42:23.680939+0000","last_became_peered":"2024-04-22T06:42:23.680939+0000","last_unstale":"2024-04-22T06:42:25.184868+0000","last_undegraded":"2024-04-22T06:42:25.184868+0000","last_fullsized":"2024-04-22T06:42:25.184868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:56.903734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476068+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:42:25.476068+0000","last_peered":"2024-04-22T06:42:25.476068+0000","last_clean":"2024-04-22T06:42:25.476068+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:42:25.476068+0000","last_undegraded":"2024-04-22T06:42:25.476068+0000","last_fullsized":"2024-04-22T06:42:25.476068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.705624+0000","last_change":"2024-04-22T06:42:23.673602+0000","last_active":"2024-04-22T06:42:26.705624+0000","last_peered":"2024-04-22T06:42:26.705624+0000","last_clean":"2024-04-22T06:42:26.705624+0000","last_became_active":"2024-04-22T06:42:23.673281+0000","last_became_peered":"2024-04-22T06:42:23.673281+0000","last_unstale":"2024-04-22T06:42:26.705624+0000","last_undegraded":"2024-04-22T06:42:26.705624+0000","last_fullsized":"2024-04-22T06:42:26.705624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:39:09.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103436+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:42:26.103436+0000","last_peered":"2024-04-22T06:42:26.103436+0000","last_clean":"2024-04-22T06:42:26.103436+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:42:26.103436+0000","last_undegraded":"2024-04-22T06:42:26.103436+0000","last_fullsized":"2024-04-22T06:42:26.103436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185159+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:42:25.185159+0000","last_peered":"2024-04-22T06:42:25.185159+0000","last_clean":"2024-04-22T06:42:25.185159+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:42:25.185159+0000","last_undegraded":"2024-04-22T06:42:25.185159+0000","last_fullsized":"2024-04-22T06:42:25.185159+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103816+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:42:26.103816+0000","last_peered":"2024-04-22T06:42:26.103816+0000","last_clean":"2024-04-22T06:42:26.103816+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:42:26.103816+0000","last_undegraded":"2024-04-22T06:42:26.103816+0000","last_fullsized":"2024-04-22T06:42:26.103816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103499+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:42:26.103499+0000","last_peered":"2024-04-22T06:42:26.103499+0000","last_clean":"2024-04-22T06:42:26.103499+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:42:26.103499+0000","last_undegraded":"2024-04-22T06:42:26.103499+0000","last_fullsized":"2024-04-22T06:42:26.103499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103995+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:42:26.103995+0000","last_peered":"2024-04-22T06:42:26.103995+0000","last_clean":"2024-04-22T06:42:26.103995+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:42:26.103995+0000","last_undegraded":"2024-04-22T06:42:26.103995+0000","last_fullsized":"2024-04-22T06:42:26.103995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183764+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:42:25.183764+0000","last_peered":"2024-04-22T06:42:25.183764+0000","last_clean":"2024-04-22T06:42:25.183764+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:42:25.183764+0000","last_undegraded":"2024-04-22T06:42:25.183764+0000","last_fullsized":"2024-04-22T06:42:25.183764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103826+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:42:26.103826+0000","last_peered":"2024-04-22T06:42:26.103826+0000","last_clean":"2024-04-22T06:42:26.103826+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:42:26.103826+0000","last_undegraded":"2024-04-22T06:42:26.103826+0000","last_fullsized":"2024-04-22T06:42:26.103826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103044+0000","last_change":"2024-04-22T06:42:24.199973+0000","last_active":"2024-04-22T06:42:26.103044+0000","last_peered":"2024-04-22T06:42:26.103044+0000","last_clean":"2024-04-22T06:42:26.103044+0000","last_became_active":"2024-04-22T06:42:24.199562+0000","last_became_peered":"2024-04-22T06:42:24.199562+0000","last_unstale":"2024-04-22T06:42:26.103044+0000","last_undegraded":"2024-04-22T06:42:26.103044+0000","last_fullsized":"2024-04-22T06:42:26.103044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:34.232346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.184986+0000","last_change":"2024-04-22T06:42:24.194984+0000","last_active":"2024-04-22T06:42:25.184986+0000","last_peered":"2024-04-22T06:42:25.184986+0000","last_clean":"2024-04-22T06:42:25.184986+0000","last_became_active":"2024-04-22T06:42:24.194674+0000","last_became_peered":"2024-04-22T06:42:24.194674+0000","last_unstale":"2024-04-22T06:42:25.184986+0000","last_undegraded":"2024-04-22T06:42:25.184986+0000","last_fullsized":"2024-04-22T06:42:25.184986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:54:52.355697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.184858+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:42:25.184858+0000","last_peered":"2024-04-22T06:42:25.184858+0000","last_clean":"2024-04-22T06:42:25.184858+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:42:25.184858+0000","last_undegraded":"2024-04-22T06:42:25.184858+0000","last_fullsized":"2024-04-22T06:42:25.184858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102269+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:42:26.102269+0000","last_peered":"2024-04-22T06:42:26.102269+0000","last_clean":"2024-04-22T06:42:26.102269+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:42:26.102269+0000","last_undegraded":"2024-04-22T06:42:26.102269+0000","last_fullsized":"2024-04-22T06:42:26.102269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183679+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:42:25.183679+0000","last_peered":"2024-04-22T06:42:25.183679+0000","last_clean":"2024-04-22T06:42:25.183679+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:42:25.183679+0000","last_undegraded":"2024-04-22T06:42:25.183679+0000","last_fullsized":"2024-04-22T06:42:25.183679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.103413+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:42:26.103413+0000","last_peered":"2024-04-22T06:42:26.103413+0000","last_clean":"2024-04-22T06:42:26.103413+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:42:26.103413+0000","last_undegraded":"2024-04-22T06:42:26.103413+0000","last_fullsized":"2024-04-22T06:42:26.103413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.185366+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:42:25.185366+0000","last_peered":"2024-04-22T06:42:25.185366+0000","last_clean":"2024-04-22T06:42:25.185366+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:42:25.185366+0000","last_undegraded":"2024-04-22T06:42:25.185366+0000","last_fullsized":"2024-04-22T06:42:25.185366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.476055+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:42:25.476055+0000","last_peered":"2024-04-22T06:42:25.476055+0000","last_clean":"2024-04-22T06:42:25.476055+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:42:25.476055+0000","last_undegraded":"2024-04-22T06:42:25.476055+0000","last_fullsized":"2024-04-22T06:42:25.476055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102628+0000","last_change":"2024-04-22T06:42:24.183528+0000","last_active":"2024-04-22T06:42:26.102628+0000","last_peered":"2024-04-22T06:42:26.102628+0000","last_clean":"2024-04-22T06:42:26.102628+0000","last_became_active":"2024-04-22T06:42:24.183002+0000","last_became_peered":"2024-04-22T06:42:24.183002+0000","last_unstale":"2024-04-22T06:42:26.102628+0000","last_undegraded":"2024-04-22T06:42:26.102628+0000","last_fullsized":"2024-04-22T06:42:26.102628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:02:47.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.475834+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:42:25.475834+0000","last_peered":"2024-04-22T06:42:25.475834+0000","last_clean":"2024-04-22T06:42:25.475834+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:42:25.475834+0000","last_undegraded":"2024-04-22T06:42:25.475834+0000","last_fullsized":"2024-04-22T06:42:25.475834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.186418+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:42:25.186418+0000","last_peered":"2024-04-22T06:42:25.186418+0000","last_clean":"2024-04-22T06:42:25.186418+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:42:25.186418+0000","last_undegraded":"2024-04-22T06:42:25.186418+0000","last_fullsized":"2024-04-22T06:42:25.186418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:25.183852+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:42:25.183852+0000","last_peered":"2024-04-22T06:42:25.183852+0000","last_clean":"2024-04-22T06:42:25.183852+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:42:25.183852+0000","last_undegraded":"2024-04-22T06:42:25.183852+0000","last_fullsized":"2024-04-22T06:42:25.183852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:26.102720+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:42:26.102720+0000","last_peered":"2024-04-22T06:42:26.102720+0000","last_clean":"2024-04-22T06:42:26.102720+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:42:26.102720+0000","last_undegraded":"2024-04-22T06:42:26.102720+0000","last_fullsized":"2024-04-22T06:42:26.102720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":23602,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":17,"num_read_kb":31,"num_write":102,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":31911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1051870,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":1029471,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":438922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":1030756,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":22864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:42:31.763 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T06:42:31.763 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T06:42:31.764 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T06:42:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:31 smithi079 ceph-mon[27946]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:31 smithi079 ceph-mon[27946]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:31 smithi079 ceph-mon[27946]: pgmap v292: 97 pgs: 97 active+clean; 600 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-22T06:42:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:31 smithi079 ceph-mon[27946]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:42:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/764107072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3700933668' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:32.034 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:31 smithi193 ceph-mon[31815]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:31 smithi193 ceph-mon[31815]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:31 smithi193 ceph-mon[31815]: pgmap v292: 97 pgs: 97 active+clean; 600 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-22T06:42:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:31 smithi193 ceph-mon[31815]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:42:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/764107072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:32.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3700933668' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:31 smithi190 ceph-mon[31421]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:31 smithi190 ceph-mon[31421]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:31 smithi190 ceph-mon[31421]: pgmap v292: 97 pgs: 97 active+clean; 600 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-22T06:42:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:31 smithi190 ceph-mon[31421]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:42:32.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/764107072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:32.204 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3700933668' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:32 smithi079 ceph-mon[27946]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:32 smithi079 ceph-mon[27946]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:33.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:32 smithi193 ceph-mon[31815]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:33.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:32 smithi193 ceph-mon[31815]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:33.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:32 smithi190 ceph-mon[31421]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:33.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:32 smithi190 ceph-mon[31421]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 -- 172.21.15.79:0/960404602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c103560 msgr2=0x7efd4c109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 --2- 172.21.15.79:0/960404602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c103560 0x7efd4c109e10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efd4400b0d0 tx=0x7efd4400b3e0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 -- 172.21.15.79:0/960404602 shutdown_connections 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 --2- 172.21.15.79:0/960404602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c103560 0x7efd4c109e10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 --2- 172.21.15.79:0/960404602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c1019c0 0x7efd4c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 --2- 172.21.15.79:0/960404602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 0x7efd4c103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 -- 172.21.15.79:0/960404602 >> 172.21.15.79:0/960404602 conn(0x7efd4c0fcf60 msgr2=0x7efd4c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:33.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 -- 172.21.15.79:0/960404602 shutdown_connections 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.648+0000 7efd51b8a700 1 -- 172.21.15.79:0/960404602 wait complete. 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 Processor -- start 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 -- start start 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 0x7efd4c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 0x7efd4c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c103560 0x7efd4c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10d5a0 con 0x7efd4c1019c0 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10d420 con 0x7efd4c103560 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd51b8a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10d720 con 0x7efd4c102bc0 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd4b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 0x7efd4c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd4b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 0x7efd4c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45980/0 (socket says 172.21.15.79:45980) 2024-04-22T06:42:33.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.649+0000 7efd4b7fe700 1 -- 172.21.15.79:0/2048218460 learned_addr learned my addr 172.21.15.79:0/2048218460 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4bfff700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c103560 0x7efd4c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd42ffd700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 0x7efd4c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4b7fe700 1 -- 172.21.15.79:0/2048218460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 msgr2=0x7efd4c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4b7fe700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 0x7efd4c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4b7fe700 1 -- 172.21.15.79:0/2048218460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c103560 msgr2=0x7efd4c19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4b7fe700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c103560 0x7efd4c19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4b7fe700 1 -- 172.21.15.79:0/2048218460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c06b1f0 con 0x7efd4c1019c0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd42ffd700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 0x7efd4c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4bfff700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c103560 0x7efd4c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd4b7fe700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 0x7efd4c197b50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00b530 tx=0x7efd3c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c015020 con 0x7efd4c1019c0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd3c011ce0 con 0x7efd4c1019c0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c010d60 con 0x7efd4c1019c0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4c06b4e0 con 0x7efd4c1019c0 2024-04-22T06:42:33.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.650+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4c06baa0 con 0x7efd4c1019c0 2024-04-22T06:42:33.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.651+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd4c050f80 con 0x7efd4c1019c0 2024-04-22T06:42:33.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.652+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efd3c018020 con 0x7efd4c1019c0 2024-04-22T06:42:33.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.653+0000 7efd497fa700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd380a3d80 0x7efd380a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.653+0000 7efd42ffd700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd380a3d80 0x7efd380a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.653+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7efd3c0cfb30 con 0x7efd4c1019c0 2024-04-22T06:42:33.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.653+0000 7efd42ffd700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd380a3d80 0x7efd380a6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efd34007900 tx=0x7efd34008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.657+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd3c01f020 con 0x7efd4c1019c0 2024-04-22T06:42:33.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.943+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7efd4c06be80 con 0x7efd4c1019c0 2024-04-22T06:42:33.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.946+0000 7efd497fa700 1 -- 172.21.15.79:0/2048218460 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7efd4c06be80 con 0x7efd4c1019c0 2024-04-22T06:42:33.944 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-04-22T06:42:33.945 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQDZBiZm8J5UOBAAapAkOSz/W+9ltsw5UvSqDQ== 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd380a3d80 msgr2=0x7efd380a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd380a3d80 0x7efd380a6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efd34007900 tx=0x7efd34008040 comp rx=0 tx=0).stop 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 msgr2=0x7efd4c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 0x7efd4c197b50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00b530 tx=0x7efd3c00b8f0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 shutdown_connections 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd4c103560 0x7efd4c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd380a3d80 0x7efd380a6240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd4c1019c0 0x7efd4c197b50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 --2- 172.21.15.79:0/2048218460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd4c102bc0 0x7efd4c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.948+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 >> 172.21.15.79:0/2048218460 conn(0x7efd4c0fcf60 msgr2=0x7efd4c105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:33.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.949+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 shutdown_connections 2024-04-22T06:42:33.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:33.949+0000 7efd51b8a700 1 -- 172.21.15.79:0/2048218460 wait complete. 2024-04-22T06:42:33.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:33 smithi079 ceph-mon[27946]: pgmap v293: 97 pgs: 97 active+clean; 601 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 7.4 KiB/s wr, 19 op/s 2024-04-22T06:42:34.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:33 smithi193 ceph-mon[31815]: pgmap v293: 97 pgs: 97 active+clean; 601 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 7.4 KiB/s wr, 19 op/s 2024-04-22T06:42:34.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:33 smithi190 ceph-mon[31421]: pgmap v293: 97 pgs: 97 active+clean; 601 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 7.4 KiB/s wr, 19 op/s 2024-04-22T06:42:34.604 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:42:34.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T06:42:34.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T06:42:34.640 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T06:42:34.640 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T06:42:34.641 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T06:42:34.916 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2048218460' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:42:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2048218460' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:42:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2048218460' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:42:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2048218460' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:42:35.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2048218460' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:42:35.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2048218460' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:42:35.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:35 smithi079 ceph-mon[27946]: pgmap v294: 97 pgs: 97 active+clean; 601 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-04-22T06:42:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:35 smithi193 ceph-mon[31815]: pgmap v294: 97 pgs: 97 active+clean; 601 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-04-22T06:42:36.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:35 smithi190 ceph-mon[31421]: pgmap v294: 97 pgs: 97 active+clean; 601 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-04-22T06:42:36.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 -- 172.21.15.79:0/2899872432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 msgr2=0x7f49301065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 --2- 172.21.15.79:0/2899872432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49301065e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f49200099c0 tx=0x7f4920009cd0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 -- 172.21.15.79:0/2899872432 shutdown_connections 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 --2- 172.21.15.79:0/2899872432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 0x7f493010e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 --2- 172.21.15.79:0/2899872432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 0x7f4930102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 --2- 172.21.15.79:0/2899872432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49301065e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 -- 172.21.15.79:0/2899872432 >> 172.21.15.79:0/2899872432 conn(0x7f4930077fb0 msgr2=0x7f49300ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 -- 172.21.15.79:0/2899872432 shutdown_connections 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.533+0000 7f4938274700 1 -- 172.21.15.79:0/2899872432 wait complete. 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 Processor -- start 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 -- start start 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 0x7f49300fd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49300fd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 0x7f49300fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111b90 con 0x7f4930102d50 2024-04-22T06:42:36.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111a10 con 0x7f4930106cc0 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4938274700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111d10 con 0x7f49301023a0 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49300fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49300fd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45990/0 (socket says 172.21.15.79:45990) 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 -- 172.21.15.79:0/1658335526 learned_addr learned my addr 172.21.15.79:0/1658335526 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4936811700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 0x7f49300fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4936010700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 0x7f49300fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 -- 172.21.15.79:0/1658335526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 msgr2=0x7f49300fd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 0x7f49300fd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 -- 172.21.15.79:0/1658335526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 msgr2=0x7f49300fdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 0x7f49300fdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 -- 172.21.15.79:0/1658335526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49301a77c0 con 0x7f4930102d50 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4936811700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 0x7f49300fdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f4936010700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 0x7f49300fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:36.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.534+0000 7f493580f700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49300fd960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4920009f50 tx=0x7f49200037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.535+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4920027070 con 0x7f4930102d50 2024-04-22T06:42:36.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.535+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4920004140 con 0x7f4930102d50 2024-04-22T06:42:36.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.535+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492000f460 con 0x7f4930102d50 2024-04-22T06:42:36.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.535+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49301a7a50 con 0x7f4930102d50 2024-04-22T06:42:36.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.535+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49301a7f00 con 0x7f4930102d50 2024-04-22T06:42:36.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.536+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4930069730 con 0x7f4930102d50 2024-04-22T06:42:36.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.538+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f492002a070 con 0x7f4930102d50 2024-04-22T06:42:36.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.538+0000 7f49277fe700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f491c0a3d50 0x7f491c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.538+0000 7f4936010700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f491c0a3d50 0x7f491c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.538+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f49200d15c0 con 0x7f4930102d50 2024-04-22T06:42:36.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.539+0000 7f4936010700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f491c0a3d50 0x7f491c0a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4928007fd0 tx=0x7f4928007810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.542+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4920013070 con 0x7f4930102d50 2024-04-22T06:42:36.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.799+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f49300fe950 con 0x7f4930102d50 2024-04-22T06:42:36.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.804+0000 7f49277fe700 1 -- 172.21.15.79:0/1658335526 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264995 (secure 0 0 0) 0x7f492009e080 con 0x7f4930102d50 2024-04-22T06:42:36.807 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:36.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f491c0a3d50 msgr2=0x7f491c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f491c0a3d50 0x7f491c0a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4928007fd0 tx=0x7f4928007810 comp rx=0 tx=0).stop 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 msgr2=0x7f49300fd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49300fd960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4920009f50 tx=0x7f49200037c0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 shutdown_connections 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4930106cc0 0x7f49300fdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49301023a0 0x7f49300fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f491c0a3d50 0x7f491c0a6210 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 --2- 172.21.15.79:0/1658335526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4930102d50 0x7f49300fd960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 >> 172.21.15.79:0/1658335526 conn(0x7f4930077fb0 msgr2=0x7f4930101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 shutdown_connections 2024-04-22T06:42:36.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:36.807+0000 7f4938274700 1 -- 172.21.15.79:0/1658335526 wait complete. 2024-04-22T06:42:37.516 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":3007546422},{"type":"v1","addr":"172.21.15.79:6801","nonce":3007546422}]},"active_addr":"172.21.15.79:6801/3007546422","active_change":"2024-04-22T06:35:16.561616+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":4022603925}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":196117513}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":482537985}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":749551431}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3859559794}]}]} 2024-04-22T06:42:37.522 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T06:42:37.522 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T06:42:37.523 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:42:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:37 smithi079 ceph-mon[27946]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-22T06:42:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:37 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1658335526' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:42:37.761 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:38.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:37 smithi193 ceph-mon[31815]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-22T06:42:38.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:37 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1658335526' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:42:38.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:37 smithi190 ceph-mon[31421]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-04-22T06:42:38.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:37 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1658335526' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:42:39.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 -- 172.21.15.79:0/3330610872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c102ac0 msgr2=0x7efd3c102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 --2- 172.21.15.79:0/3330610872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c102ac0 0x7efd3c102f40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd24009990 tx=0x7efd24009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 -- 172.21.15.79:0/3330610872 shutdown_connections 2024-04-22T06:42:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 --2- 172.21.15.79:0/3330610872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c103480 0x7efd3c10f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 --2- 172.21.15.79:0/3330610872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd3c108a70 0x7efd3c108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 --2- 172.21.15.79:0/3330610872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c102ac0 0x7efd3c102f40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 -- 172.21.15.79:0/3330610872 >> 172.21.15.79:0/3330610872 conn(0x7efd3c079bb0 msgr2=0x7efd3c079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 -- 172.21.15.79:0/3330610872 shutdown_connections 2024-04-22T06:42:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.566+0000 7efd41d2e700 1 -- 172.21.15.79:0/3330610872 wait complete. 2024-04-22T06:42:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 Processor -- start 2024-04-22T06:42:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 -- start start 2024-04-22T06:42:39.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd3c102ac0 0x7efd3c19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 0x7efd3c19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c108a70 0x7efd3c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c112ca0 con 0x7efd3c103480 2024-04-22T06:42:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c112b20 con 0x7efd3c108a70 2024-04-22T06:42:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd41d2e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c112e20 con 0x7efd3c102ac0 2024-04-22T06:42:39.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 0x7efd3c19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 0x7efd3c19b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46014/0 (socket says 172.21.15.79:46014) 2024-04-22T06:42:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 -- 172.21.15.79:0/2145331869 learned_addr learned my addr 172.21.15.79:0/2145331869 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd3b7fe700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd3c102ac0 0x7efd3c19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd3bfff700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c108a70 0x7efd3c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:39.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 -- 172.21.15.79:0/2145331869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd3c102ac0 msgr2=0x7efd3c19b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd3c102ac0 0x7efd3c19b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 -- 172.21.15.79:0/2145331869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c108a70 msgr2=0x7efd3c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c108a70 0x7efd3c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd33fff700 1 -- 172.21.15.79:0/2145331869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c1a0380 con 0x7efd3c103480 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.567+0000 7efd3bfff700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c108a70 0x7efd3c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.568+0000 7efd33fff700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 0x7efd3c19b8b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efd24000c00 tx=0x7efd240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:39.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.568+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24027070 con 0x7efd3c103480 2024-04-22T06:42:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.568+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd24003f20 con 0x7efd3c103480 2024-04-22T06:42:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.568+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3c1a0610 con 0x7efd3c103480 2024-04-22T06:42:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.568+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24022cf0 con 0x7efd3c103480 2024-04-22T06:42:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.568+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3c1a7ef0 con 0x7efd3c103480 2024-04-22T06:42:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.569+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3c069730 con 0x7efd3c103480 2024-04-22T06:42:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.570+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efd2402a070 con 0x7efd3c103480 2024-04-22T06:42:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.570+0000 7efd397fa700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd280a3d50 0x7efd280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.570+0000 7efd3b7fe700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd280a3d50 0x7efd280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.571+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7efd240d14b0 con 0x7efd3c103480 2024-04-22T06:42:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.571+0000 7efd3b7fe700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd280a3d50 0x7efd280a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd2c004630 tx=0x7efd2c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:39.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.574+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd24013070 con 0x7efd3c103480 2024-04-22T06:42:39.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.776+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd3c02df80 con 0x7efd3c103480 2024-04-22T06:42:39.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.777+0000 7efd397fa700 1 -- 172.21.15.79:0/2145331869 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22217 (secure 0 0 0) 0x7efd2409df70 con 0x7efd3c103480 2024-04-22T06:42:39.776 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:39.776 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":85,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:42:25.177009+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd280a3d50 msgr2=0x7efd280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd280a3d50 0x7efd280a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd2c004630 tx=0x7efd2c00b3c0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 msgr2=0x7efd3c19b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 0x7efd3c19b8b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efd24000c00 tx=0x7efd240047b0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 shutdown_connections 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd3c108a70 0x7efd3c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd3c102ac0 0x7efd3c19b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efd280a3d50 0x7efd280a6210 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 --2- 172.21.15.79:0/2145331869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd3c103480 0x7efd3c19b8b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 >> 172.21.15.79:0/2145331869 conn(0x7efd3c079bb0 msgr2=0x7efd3c10df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 shutdown_connections 2024-04-22T06:42:39.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:39.779+0000 7efd41d2e700 1 -- 172.21.15.79:0/2145331869 wait complete. 2024-04-22T06:42:39.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:39 smithi079 ceph-mon[27946]: pgmap v296: 97 pgs: 97 active+clean; 623 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.0 KiB/s wr, 16 op/s 2024-04-22T06:42:40.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:39 smithi193 ceph-mon[31815]: pgmap v296: 97 pgs: 97 active+clean; 623 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.0 KiB/s wr, 16 op/s 2024-04-22T06:42:40.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:39 smithi190 ceph-mon[31421]: pgmap v296: 97 pgs: 97 active+clean; 623 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.0 KiB/s wr, 16 op/s 2024-04-22T06:42:40.431 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T06:42:40.432 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:42:40.676 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:40.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:40 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2145331869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:40.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:40 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2145331869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:41.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:40 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2145331869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:41.879 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:41 smithi079 ceph-mon[27946]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-04-22T06:42:42.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:41 smithi193 ceph-mon[31815]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-04-22T06:42:42.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:41 smithi190 ceph-mon[31421]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-04-22T06:42:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 -- 172.21.15.79:0/2425417867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd128108a30 msgr2=0x7fd128108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/2425417867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd128108a30 0x7fd128108e10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd118009990 tx=0x7fd118009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 -- 172.21.15.79:0/2425417867 shutdown_connections 2024-04-22T06:42:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/2425417867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd12806b0f0 0x7fd12810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/2425417867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd128108a30 0x7fd128108e10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/2425417867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd12806a750 0x7fd12806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 -- 172.21.15.79:0/2425417867 >> 172.21.15.79:0/2425417867 conn(0x7fd1280788f0 msgr2=0x7fd128078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 -- 172.21.15.79:0/2425417867 shutdown_connections 2024-04-22T06:42:42.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.409+0000 7fd12d2ea700 1 -- 172.21.15.79:0/2425417867 wait complete. 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.410+0000 7fd12d2ea700 1 Processor -- start 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.410+0000 7fd12d2ea700 1 -- start start 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd12806a750 0x7fd128199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 0x7fd128199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128108a30 0x7fd12819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd128112cb0 con 0x7fd12806b0f0 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd128112b30 con 0x7fd12806a750 2024-04-22T06:42:42.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd128112e30 con 0x7fd128108a30 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 0x7fd128199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 0x7fd128199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46032/0 (socket says 172.21.15.79:46032) 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 -- 172.21.15.79:0/805526801 learned_addr learned my addr 172.21.15.79:0/805526801 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd126ffd700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd12806a750 0x7fd128199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1277fe700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128108a30 0x7fd12819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 -- 172.21.15.79:0/805526801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128108a30 msgr2=0x7fd12819db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128108a30 0x7fd12819db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 -- 172.21.15.79:0/805526801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd12806a750 msgr2=0x7fd128199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd12806a750 0x7fd128199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 -- 172.21.15.79:0/805526801 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12819e250 con 0x7fd12806b0f0 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1277fe700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128108a30 0x7fd12819db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd1267fc700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 0x7fd128199780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00b530 tx=0x7fd11c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c015020 con 0x7fd12806b0f0 2024-04-22T06:42:42.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12819e540 con 0x7fd12806b0f0 2024-04-22T06:42:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd11c011ce0 con 0x7fd12806b0f0 2024-04-22T06:42:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1281a5dc0 con 0x7fd12806b0f0 2024-04-22T06:42:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.411+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c010d20 con 0x7fd12806b0f0 2024-04-22T06:42:42.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.412+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd128069730 con 0x7fd12806b0f0 2024-04-22T06:42:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.414+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd11c018020 con 0x7fd12806b0f0 2024-04-22T06:42:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.415+0000 7fd10ffff700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1100a3cd0 0x7fd1100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.415+0000 7fd126ffd700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1100a3cd0 0x7fd1100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.415+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd11c0d00d0 con 0x7fd12806b0f0 2024-04-22T06:42:42.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.415+0000 7fd126ffd700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1100a3cd0 0x7fd1100a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd11800b5c0 tx=0x7fd118005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:42.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.418+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd11c01f020 con 0x7fd12806b0f0 2024-04-22T06:42:42.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.620+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd12802df80 con 0x7fd12806b0f0 2024-04-22T06:42:42.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.621+0000 7fd10ffff700 1 -- 172.21.15.79:0/805526801 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22217 (secure 0 0 0) 0x7fd11c09cc10 con 0x7fd12806b0f0 2024-04-22T06:42:42.621 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:42:42.621 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":85,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:42:25.177009+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:42.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1100a3cd0 msgr2=0x7fd1100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1100a3cd0 0x7fd1100a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd11800b5c0 tx=0x7fd118005fb0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 msgr2=0x7fd128199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 0x7fd128199780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00b530 tx=0x7fd11c00b8f0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 shutdown_connections 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128108a30 0x7fd12819db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd12806a750 0x7fd128199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd1100a3cd0 0x7fd1100a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 --2- 172.21.15.79:0/805526801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd12806b0f0 0x7fd128199780 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 >> 172.21.15.79:0/805526801 conn(0x7fd1280788f0 msgr2=0x7fd1280ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 shutdown_connections 2024-04-22T06:42:42.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:42.623+0000 7fd12d2ea700 1 -- 172.21.15.79:0/805526801 wait complete. 2024-04-22T06:42:43.058 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:42 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/805526801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:42 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/805526801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:43.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:42 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/805526801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:43.415 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T06:42:43.416 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T06:42:43.416 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T06:42:43.417 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T06:42:43.417 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T06:42:43.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T06:42:43.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T06:42:43.418 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T06:42:43.419 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-22T06:42:43.419 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-22T06:42:43.420 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-22T06:42:43.420 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-22T06:42:44.058 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.165 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.167 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.167 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.171 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.171 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.171 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:43 smithi193 ceph-mon[31815]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 11 op/s 2024-04-22T06:42:44.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:43 smithi190 ceph-mon[31421]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 11 op/s 2024-04-22T06:42:44.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:43 smithi079 ceph-mon[27946]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.4 KiB/s wr, 11 op/s 2024-04-22T06:42:44.400 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.421 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:44.421 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:45 smithi079 ceph-mon[27946]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.4 KiB/s wr, 7 op/s 2024-04-22T06:42:46.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:45 smithi193 ceph-mon[31815]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.4 KiB/s wr, 7 op/s 2024-04-22T06:42:46.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:45 smithi190 ceph-mon[31421]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.4 KiB/s wr, 7 op/s 2024-04-22T06:42:48.023 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:47 smithi079 ceph-mon[27946]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-22T06:42:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:47 smithi193 ceph-mon[31815]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-22T06:42:48.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:47 smithi190 ceph-mon[31421]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-04-22T06:42:49.790 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:49 smithi079 ceph-mon[27946]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T06:42:50.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:49 smithi193 ceph-mon[31815]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T06:42:50.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:49 smithi190 ceph-mon[31421]: pgmap v301: 97 pgs: 97 active+clean; 636 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1815215368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 msgr2=0x7fcb94107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1815215368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb94107d70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009990 tx=0x7fcb88009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1815215368 shutdown_connections 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1815215368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb94102300 0x7fcb9410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1815215368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb94107d70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1815215368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb94101940 0x7fcb94101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1815215368 >> 172.21.15.79:0/1815215368 conn(0x7fcb940788f0 msgr2=0x7fcb94078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1815215368 shutdown_connections 2024-04-22T06:42:50.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.541+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1815215368 wait complete. 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 Processor -- start 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 -- start start 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb94101940 0x7fcb9419b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb94102300 0x7fcb9419b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb9419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94111bd0 con 0x7fcb94107990 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94111a50 con 0x7fcb94101940 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb98e3d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94111d50 con 0x7fcb94102300 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb9419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb9419fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54974/0 (socket says 172.21.15.79:54974) 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 -- 172.21.15.79:0/1555207329 learned_addr learned my addr 172.21.15.79:0/1555207329 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb9259c700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb94101940 0x7fcb9419b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:50.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb91d9b700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb94102300 0x7fcb9419b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 -- 172.21.15.79:0/1555207329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb94102300 msgr2=0x7fcb9419b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb94102300 0x7fcb9419b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 -- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb94101940 msgr2=0x7fcb9419b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.542+0000 7fcb92d9d700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb94101940 0x7fcb9419b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb92d9d700 1 -- 172.21.15.79:0/1555207329 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb941a0460 con 0x7fcb94107990 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb9259c700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb94101940 0x7fcb9419b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb92d9d700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb9419fd20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00b970 tx=0x7fcb8c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c019070 con 0x7fcb94107990 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c014440 con 0x7fcb94107990 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb941a0750 con 0x7fcb94107990 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c00ce60 con 0x7fcb94107990 2024-04-22T06:42:50.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.543+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb941a7fd0 con 0x7fcb94107990 2024-04-22T06:42:50.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.544+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb9406b1a0 con 0x7fcb94107990 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.545+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcb8c01c070 con 0x7fcb94107990 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.545+0000 7fcb837fe700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcb7c0a3dd0 0x7fcb7c0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb9259c700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcb7c0a3dd0 0x7fcb7c0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcb8c0cffc0 con 0x7fcb94107990 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb837fe700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] conn(0x7fcb7c0ad5b0 0x7fcb7c0af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 --> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb7c0b0080 con 0x7fcb7c0ad5b0 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fcb8c0d03b0 con 0x7fcb94107990 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb9259c700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcb7c0a3dd0 0x7fcb7c0a6290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcb88006010 tx=0x7fcb8800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.546+0000 7fcb91d9b700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] conn(0x7fcb7c0ad5b0 0x7fcb7c0af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:50.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.547+0000 7fcb91d9b700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] conn(0x7fcb7c0ad5b0 0x7fcb7c0af9d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:50.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.548+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== osd.10 v2:172.21.15.190:6824/2195513042 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb7c0b0080 con 0x7fcb7c0ad5b0 2024-04-22T06:42:50.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.563+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 --> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb94103060 con 0x7fcb7c0ad5b0 2024-04-22T06:42:50.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb837fe700 1 -- 172.21.15.79:0/1555207329 <== osd.10 v2:172.21.15.190:6824/2195513042 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcb94103060 con 0x7fcb7c0ad5b0 2024-04-22T06:42:50.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] conn(0x7fcb7c0ad5b0 msgr2=0x7fcb7c0af9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:50.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] conn(0x7fcb7c0ad5b0 0x7fcb7c0af9d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:50.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcb7c0a3dd0 msgr2=0x7fcb7c0a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:50.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fcb7c0a3dd0 0x7fcb7c0a6290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcb88006010 tx=0x7fcb8800b540 comp rx=0 tx=0).stop 2024-04-22T06:42:50.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 msgr2=0x7fcb9419fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:50.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 --2- 172.21.15.79:0/1555207329 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb94107990 0x7fcb9419fd20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00b970 tx=0x7fcb8c00bd30 comp rx=0 tx=0).stop 2024-04-22T06:42:50.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb92d9d700 1 -- 172.21.15.79:0/1555207329 reap_dead start 2024-04-22T06:42:50.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 shutdown_connections 2024-04-22T06:42:50.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 >> 172.21.15.79:0/1555207329 conn(0x7fcb940788f0 msgr2=0x7fcb9406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:50.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 shutdown_connections 2024-04-22T06:42:50.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:50.564+0000 7fcb98e3d700 1 -- 172.21.15.79:0/1555207329 wait complete. 2024-04-22T06:42:52.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:51 smithi079 ceph-mon[27946]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-04-22T06:42:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:51 smithi193 ceph-mon[31815]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-04-22T06:42:52.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:51 smithi190 ceph-mon[31421]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-04-22T06:42:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:53 smithi193 ceph-mon[31815]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:42:54.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:53 smithi190 ceph-mon[31421]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:42:54.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:53 smithi079 ceph-mon[27946]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:42:54.396 INFO:teuthology.orchestra.run.smithi079.stdout:309237645336 2024-04-22T06:42:54.396 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-22T06:42:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:55 smithi079 ceph-mon[27946]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:42:56.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:55 smithi193 ceph-mon[31815]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:42:56.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:55 smithi190 ceph-mon[31421]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 -- 172.21.15.79:0/3404772028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7974108a30 msgr2=0x7f7974108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 --2- 172.21.15.79:0/3404772028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7974108a30 0x7f7974108e10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7970009990 tx=0x7f7970009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 -- 172.21.15.79:0/3404772028 shutdown_connections 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 --2- 172.21.15.79:0/3404772028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f797406b0f0 0x7f797410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 --2- 172.21.15.79:0/3404772028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7974108a30 0x7f7974108e10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 --2- 172.21.15.79:0/3404772028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f797406a750 0x7f797406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 -- 172.21.15.79:0/3404772028 >> 172.21.15.79:0/3404772028 conn(0x7f79740788f0 msgr2=0x7f7974078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 -- 172.21.15.79:0/3404772028 shutdown_connections 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.451+0000 7f797bbe3700 1 -- 172.21.15.79:0/3404772028 wait complete. 2024-04-22T06:42:57.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 Processor -- start 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 -- start start 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f797406a750 0x7f7974199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f797406b0f0 0x7f7974199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7974108a30 0x7f797419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7974112c30 con 0x7f797406b0f0 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7974112ab0 con 0x7f7974108a30 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797bbe3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7974112db0 con 0x7f797406a750 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797917e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f797406b0f0 0x7f7974199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797917e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f797406b0f0 0x7f7974199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53096/0 (socket says 172.21.15.79:53096) 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797917e700 1 -- 172.21.15.79:0/274644998 learned_addr learned my addr 172.21.15.79:0/274644998 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797997f700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f797406a750 0x7f7974199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.452+0000 7f797a180700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7974108a30 0x7f797419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:57.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797917e700 1 -- 172.21.15.79:0/274644998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f797406a750 msgr2=0x7f7974199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797917e700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f797406a750 0x7f7974199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797917e700 1 -- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7974108a30 msgr2=0x7f797419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797917e700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7974108a30 0x7f797419da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797917e700 1 -- 172.21.15.79:0/274644998 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797419e150 con 0x7f797406b0f0 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797917e700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f797406b0f0 0x7f7974199680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f796400b530 tx=0x7f796400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964015020 con 0x7f797406b0f0 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797419e440 con 0x7f797406b0f0 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7964011ce0 con 0x7f797406b0f0 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964010dd0 con 0x7f797406b0f0 2024-04-22T06:42:57.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.453+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79741a5cc0 con 0x7f797406b0f0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.454+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f79740631d0 con 0x7f797406b0f0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.455+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7964018020 con 0x7f797406b0f0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f796affd700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79600a3cd0 0x7f79600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f797997f700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79600a3cd0 0x7f79600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f79640cf990 con 0x7f797406b0f0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f796affd700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] conn(0x7f79600ad4b0 0x7f79600af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 --> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79600aff80 con 0x7f79600ad4b0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f79640d42c0 con 0x7f797406b0f0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f797997f700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79600a3cd0 0x7f79600a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f797000b5c0 tx=0x7f7970005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.456+0000 7f797a180700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] conn(0x7f79600ad4b0 0x7f79600af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.457+0000 7f797a180700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] conn(0x7f79600ad4b0 0x7f79600af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:57.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.457+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== osd.1 v2:172.21.15.190:6800/237015036 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f79600aff80 con 0x7f79600ad4b0 2024-04-22T06:42:57.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 --> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f79740635d0 con 0x7f79600ad4b0 2024-04-22T06:42:57.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f796affd700 1 -- 172.21.15.79:0/274644998 <== osd.1 v2:172.21.15.190:6800/237015036 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f79740635d0 con 0x7f79600ad4b0 2024-04-22T06:42:57.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] conn(0x7f79600ad4b0 msgr2=0x7f79600af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:57.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] conn(0x7f79600ad4b0 0x7f79600af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:57.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79600a3cd0 msgr2=0x7f79600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:57.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79600a3cd0 0x7f79600a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f797000b5c0 tx=0x7f7970005960 comp rx=0 tx=0).stop 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f797406b0f0 msgr2=0x7f7974199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797bbe3700 1 --2- 172.21.15.79:0/274644998 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f797406b0f0 0x7f7974199680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f796400b530 tx=0x7f796400b840 comp rx=0 tx=0).stop 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.474+0000 7f797a180700 1 -- 172.21.15.79:0/274644998 reap_dead start 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.475+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 shutdown_connections 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.475+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 >> 172.21.15.79:0/274644998 conn(0x7f79740788f0 msgr2=0x7f79740ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.475+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 shutdown_connections 2024-04-22T06:42:57.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:57.475+0000 7f797bbe3700 1 -- 172.21.15.79:0/274644998 wait complete. 2024-04-22T06:42:58.087 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:57 smithi079 ceph-mon[27946]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T06:42:58.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:57 smithi193 ceph-mon[31815]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T06:42:58.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 -- 172.21.15.79:0/2740445602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444101940 msgr2=0x7f4444101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 --2- 172.21.15.79:0/2740445602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444101940 0x7f4444101dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4434009990 tx=0x7f4434009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 -- 172.21.15.79:0/2740445602 shutdown_connections 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 --2- 172.21.15.79:0/2740445602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4444102300 0x7f444410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 --2- 172.21.15.79:0/2740445602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4444107990 0x7f4444107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 --2- 172.21.15.79:0/2740445602 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444101940 0x7f4444101dc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 -- 172.21.15.79:0/2740445602 >> 172.21.15.79:0/2740445602 conn(0x7f44440788f0 msgr2=0x7f4444078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 -- 172.21.15.79:0/2740445602 shutdown_connections 2024-04-22T06:42:58.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.190+0000 7f444b2b0700 1 -- 172.21.15.79:0/2740445602 wait complete. 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 Processor -- start 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 -- start start 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4444101940 0x7f4444198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4444102300 0x7f44441985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444107990 0x7f444419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4444111b70 con 0x7f4444107990 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f44441119f0 con 0x7f4444101940 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444b2b0700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4444111cf0 con 0x7f4444102300 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444107990 0x7f444419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444107990 0x7f444419c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53116/0 (socket says 172.21.15.79:53116) 2024-04-22T06:42:58.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 -- 172.21.15.79:0/1124293298 learned_addr learned my addr 172.21.15.79:0/1124293298 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444884b700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4444102300 0x7f44441985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444904c700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4444101940 0x7f4444198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 -- 172.21.15.79:0/1124293298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4444102300 msgr2=0x7f44441985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4444102300 0x7f44441985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 -- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4444101940 msgr2=0x7f4444198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4444101940 0x7f4444198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 -- 172.21.15.79:0/1124293298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444419d090 con 0x7f4444107990 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.191+0000 7f444984d700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444107990 0x7f444419c950 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f443c00ada0 tx=0x7f443c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.192+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c017020 con 0x7f4444107990 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.192+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f443c00ce30 con 0x7f4444107990 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.192+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c005350 con 0x7f4444107990 2024-04-22T06:42:58.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.192+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444419d380 con 0x7f4444107990 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.192+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44441a4c00 con 0x7f4444107990 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.192+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44440631d0 con 0x7f4444107990 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f443c00c980 con 0x7f4444107990 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f443a7fc700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44300a3cd0 0x7f44300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f444904c700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44300a3cd0 0x7f44300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f443c0d04a0 con 0x7f4444107990 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f443a7fc700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] conn(0x7f44300ad4b0 0x7f44300af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 --> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f443c023e50 con 0x7f44300ad4b0 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.194+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f443c0d0980 con 0x7f4444107990 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.195+0000 7f444904c700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44300a3cd0 0x7f44300a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4440006fd0 tx=0x7f4440008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.195+0000 7f444884b700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] conn(0x7f44300ad4b0 0x7f44300af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.195+0000 7f444884b700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] conn(0x7f44300ad4b0 0x7f44300af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.196+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== osd.4 v2:172.21.15.190:6808/3437671425 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f443c023e50 con 0x7f44300ad4b0 2024-04-22T06:42:58.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:57 smithi190 ceph-mon[31421]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T06:42:58.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.212+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 --> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4444063650 con 0x7f44300ad4b0 2024-04-22T06:42:58.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f443a7fc700 1 -- 172.21.15.79:0/1124293298 <== osd.4 v2:172.21.15.190:6808/3437671425 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4444063650 con 0x7f44300ad4b0 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] conn(0x7f44300ad4b0 msgr2=0x7f44300af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] conn(0x7f44300ad4b0 0x7f44300af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44300a3cd0 msgr2=0x7f44300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f44300a3cd0 0x7f44300a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4440006fd0 tx=0x7f4440008040 comp rx=0 tx=0).stop 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444107990 msgr2=0x7f444419c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 --2- 172.21.15.79:0/1124293298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4444107990 0x7f444419c950 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f443c00ada0 tx=0x7f443c00c330 comp rx=0 tx=0).stop 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444984d700 1 -- 172.21.15.79:0/1124293298 reap_dead start 2024-04-22T06:42:58.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 shutdown_connections 2024-04-22T06:42:58.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 >> 172.21.15.79:0/1124293298 conn(0x7f44440788f0 msgr2=0x7f44440ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:58.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.213+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 shutdown_connections 2024-04-22T06:42:58.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.214+0000 7f444b2b0700 1 -- 172.21.15.79:0/1124293298 wait complete. 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 -- 172.21.15.79:0/500444179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 msgr2=0x7fa7a0108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 --2- 172.21.15.79:0/500444179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0108e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa794009990 tx=0x7fa794009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 -- 172.21.15.79:0/500444179 shutdown_connections 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 --2- 172.21.15.79:0/500444179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a006b980 0x7fa7a010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 --2- 172.21.15.79:0/500444179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0108e00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 --2- 172.21.15.79:0/500444179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7a006afe0 0x7fa7a006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 -- 172.21.15.79:0/500444179 >> 172.21.15.79:0/500444179 conn(0x7fa7a00788b0 msgr2=0x7fa7a0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 -- 172.21.15.79:0/500444179 shutdown_connections 2024-04-22T06:42:58.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.408+0000 7fa7a5731700 1 -- 172.21.15.79:0/500444179 wait complete. 2024-04-22T06:42:58.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 Processor -- start 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 -- start start 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7a006afe0 0x7fa7a0101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a006b980 0x7fa7a0101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112ca0 con 0x7fa7a0108a20 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112b20 con 0x7fa7a006b980 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa7a5731700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112e20 con 0x7fa7a006afe0 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53132/0 (socket says 172.21.15.79:53132) 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 -- 172.21.15.79:0/3563312405 learned_addr learned my addr 172.21.15.79:0/3563312405 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79e7fc700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a006b980 0x7fa7a0101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79effd700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7a006afe0 0x7fa7a0101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 -- 172.21.15.79:0/3563312405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7a006afe0 msgr2=0x7fa7a0101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7a006afe0 0x7fa7a0101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 -- 172.21.15.79:0/3563312405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a006b980 msgr2=0x7fa7a0101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7a006b980 0x7fa7a0101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.409+0000 7fa79f7fe700 1 -- 172.21.15.79:0/3563312405 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a01a77d0 con 0x7fa7a0108a20 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.410+0000 7fa79f7fe700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0102480 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa79800b970 tx=0x7fa79800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.410+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798019070 con 0x7fa7a0108a20 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.410+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a01a7ac0 con 0x7fa7a0108a20 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.410+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa79800cde0 con 0x7fa7a0108a20 2024-04-22T06:42:58.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.410+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798007370 con 0x7fa7a0108a20 2024-04-22T06:42:58.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.410+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a8000 con 0x7fa7a0108a20 2024-04-22T06:42:58.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.411+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa7a00630d0 con 0x7fa7a0108a20 2024-04-22T06:42:58.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.412+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa79801c070 con 0x7fa7a0108a20 2024-04-22T06:42:58.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa787fff700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7880a3d50 0x7fa7880a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa79effd700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7880a3d50 0x7fa7880a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa7980cfe80 con 0x7fa7a0108a20 2024-04-22T06:42:58.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa787fff700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] conn(0x7fa7880ad530 0x7fa7880af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 --> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa79800ee50 con 0x7fa7880ad530 2024-04-22T06:42:58.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa7980d0360 con 0x7fa7a0108a20 2024-04-22T06:42:58.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa79e7fc700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] conn(0x7fa7880ad530 0x7fa7880af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.413+0000 7fa79effd700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7880a3d50 0x7fa7880a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa794006010 tx=0x7fa79401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.414+0000 7fa79e7fc700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] conn(0x7fa7880ad530 0x7fa7880af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.414+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== osd.9 v2:172.21.15.79:6826/4293144131 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa79800ee50 con 0x7fa7880ad530 2024-04-22T06:42:58.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 --> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7a00634d0 con 0x7fa7880ad530 2024-04-22T06:42:58.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa787fff700 1 -- 172.21.15.79:0/3563312405 <== osd.9 v2:172.21.15.79:6826/4293144131 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa7a00634d0 con 0x7fa7880ad530 2024-04-22T06:42:58.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] conn(0x7fa7880ad530 msgr2=0x7fa7880af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] conn(0x7fa7880ad530 0x7fa7880af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7880a3d50 msgr2=0x7fa7880a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fa7880a3d50 0x7fa7880a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa794006010 tx=0x7fa79401a040 comp rx=0 tx=0).stop 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 msgr2=0x7fa7a0102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 --2- 172.21.15.79:0/3563312405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa7a0108a20 0x7fa7a0102480 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa79800b970 tx=0x7fa79800bc80 comp rx=0 tx=0).stop 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa79f7fe700 1 -- 172.21.15.79:0/3563312405 reap_dead start 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 shutdown_connections 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.431+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 >> 172.21.15.79:0/3563312405 conn(0x7fa7a00788b0 msgr2=0x7fa7a00ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.432+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 shutdown_connections 2024-04-22T06:42:58.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.432+0000 7fa7a5731700 1 -- 172.21.15.79:0/3563312405 wait complete. 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 -- 172.21.15.79:0/288142723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc102ac0 msgr2=0x7f2bcc102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/288142723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc102ac0 0x7f2bcc102f40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009990 tx=0x7f2bbc009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 -- 172.21.15.79:0/288142723 shutdown_connections 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/288142723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bcc103480 0x7f2bcc10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/288142723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bcc108ab0 0x7f2bcc108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/288142723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc102ac0 0x7f2bcc102f40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 -- 172.21.15.79:0/288142723 >> 172.21.15.79:0/288142723 conn(0x7f2bcc079bf0 msgr2=0x7f2bcc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:58.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 -- 172.21.15.79:0/288142723 shutdown_connections 2024-04-22T06:42:58.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.830+0000 7f2bd38b7700 1 -- 172.21.15.79:0/288142723 wait complete. 2024-04-22T06:42:58.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 Processor -- start 2024-04-22T06:42:58.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 -- start start 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bcc102ac0 0x7f2bcc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bcc103480 0x7f2bcc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc108ab0 0x7f2bcc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc112d50 con 0x7f2bcc108ab0 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc112bd0 con 0x7f2bcc102ac0 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd38b7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc112ed0 con 0x7f2bcc103480 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd1e54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc108ab0 0x7f2bcc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd1e54700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc108ab0 0x7f2bcc0757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53146/0 (socket says 172.21.15.79:53146) 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.831+0000 7f2bd1e54700 1 -- 172.21.15.79:0/537131440 learned_addr learned my addr 172.21.15.79:0/537131440 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd0e52700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bcc103480 0x7f2bcc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1653700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bcc102ac0 0x7f2bcc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1e54700 1 -- 172.21.15.79:0/537131440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bcc103480 msgr2=0x7f2bcc075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1e54700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bcc103480 0x7f2bcc075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1e54700 1 -- 172.21.15.79:0/537131440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bcc102ac0 msgr2=0x7f2bcc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1e54700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bcc102ac0 0x7f2bcc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1e54700 1 -- 172.21.15.79:0/537131440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc1a7770 con 0x7f2bcc108ab0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd1e54700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc108ab0 0x7f2bcc0757c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400b970 tx=0x7f2bc400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4019070 con 0x7f2bcc108ab0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc1a7a60 con 0x7f2bcc108ab0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bc400cde0 con 0x7f2bcc108ab0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4007370 con 0x7f2bcc108ab0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.832+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc1a7f70 con 0x7f2bcc108ab0 2024-04-22T06:42:58.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.833+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bcc0630d0 con 0x7f2bcc108ab0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.834+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2bc401c070 con 0x7f2bcc108ab0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.834+0000 7f2bc27fc700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2bb80a3d50 0x7f2bb80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bd1653700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2bb80a3d50 0x7f2bb80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2bc40cfaa0 con 0x7f2bcc108ab0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bc27fc700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] conn(0x7f2bb80ad530 0x7f2bb80af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 --> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bc4015e80 con 0x7f2bb80ad530 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f2bc40cff80 con 0x7f2bcc108ab0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bd0e52700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] conn(0x7f2bb80ad530 0x7f2bb80af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bd1653700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2bb80a3d50 0x7f2bb80a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8006fd0 tx=0x7f2bc8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.835+0000 7f2bd0e52700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] conn(0x7f2bb80ad530 0x7f2bb80af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:58.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.836+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== osd.6 v2:172.21.15.79:6818/1077153926 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bc4015e80 con 0x7f2bb80ad530 2024-04-22T06:42:58.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.852+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 --> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bcc063550 con 0x7f2bb80ad530 2024-04-22T06:42:58.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.852+0000 7f2bc27fc700 1 -- 172.21.15.79:0/537131440 <== osd.6 v2:172.21.15.79:6818/1077153926 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bcc063550 con 0x7f2bb80ad530 2024-04-22T06:42:58.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] conn(0x7f2bb80ad530 msgr2=0x7f2bb80af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] conn(0x7f2bb80ad530 0x7f2bb80af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2bb80a3d50 msgr2=0x7f2bb80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2bb80a3d50 0x7f2bb80a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8006fd0 tx=0x7f2bc8008040 comp rx=0 tx=0).stop 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc108ab0 msgr2=0x7f2bcc0757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 --2- 172.21.15.79:0/537131440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bcc108ab0 0x7f2bcc0757c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400b970 tx=0x7f2bc400bc80 comp rx=0 tx=0).stop 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd1e54700 1 -- 172.21.15.79:0/537131440 reap_dead start 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 shutdown_connections 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 >> 172.21.15.79:0/537131440 conn(0x7f2bcc079bf0 msgr2=0x7f2bcc100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 shutdown_connections 2024-04-22T06:42:58.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:58.853+0000 7f2bd38b7700 1 -- 172.21.15.79:0/537131440 wait complete. 2024-04-22T06:42:59.838 INFO:teuthology.orchestra.run.smithi079.stdout:55834574919 2024-04-22T06:42:59.839 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T06:43:00.073 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:42:59 smithi079 ceph-mon[27946]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:43:00.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:42:59 smithi193 ceph-mon[31815]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:43:00.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:42:59 smithi190 ceph-mon[31421]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 -- 172.21.15.79:0/108377885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc102300 msgr2=0x7f8efc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 --2- 172.21.15.79:0/108377885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc102300 0x7f8efc10e7f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400b0a0 tx=0x7f8ef400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 -- 172.21.15.79:0/108377885 shutdown_connections 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 --2- 172.21.15.79:0/108377885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc102300 0x7f8efc10e7f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 --2- 172.21.15.79:0/108377885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc107990 0x7f8efc107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 --2- 172.21.15.79:0/108377885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8efc101940 0x7f8efc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 -- 172.21.15.79:0/108377885 >> 172.21.15.79:0/108377885 conn(0x7f8efc0788f0 msgr2=0x7f8efc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 -- 172.21.15.79:0/108377885 shutdown_connections 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.125+0000 7f8f01757700 1 -- 172.21.15.79:0/108377885 wait complete. 2024-04-22T06:43:00.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 Processor -- start 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 -- start start 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc101940 0x7f8efc199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8efc102300 0x7f8efc19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc107990 0x7f8efc1a5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc111b60 con 0x7f8efc107990 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc1119e0 con 0x7f8efc102300 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.126+0000 7f8f01757700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc111ce0 con 0x7f8efc101940 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc107990 0x7f8efc1a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc107990 0x7f8efc1a5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53168/0 (socket says 172.21.15.79:53168) 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 -- 172.21.15.79:0/4206138647 learned_addr learned my addr 172.21.15.79:0/4206138647 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efa7fc700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8efc102300 0x7f8efc19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efaffd700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc101940 0x7f8efc199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 -- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc101940 msgr2=0x7f8efc199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc101940 0x7f8efc199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 -- 172.21.15.79:0/4206138647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8efc102300 msgr2=0x7f8efc19a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8efc102300 0x7f8efc19a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 -- 172.21.15.79:0/4206138647 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc1a55a0 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efa7fc700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8efc102300 0x7f8efc19a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efaffd700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8efc101940 0x7f8efc199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8efb7fe700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc107990 0x7f8efc1a5060 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400b830 tx=0x7f8ef4004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4021070 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef4003c00 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8efc1a5770 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4004730 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.127+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8efc1a5b70 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.128+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8efc063110 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.129+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ef4003db0 con 0x7f8efc107990 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8ee3fff700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee40a3cd0 0x7f8ee40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8efaffd700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee40a3cd0 0x7f8ee40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8ef400f020 con 0x7f8efc107990 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8ee3fff700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] conn(0x7f8ee40ad4b0 0x7f8ee40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 --> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ef401de50 con 0x7f8ee40ad4b0 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f8ef40d21c0 con 0x7f8efc107990 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8efaffd700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee40a3cd0 0x7f8ee40a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007900 tx=0x7f8eec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.130+0000 7f8efa7fc700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] conn(0x7f8ee40ad4b0 0x7f8ee40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.131+0000 7f8efa7fc700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] conn(0x7f8ee40ad4b0 0x7f8ee40af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.132+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== osd.8 v2:172.21.15.193:6816/3840715130 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ef401de50 con 0x7f8ee40ad4b0 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.148+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 --> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8efc063590 con 0x7f8ee40ad4b0 2024-04-22T06:43:00.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8ee3fff700 1 -- 172.21.15.79:0/4206138647 <== osd.8 v2:172.21.15.193:6816/3840715130 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8efc063590 con 0x7f8ee40ad4b0 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] conn(0x7f8ee40ad4b0 msgr2=0x7f8ee40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] conn(0x7f8ee40ad4b0 0x7f8ee40af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee40a3cd0 msgr2=0x7f8ee40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee40a3cd0 0x7f8ee40a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007900 tx=0x7f8eec008040 comp rx=0 tx=0).stop 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc107990 msgr2=0x7f8efc1a5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 --2- 172.21.15.79:0/4206138647 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8efc107990 0x7f8efc1a5060 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400b830 tx=0x7f8ef4004d90 comp rx=0 tx=0).stop 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8efb7fe700 1 -- 172.21.15.79:0/4206138647 reap_dead start 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 shutdown_connections 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 >> 172.21.15.79:0/4206138647 conn(0x7f8efc0788f0 msgr2=0x7f8efc0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 shutdown_connections 2024-04-22T06:43:00.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:42:59.149+0000 7f8f01757700 1 -- 172.21.15.79:0/4206138647 wait complete. 2024-04-22T06:43:00.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 -- 172.21.15.79:0/870364488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 msgr2=0x7f939810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 --2- 172.21.15.79:0/870364488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939810e7f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f939000b0a0 tx=0x7f939000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 -- 172.21.15.79:0/870364488 shutdown_connections 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 --2- 172.21.15.79:0/870364488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939810e7f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 --2- 172.21.15.79:0/870364488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9398107990 0x7f9398107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 --2- 172.21.15.79:0/870364488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9398101940 0x7f9398101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 -- 172.21.15.79:0/870364488 >> 172.21.15.79:0/870364488 conn(0x7f93980788f0 msgr2=0x7f9398078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 -- 172.21.15.79:0/870364488 shutdown_connections 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.817+0000 7f939ec4c700 1 -- 172.21.15.79:0/870364488 wait complete. 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 Processor -- start 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 -- start start 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9398101940 0x7f9398199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9398107990 0x7f93981a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9398111ae0 con 0x7f9398102300 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9398111960 con 0x7f9398107990 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939ec4c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9398111c60 con 0x7f9398101940 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53188/0 (socket says 172.21.15.79:53188) 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 -- 172.21.15.79:0/1870768549 learned_addr learned my addr 172.21.15.79:0/1870768549 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939d1e9700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9398107990 0x7f93981a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939c9e8700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9398101940 0x7f9398199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 -- 172.21.15.79:0/1870768549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9398101940 msgr2=0x7f9398199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9398101940 0x7f9398199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 -- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9398107990 msgr2=0x7f93981a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9398107990 0x7f93981a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 -- 172.21.15.79:0/1870768549 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93981a53c0 con 0x7f9398102300 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f939c9e8700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9398101940 0x7f9398199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.818+0000 7f9397fff700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939819a340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f938400c9b0 tx=0x7f938400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:00.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.819+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384007b00 con 0x7f9398102300 2024-04-22T06:43:00.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.819+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9384007ca0 con 0x7f9398102300 2024-04-22T06:43:00.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.819+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93981a55f0 con 0x7f9398102300 2024-04-22T06:43:00.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.819+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384018730 con 0x7f9398102300 2024-04-22T06:43:00.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.819+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93981a5b00 con 0x7f9398102300 2024-04-22T06:43:00.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.819+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9398063110 con 0x7f9398102300 2024-04-22T06:43:00.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.821+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f938400f450 con 0x7f9398102300 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.821+0000 7f9395ffb700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93880a3cd0 0x7f93880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.821+0000 7f939c9e8700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93880a3cd0 0x7f93880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f93840d08b0 con 0x7f9398102300 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f9395ffb700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] conn(0x7f93880ad4b0 0x7f93880af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 --> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9384004500 con 0x7f93880ad4b0 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f93840d0d90 con 0x7f9398102300 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f939c9e8700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93880a3cd0 0x7f93880a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f938c00baa0 tx=0x7f938c005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f939d1e9700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] conn(0x7f93880ad4b0 0x7f93880af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.822+0000 7f939d1e9700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] conn(0x7f93880ad4b0 0x7f93880af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:00.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.823+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== osd.7 v2:172.21.15.190:6816/2271031515 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9384004500 con 0x7f93880ad4b0 2024-04-22T06:43:00.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.839+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 --> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9398063590 con 0x7f93880ad4b0 2024-04-22T06:43:00.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.839+0000 7f9395ffb700 1 -- 172.21.15.79:0/1870768549 <== osd.7 v2:172.21.15.190:6816/2271031515 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9398063590 con 0x7f93880ad4b0 2024-04-22T06:43:00.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] conn(0x7f93880ad4b0 msgr2=0x7f93880af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] conn(0x7f93880ad4b0 0x7f93880af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93880a3cd0 msgr2=0x7f93880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f93880a3cd0 0x7f93880a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f938c00baa0 tx=0x7f938c005db0 comp rx=0 tx=0).stop 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 msgr2=0x7f939819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 --2- 172.21.15.79:0/1870768549 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9398102300 0x7f939819a340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f938400c9b0 tx=0x7f938400cd70 comp rx=0 tx=0).stop 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939d1e9700 1 -- 172.21.15.79:0/1870768549 reap_dead start 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 shutdown_connections 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 >> 172.21.15.79:0/1870768549 conn(0x7f93980788f0 msgr2=0x7f93980ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 shutdown_connections 2024-04-22T06:43:00.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:00.840+0000 7f939ec4c700 1 -- 172.21.15.79:0/1870768549 wait complete. 2024-04-22T06:43:01.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3907411652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4108a50 msgr2=0x7f4fc4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:01.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3907411652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4108a50 0x7f4fc4108e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009990 tx=0x7f4fc0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3907411652 shutdown_connections 2024-04-22T06:43:01.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3907411652 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc4103450 0x7f4fc410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3907411652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4108a50 0x7f4fc4108e30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3907411652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc4102a90 0x7f4fc4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3907411652 >> 172.21.15.79:0/3907411652 conn(0x7f4fc4079ba0 msgr2=0x7f4fc4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:01.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3907411652 shutdown_connections 2024-04-22T06:43:01.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.431+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3907411652 wait complete. 2024-04-22T06:43:01.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 Processor -- start 2024-04-22T06:43:01.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 -- start start 2024-04-22T06:43:01.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc4102a90 0x7f4fc4199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:01.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4103450 0x7f4fc41996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:01.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc4108a50 0x7f4fc419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:01.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4112c60 con 0x7f4fc4103450 2024-04-22T06:43:01.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4112ae0 con 0x7f4fc4108a50 2024-04-22T06:43:01.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.432+0000 7f4fcbc1d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4112de0 con 0x7f4fc4102a90 2024-04-22T06:43:01.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4103450 0x7f4fc41996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:01.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4103450 0x7f4fc41996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53206/0 (socket says 172.21.15.79:53206) 2024-04-22T06:43:01.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 -- 172.21.15.79:0/3304525676 learned_addr learned my addr 172.21.15.79:0/3304525676 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:01.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fca1ba700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc4108a50 0x7f4fc419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:01.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc99b9700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc4102a90 0x7f4fc4199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:01.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 -- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc4102a90 msgr2=0x7f4fc4199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:01.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc4102a90 0x7f4fc4199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 -- 172.21.15.79:0/3304525676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc4108a50 msgr2=0x7f4fc419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:01.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fc4108a50 0x7f4fc419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 -- 172.21.15.79:0/3304525676 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc419e1a0 con 0x7f4fc4103450 2024-04-22T06:43:01.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fc91b8700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4103450 0x7f4fc41996d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400b530 tx=0x7f4fb400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:01.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4015020 con 0x7f4fc4103450 2024-04-22T06:43:01.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fb4011ce0 con 0x7f4fc4103450 2024-04-22T06:43:01.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc419e490 con 0x7f4fc4103450 2024-04-22T06:43:01.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4010e40 con 0x7f4fc4103450 2024-04-22T06:43:01.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.433+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc41a5d10 con 0x7f4fc4103450 2024-04-22T06:43:01.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.434+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4fc40631d0 con 0x7f4fc4103450 2024-04-22T06:43:01.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.435+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4fb4018020 con 0x7f4fc4103450 2024-04-22T06:43:01.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fbaffd700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fb00a3cd0 0x7f4fb00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:01.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fc99b9700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fb00a3cd0 0x7f4fb00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:01.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4fb40d0290 con 0x7f4fc4103450 2024-04-22T06:43:01.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fbaffd700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] conn(0x7f4fb00ad4b0 0x7f4fb00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:01.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 --> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fb00aff80 con 0x7f4fb00ad4b0 2024-04-22T06:43:01.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4fb40d0770 con 0x7f4fc4103450 2024-04-22T06:43:01.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.436+0000 7f4fc99b9700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fb00a3cd0 0x7f4fb00a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0006010 tx=0x7f4fc001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:01.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.437+0000 7f4fca1ba700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] conn(0x7f4fb00ad4b0 0x7f4fb00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:01.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.437+0000 7f4fca1ba700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] conn(0x7f4fb00ad4b0 0x7f4fb00af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:01.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.438+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== osd.11 v2:172.21.15.193:6824/3747742483 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fb00aff80 con 0x7f4fb00ad4b0 2024-04-22T06:43:01.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.454+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 --> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4fc40635d0 con 0x7f4fb00ad4b0 2024-04-22T06:43:01.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.454+0000 7f4fbaffd700 1 -- 172.21.15.79:0/3304525676 <== osd.11 v2:172.21.15.193:6824/3747742483 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4fc40635d0 con 0x7f4fb00ad4b0 2024-04-22T06:43:01.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] conn(0x7f4fb00ad4b0 msgr2=0x7f4fb00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:01.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] conn(0x7f4fb00ad4b0 0x7f4fb00af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:01.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fb00a3cd0 msgr2=0x7f4fb00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:01.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fb00a3cd0 0x7f4fb00a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0006010 tx=0x7f4fc001a040 comp rx=0 tx=0).stop 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4103450 msgr2=0x7f4fc41996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 --2- 172.21.15.79:0/3304525676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc4103450 0x7f4fc41996d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400b530 tx=0x7f4fb400b840 comp rx=0 tx=0).stop 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fca1ba700 1 -- 172.21.15.79:0/3304525676 reap_dead start 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 shutdown_connections 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 >> 172.21.15.79:0/3304525676 conn(0x7f4fc4079ba0 msgr2=0x7f4fc4101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 shutdown_connections 2024-04-22T06:43:01.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:01.455+0000 7f4fcbc1d700 1 -- 172.21.15.79:0/3304525676 wait complete. 2024-04-22T06:43:01.976 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:01 smithi079 ceph-mon[27946]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T06:43:02.068 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 -- 172.21.15.79:0/917428991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006afe0 msgr2=0x7f27c006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 --2- 172.21.15.79:0/917428991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006afe0 0x7f27c006b440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009990 tx=0x7f27b4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 -- 172.21.15.79:0/917428991 shutdown_connections 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 --2- 172.21.15.79:0/917428991 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c006b980 0x7f27c010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 --2- 172.21.15.79:0/917428991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0108a20 0x7f27c0108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 --2- 172.21.15.79:0/917428991 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006afe0 0x7f27c006b440 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 -- 172.21.15.79:0/917428991 >> 172.21.15.79:0/917428991 conn(0x7f27c00788b0 msgr2=0x7f27c0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 -- 172.21.15.79:0/917428991 shutdown_connections 2024-04-22T06:43:02.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.080+0000 7f27c498f700 1 -- 172.21.15.79:0/917428991 wait complete. 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.081+0000 7f27c498f700 1 Processor -- start 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.081+0000 7f27c498f700 1 -- start start 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.081+0000 7f27c498f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c006afe0 0x7f27c0199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.081+0000 7f27c498f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006b980 0x7f27c0199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.081+0000 7f27c498f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0108a20 0x7f27c019d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27c498f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f27c0112c20 con 0x7f27c006b980 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27c498f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27c0112aa0 con 0x7f27c006afe0 2024-04-22T06:43:02.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27c498f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f27c0112da0 con 0x7f27c0108a20 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006b980 0x7f27c0199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006b980 0x7f27c0199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53236/0 (socket says 172.21.15.79:53236) 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 -- 172.21.15.79:0/3917887625 learned_addr learned my addr 172.21.15.79:0/3917887625 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bed9d700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0108a20 0x7f27c019d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27be59c700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c006afe0 0x7f27c0199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 -- 172.21.15.79:0/3917887625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0108a20 msgr2=0x7f27c019d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0108a20 0x7f27c019d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 -- 172.21.15.79:0/3917887625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c006afe0 msgr2=0x7f27c0199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c006afe0 0x7f27c0199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 -- 172.21.15.79:0/3917887625 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c019e110 con 0x7f27c006b980 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bed9d700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0108a20 0x7f27c019d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27be59c700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27c006afe0 0x7f27c0199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:02.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.082+0000 7f27bdd9b700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006b980 0x7f27c0199690 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f27b4000c00 tx=0x7f27b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.083+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4027070 con 0x7f27c006b980 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.083+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c019e3a0 con 0x7f27c006b980 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.083+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27b40041d0 con 0x7f27c006b980 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.083+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c01a5c80 con 0x7f27c006b980 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.083+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b400f460 con 0x7f27c006b980 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.084+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27c00631d0 con 0x7f27c006b980 2024-04-22T06:43:02.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.085+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f27b402a070 con 0x7f27c006b980 2024-04-22T06:43:02.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.085+0000 7f27af7fe700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f27a80a3d50 0x7f27a80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:02.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f27b40d1a00 con 0x7f27c006b980 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27be59c700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f27a80a3d50 0x7f27a80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27af7fe700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] conn(0x7f27a80ad530 0x7f27a80af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 --> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27b4009030 con 0x7f27a80ad530 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f27b40d1ee0 con 0x7f27c006b980 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27bed9d700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] conn(0x7f27a80ad530 0x7f27a80af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27be59c700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f27a80a3d50 0x7f27a80a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f27b0007900 tx=0x7f27b0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.086+0000 7f27bed9d700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] conn(0x7f27a80ad530 0x7f27a80af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:02.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.087+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== osd.0 v2:172.21.15.79:6802/2633795155 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27b4009030 con 0x7f27a80ad530 2024-04-22T06:43:02.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.109+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 --> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f27c00635d0 con 0x7f27a80ad530 2024-04-22T06:43:02.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.109+0000 7f27af7fe700 1 -- 172.21.15.79:0/3917887625 <== osd.0 v2:172.21.15.79:6802/2633795155 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f27c00635d0 con 0x7f27a80ad530 2024-04-22T06:43:02.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.109+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] conn(0x7f27a80ad530 msgr2=0x7f27a80af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:02.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.109+0000 7f27c498f700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] conn(0x7f27a80ad530 0x7f27a80af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f27a80a3d50 msgr2=0x7f27a80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f27a80a3d50 0x7f27a80a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f27b0007900 tx=0x7f27b0008040 comp rx=0 tx=0).stop 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006b980 msgr2=0x7f27c0199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 --2- 172.21.15.79:0/3917887625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f27c006b980 0x7f27c0199690 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f27b4000c00 tx=0x7f27b40047b0 comp rx=0 tx=0).stop 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27bed9d700 1 -- 172.21.15.79:0/3917887625 reap_dead start 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 shutdown_connections 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 >> 172.21.15.79:0/3917887625 conn(0x7f27c00788b0 msgr2=0x7f27c00ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 shutdown_connections 2024-04-22T06:43:02.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:02.110+0000 7f27c498f700 1 -- 172.21.15.79:0/3917887625 wait complete. 2024-04-22T06:43:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:01 smithi193 ceph-mon[31815]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T06:43:02.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:01 smithi190 ceph-mon[31421]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi079.stdout:343597383701 2024-04-22T06:43:03.353 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-22T06:43:03.949 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:03 smithi079 ceph-mon[27946]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:03.975 INFO:teuthology.orchestra.run.smithi079.stdout:253403070501 2024-04-22T06:43:03.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-22T06:43:04.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:03 smithi193 ceph-mon[31815]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:04.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:03 smithi190 ceph-mon[31421]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:04.649 INFO:teuthology.orchestra.run.smithi079.stdout:141733920824 2024-04-22T06:43:04.649 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 -- 172.21.15.79:0/1178843740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380410f4e0 msgr2=0x7f38041118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 --2- 172.21.15.79:0/1178843740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380410f4e0 0x7f38041118d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00b0a0 tx=0x7f37fc00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 -- 172.21.15.79:0/1178843740 shutdown_connections 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 --2- 172.21.15.79:0/1178843740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380410f4e0 0x7f38041118d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 --2- 172.21.15.79:0/1178843740 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f380406afe0 0x7f380406b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 --2- 172.21.15.79:0/1178843740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f380406b900 0x7f380410efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 -- 172.21.15.79:0/1178843740 >> 172.21.15.79:0/1178843740 conn(0x7f38040788b0 msgr2=0x7f3804078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:04.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 -- 172.21.15.79:0/1178843740 shutdown_connections 2024-04-22T06:43:04.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.812+0000 7f380c767700 1 -- 172.21.15.79:0/1178843740 wait complete. 2024-04-22T06:43:04.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 Processor -- start 2024-04-22T06:43:04.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 -- start start 2024-04-22T06:43:04.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380406afe0 0x7f380419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f380406b900 0x7f380419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f380410f4e0 0x7f380419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3804114e60 con 0x7f380406afe0 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3804114ce0 con 0x7f380410f4e0 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380c767700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3804114fe0 con 0x7f380406b900 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.813+0000 7f380a503700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380406afe0 0x7f380419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380406afe0 0x7f380419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47454/0 (socket says 172.21.15.79:47454) 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 -- 172.21.15.79:0/3669879754 learned_addr learned my addr 172.21.15.79:0/3669879754 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f3809d02700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f380406b900 0x7f380419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380ad04700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f380410f4e0 0x7f380419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 -- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f380406b900 msgr2=0x7f380419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f380406b900 0x7f380419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 -- 172.21.15.79:0/3669879754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f380410f4e0 msgr2=0x7f380419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f380410f4e0 0x7f380419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 -- 172.21.15.79:0/3669879754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38041a02d0 con 0x7f380406afe0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f3809d02700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f380406b900 0x7f380419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380a503700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380406afe0 0x7f380419b310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f380000c9b0 tx=0x7f380000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800007b00 con 0x7f380406afe0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38041a05c0 con 0x7f380406afe0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.814+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3800007ca0 con 0x7f380406afe0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.815+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800018730 con 0x7f380406afe0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.815+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38041a7e40 con 0x7f380406afe0 2024-04-22T06:43:04.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.816+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3804063150 con 0x7f380406afe0 2024-04-22T06:43:04.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.817+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f380000f450 con 0x7f380406afe0 2024-04-22T06:43:04.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f37fb7fe700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f37f00a3cd0 0x7f37f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:04.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f3809d02700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f37f00a3cd0 0x7f37f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:04.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f38000d0440 con 0x7f380406afe0 2024-04-22T06:43:04.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f37fb7fe700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] conn(0x7f37f00ad4b0 0x7f37f00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:04.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 --> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3800012ec0 con 0x7f37f00ad4b0 2024-04-22T06:43:04.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f38000d7050 con 0x7f380406afe0 2024-04-22T06:43:04.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.818+0000 7f3809d02700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f37f00a3cd0 0x7f37f00a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3804102db0 tx=0x7f37f4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:04.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.819+0000 7f380ad04700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] conn(0x7f37f00ad4b0 0x7f37f00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:04.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.819+0000 7f380ad04700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] conn(0x7f37f00ad4b0 0x7f37f00af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:04.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.820+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== osd.2 v2:172.21.15.193:6800/4086025232 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3800012ec0 con 0x7f37f00ad4b0 2024-04-22T06:43:04.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.837+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 --> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3804063580 con 0x7f37f00ad4b0 2024-04-22T06:43:04.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f37fb7fe700 1 -- 172.21.15.79:0/3669879754 <== osd.2 v2:172.21.15.193:6800/4086025232 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3804063580 con 0x7f37f00ad4b0 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] conn(0x7f37f00ad4b0 msgr2=0x7f37f00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380c767700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] conn(0x7f37f00ad4b0 0x7f37f00af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f37f00a3cd0 msgr2=0x7f37f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380c767700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f37f00a3cd0 0x7f37f00a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3804102db0 tx=0x7f37f4008040 comp rx=0 tx=0).stop 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380406afe0 msgr2=0x7f380419b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380c767700 1 --2- 172.21.15.79:0/3669879754 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f380406afe0 0x7f380419b310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f380000c9b0 tx=0x7f380000cd70 comp rx=0 tx=0).stop 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.838+0000 7f380ad04700 1 -- 172.21.15.79:0/3669879754 reap_dead start 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.839+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 shutdown_connections 2024-04-22T06:43:04.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.839+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 >> 172.21.15.79:0/3669879754 conn(0x7f38040788b0 msgr2=0x7f3804110930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:04.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.839+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 shutdown_connections 2024-04-22T06:43:04.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:04.839+0000 7f380c767700 1 -- 172.21.15.79:0/3669879754 wait complete. 2024-04-22T06:43:05.247 INFO:teuthology.orchestra.run.smithi079.stdout:197568495663 2024-04-22T06:43:05.248 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T06:43:05.627 INFO:teuthology.orchestra.run.smithi079.stdout:219043332137 2024-04-22T06:43:05.627 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T06:43:05.899 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:05 smithi079 ceph-mon[27946]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:05 smithi193 ceph-mon[31815]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:06.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:05 smithi190 ceph-mon[31421]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:06.328 INFO:teuthology.orchestra.run.smithi079.stdout:34359738445 2024-04-22T06:43:06.329 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T06:43:07.184 INFO:teuthology.orchestra.run.smithi079.stdout:287762808863 2024-04-22T06:43:07.184 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-22T06:43:07.789 INFO:teuthology.orchestra.run.smithi079.stdout:77309411396 2024-04-22T06:43:07.789 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T06:43:08.107 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:07 smithi079 ceph-mon[27946]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:08.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:07 smithi193 ceph-mon[31815]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:08.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:07 smithi190 ceph-mon[31421]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 -- 172.21.15.79:0/1824210370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 msgr2=0x7f452410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 --2- 172.21.15.79:0/1824210370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f452410dfc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f451c00b0d0 tx=0x7f451c00b3e0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 -- 172.21.15.79:0/1824210370 shutdown_connections 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 --2- 172.21.15.79:0/1824210370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f452410dfc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 --2- 172.21.15.79:0/1824210370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4524101360 0x7f4524101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 --2- 172.21.15.79:0/1824210370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101d10 0x7f4524105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 -- 172.21.15.79:0/1824210370 >> 172.21.15.79:0/1824210370 conn(0x7f4524077fb0 msgr2=0x7f45240ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 -- 172.21.15.79:0/1824210370 shutdown_connections 2024-04-22T06:43:08.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.375+0000 7f452ad26700 1 -- 172.21.15.79:0/1824210370 wait complete. 2024-04-22T06:43:08.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 Processor -- start 2024-04-22T06:43:08.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 -- start start 2024-04-22T06:43:08.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4524101360 0x7f4524199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:08.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101d10 0x7f452419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f45241a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f45241112b0 con 0x7f45241063f0 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111130 con 0x7f4524101360 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f452ad26700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111430 con 0x7f4524101d10 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f45292c3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f45241a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f45292c3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f45241a4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47472/0 (socket says 172.21.15.79:47472) 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.376+0000 7f45292c3700 1 -- 172.21.15.79:0/3430270488 learned_addr learned my addr 172.21.15.79:0/3430270488 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4528ac2700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4524101360 0x7f4524199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4523fff700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101d10 0x7f452419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f45292c3700 1 -- 172.21.15.79:0/3430270488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101d10 msgr2=0x7f452419a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f45292c3700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101d10 0x7f452419a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f45292c3700 1 -- 172.21.15.79:0/3430270488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4524101360 msgr2=0x7f4524199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:08.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f45292c3700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4524101360 0x7f4524199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f45292c3700 1 -- 172.21.15.79:0/3430270488 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45241a54d0 con 0x7f45241063f0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4528ac2700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4524101360 0x7f4524199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4523fff700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101d10 0x7f452419a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f45292c3700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f45241a4f90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f451c009fd0 tx=0x7f451c007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c01f070 con 0x7f45241063f0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45241a56a0 con 0x7f45241063f0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f451c003c00 con 0x7f45241063f0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c01b8e0 con 0x7f45241063f0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.377+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241a5b50 con 0x7f45241063f0 2024-04-22T06:43:08.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.378+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4524104480 con 0x7f45241063f0 2024-04-22T06:43:08.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.380+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f451c022070 con 0x7f45241063f0 2024-04-22T06:43:08.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4521ffb700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45140a3cd0 0x7f45140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:08.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f451c00f020 con 0x7f45241063f0 2024-04-22T06:43:08.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4528ac2700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45140a3cd0 0x7f45140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:08.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4521ffb700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] conn(0x7f45140ad4b0 0x7f45140af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:08.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 --> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f451c0020b0 con 0x7f45140ad4b0 2024-04-22T06:43:08.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f451c0d1da0 con 0x7f45241063f0 2024-04-22T06:43:08.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4523fff700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] conn(0x7f45140ad4b0 0x7f45140af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:08.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4528ac2700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45140a3cd0 0x7f45140a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4518005fd0 tx=0x7f4518005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:08.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.381+0000 7f4523fff700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] conn(0x7f45140ad4b0 0x7f45140af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:08.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.383+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== osd.3 v2:172.21.15.79:6810/433108201 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f451c0020b0 con 0x7f45140ad4b0 2024-04-22T06:43:08.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.399+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 --> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f45241027f0 con 0x7f45140ad4b0 2024-04-22T06:43:08.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.399+0000 7f4521ffb700 1 -- 172.21.15.79:0/3430270488 <== osd.3 v2:172.21.15.79:6810/433108201 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f45241027f0 con 0x7f45140ad4b0 2024-04-22T06:43:08.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.399+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] conn(0x7f45140ad4b0 msgr2=0x7f45140af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:08.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] conn(0x7f45140ad4b0 0x7f45140af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45140a3cd0 msgr2=0x7f45140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f45140a3cd0 0x7f45140a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4518005fd0 tx=0x7f4518005dc0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 msgr2=0x7f45241a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 --2- 172.21.15.79:0/3430270488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45241063f0 0x7f45241a4f90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f451c009fd0 tx=0x7f451c007ad0 comp rx=0 tx=0).stop 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f45292c3700 1 -- 172.21.15.79:0/3430270488 reap_dead start 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 shutdown_connections 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 >> 172.21.15.79:0/3430270488 conn(0x7f4524077fb0 msgr2=0x7f4524103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 shutdown_connections 2024-04-22T06:43:08.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:08.400+0000 7f452ad26700 1 -- 172.21.15.79:0/3430270488 wait complete. 2024-04-22T06:43:09.090 INFO:teuthology.orchestra.run.smithi079.stdout:107374182463 2024-04-22T06:43:09.090 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T06:43:09.586 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:09.591 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:09.591 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:09.594 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:09.670 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:09.995 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:10.103 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:09 smithi079 ceph-mon[27946]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:09 smithi193 ceph-mon[31815]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:10.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:09 smithi190 ceph-mon[31421]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:11.226 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:11.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:11 smithi079 ceph-mon[27946]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:12.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:11 smithi193 ceph-mon[31815]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:12.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:11 smithi190 ceph-mon[31421]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:13.225 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:13.225 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:13.953 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:13 smithi079 ceph-mon[27946]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-22T06:43:14.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:13 smithi193 ceph-mon[31815]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-22T06:43:14.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:13 smithi190 ceph-mon[31421]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-22T06:43:15.764 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:15.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 -- 172.21.15.79:0/977474107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11281079c0 msgr2=0x7f1128107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:15.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 --2- 172.21.15.79:0/977474107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11281079c0 0x7f1128107da0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f111c009990 tx=0x7f111c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 -- 172.21.15.79:0/977474107 shutdown_connections 2024-04-22T06:43:15.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 --2- 172.21.15.79:0/977474107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1128102330 0x7f112810e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 --2- 172.21.15.79:0/977474107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f11281079c0 0x7f1128107da0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 --2- 172.21.15.79:0/977474107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1128101970 0x7f1128101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.766+0000 7f112ef52700 1 -- 172.21.15.79:0/977474107 >> 172.21.15.79:0/977474107 conn(0x7f1128078980 msgr2=0x7f1128078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 -- 172.21.15.79:0/977474107 shutdown_connections 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 -- 172.21.15.79:0/977474107 wait complete. 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 Processor -- start 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 -- start start 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 0x7f112810efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1128102330 0x7f112810f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11281079c0 0x7f112810fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1128112c60 con 0x7f1128101970 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1128112ae0 con 0x7f1128102330 2024-04-22T06:43:15.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ef52700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1128112de0 con 0x7f11281079c0 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.767+0000 7f112ccee700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 0x7f112810efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 0x7f112810efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55338/0 (socket says 172.21.15.79:55338) 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 -- 172.21.15.79:0/2722580053 learned_addr learned my addr 172.21.15.79:0/2722580053 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112d4ef700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11281079c0 0x7f112810fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f1127fff700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1128102330 0x7f112810f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 -- 172.21.15.79:0/2722580053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11281079c0 msgr2=0x7f112810fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11281079c0 0x7f112810fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 -- 172.21.15.79:0/2722580053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1128102330 msgr2=0x7f112810f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1128102330 0x7f112810f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 -- 172.21.15.79:0/2722580053 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11281a7580 con 0x7f1128101970 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ccee700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 0x7f112810efb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f111c0057f0 tx=0x7f111c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c027070 con 0x7f1128101970 2024-04-22T06:43:15.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f111c0040a0 con 0x7f1128101970 2024-04-22T06:43:15.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11281a7810 con 0x7f1128101970 2024-04-22T06:43:15.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.768+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c022e20 con 0x7f1128101970 2024-04-22T06:43:15.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.769+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11281a7cf0 con 0x7f1128101970 2024-04-22T06:43:15.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.770+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1128103090 con 0x7f1128101970 2024-04-22T06:43:15.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.771+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f111c02a070 con 0x7f1128101970 2024-04-22T06:43:15.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.771+0000 7f1125ffb700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f11180a3cd0 0x7f11180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:15.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.772+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f111c0d1100 con 0x7f1128101970 2024-04-22T06:43:15.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.772+0000 7f1127fff700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f11180a3cd0 0x7f11180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:15.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.772+0000 7f1127fff700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f11180a3cd0 0x7f11180a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1114005ea0 tx=0x7f1114005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:15.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.780+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f111c013070 con 0x7f1128101970 2024-04-22T06:43:15.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.985+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f112802df80 con 0x7f1128101970 2024-04-22T06:43:15.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.986+0000 7f1125ffb700 1 -- 172.21.15.79:0/2722580053 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f111c09dbc0 con 0x7f1128101970 2024-04-22T06:43:15.985 INFO:teuthology.orchestra.run.smithi079.stdout:343597383704 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f11180a3cd0 msgr2=0x7f11180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f11180a3cd0 0x7f11180a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1114005ea0 tx=0x7f1114005e30 comp rx=0 tx=0).stop 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 msgr2=0x7f112810efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 0x7f112810efb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f111c0057f0 tx=0x7f111c003680 comp rx=0 tx=0).stop 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 shutdown_connections 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11281079c0 0x7f112810fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f11180a3cd0 0x7f11180a6190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1128101970 0x7f112810efb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 --2- 172.21.15.79:0/2722580053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1128102330 0x7f112810f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 >> 172.21.15.79:0/2722580053 conn(0x7f1128078980 msgr2=0x7f11280ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 shutdown_connections 2024-04-22T06:43:15.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:15.988+0000 7f112ef52700 1 -- 172.21.15.79:0/2722580053 wait complete. 2024-04-22T06:43:15.997 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:15 smithi079 ceph-mon[27946]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:15 smithi193 ceph-mon[31815]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:16.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:15 smithi190 ceph-mon[31421]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:16.669 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383701 got 343597383704 for osd.11 2024-04-22T06:43:16.669 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2722580053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:43:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-22T06:43:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-22T06:43:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-04-22T06:43:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-22T06:43:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-22T06:43:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-22T06:43:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]: dispatch 2024-04-22T06:43:16.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:16 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-22T06:43:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2722580053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:43:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]: dispatch 2024-04-22T06:43:17.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:16 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-22T06:43:17.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2722580053' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]: dispatch 2024-04-22T06:43:17.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:16 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-22T06:43:17.932 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-22T06:43:17.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:17 smithi079 ceph-mon[27946]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:43:18.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-22T06:43:18.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:17 smithi193 ceph-mon[31815]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-22T06:43:18.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:17 smithi190 ceph-mon[31421]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:43:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.550+0000 7fe82826b700 1 -- 172.21.15.79:0/3890721518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820103720 msgr2=0x7fe82010f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.550+0000 7fe82826b700 1 --2- 172.21.15.79:0/3890721518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820103720 0x7fe82010f990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe81800b0a0 tx=0x7fe81800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 -- 172.21.15.79:0/3890721518 shutdown_connections 2024-04-22T06:43:18.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 --2- 172.21.15.79:0/3890721518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820103720 0x7fe82010f990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 --2- 172.21.15.79:0/3890721518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 0x7fe820108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 --2- 172.21.15.79:0/3890721518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820102d60 0x7fe8201031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 -- 172.21.15.79:0/3890721518 >> 172.21.15.79:0/3890721518 conn(0x7fe82007a9d0 msgr2=0x7fe82007ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 -- 172.21.15.79:0/3890721518 shutdown_connections 2024-04-22T06:43:18.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 -- 172.21.15.79:0/3890721518 wait complete. 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 Processor -- start 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.551+0000 7fe82826b700 1 -- start start 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 0x7fe820199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820103720 0x7fe820199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 0x7fe82019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe820112d20 con 0x7fe820102d60 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe820112ba0 con 0x7fe820103720 2024-04-22T06:43:18.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe820112ea0 con 0x7fe820108bc0 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 0x7fe820199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 0x7fe820199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55356/0 (socket says 172.21.15.79:55356) 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 -- 172.21.15.79:0/1090571298 learned_addr learned my addr 172.21.15.79:0/1090571298 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe825806700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820103720 0x7fe820199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826808700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 0x7fe82019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 -- 172.21.15.79:0/1090571298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 msgr2=0x7fe82019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 0x7fe82019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 -- 172.21.15.79:0/1090571298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820103720 msgr2=0x7fe820199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820103720 0x7fe820199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 -- 172.21.15.79:0/1090571298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82019e140 con 0x7fe820102d60 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe825806700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820103720 0x7fe820199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826808700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 0x7fe82019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe826007700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 0x7fe820199130 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00d980 tx=0x7fe81c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:18.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c017070 con 0x7fe820102d60 2024-04-22T06:43:18.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.552+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82019e430 con 0x7fe820102d60 2024-04-22T06:43:18.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.553+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe81c010460 con 0x7fe820102d60 2024-04-22T06:43:18.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.553+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c00de60 con 0x7fe820102d60 2024-04-22T06:43:18.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.553+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8201a5cb0 con 0x7fe820102d60 2024-04-22T06:43:18.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.554+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe820069730 con 0x7fe820102d60 2024-04-22T06:43:18.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.555+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe81c009b40 con 0x7fe820102d60 2024-04-22T06:43:18.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.555+0000 7fe8177fe700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.555+0000 7fe825806700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.555+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fe81c0d00a0 con 0x7fe820102d60 2024-04-22T06:43:18.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.556+0000 7fe825806700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe82007b020 tx=0x7fe810005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:18.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.560+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe81c09c930 con 0x7fe820102d60 2024-04-22T06:43:18.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.766+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe82019a490 con 0x7fe820102d60 2024-04-22T06:43:18.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.766+0000 7fe8177fe700 1 -- 172.21.15.79:0/1090571298 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe81c09c230 con 0x7fe820102d60 2024-04-22T06:43:18.765 INFO:teuthology.orchestra.run.smithi079.stdout:219043332140 2024-04-22T06:43:18.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe80c0a3cd0 msgr2=0x7fe80c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe82007b020 tx=0x7fe810005f50 comp rx=0 tx=0).stop 2024-04-22T06:43:18.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 msgr2=0x7fe820199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 0x7fe820199130 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00d980 tx=0x7fe81c00dd40 comp rx=0 tx=0).stop 2024-04-22T06:43:18.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 shutdown_connections 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe820108bc0 0x7fe82019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe820102d60 0x7fe820199130 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 --2- 172.21.15.79:0/1090571298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe820103720 0x7fe820199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.768+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 >> 172.21.15.79:0/1090571298 conn(0x7fe82007a9d0 msgr2=0x7fe8201019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 shutdown_connections 2024-04-22T06:43:18.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7fe82826b700 1 -- 172.21.15.79:0/1090571298 wait complete. 2024-04-22T06:43:18.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 -- 172.21.15.79:0/1949914312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 msgr2=0x7ff7d4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/1949914312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4102f10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009990 tx=0x7ff7c8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 -- 172.21.15.79:0/1949914312 shutdown_connections 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/1949914312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d4103450 0x7ff7d410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/1949914312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 0x7ff7d4108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/1949914312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4102f10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 -- 172.21.15.79:0/1949914312 >> 172.21.15.79:0/1949914312 conn(0x7ff7d4079bf0 msgr2=0x7ff7d407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 -- 172.21.15.79:0/1949914312 shutdown_connections 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.769+0000 7ff7da8fb700 1 -- 172.21.15.79:0/1949914312 wait complete. 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.770+0000 7ff7da8fb700 1 Processor -- start 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.770+0000 7ff7da8fb700 1 -- start start 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7da8fb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7da8fb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d4103450 0x7ff7d4199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7da8fb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 0x7ff7d419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7da8fb700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4112cb0 con 0x7ff7d4102a90 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7da8fb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4112b30 con 0x7ff7d4108a80 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7da8fb700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4112e30 con 0x7ff7d4103450 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55370/0 (socket says 172.21.15.79:55370) 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 -- 172.21.15.79:0/3087726323 learned_addr learned my addr 172.21.15.79:0/3087726323 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d37fe700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d4103450 0x7ff7d4199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d8e98700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 0x7ff7d419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 -- 172.21.15.79:0/3087726323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d4103450 msgr2=0x7ff7d4199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d4103450 0x7ff7d4199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 -- 172.21.15.79:0/3087726323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 msgr2=0x7ff7d419d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 0x7ff7d419d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 -- 172.21.15.79:0/3087726323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d419e100 con 0x7ff7d4102a90 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d8e98700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 0x7ff7d419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:18.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d3fff700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4199140 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c9b0 tx=0x7ff7c400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:18.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.771+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4007b00 con 0x7ff7d4102a90 2024-04-22T06:43:18.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.772+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c4007ca0 con 0x7ff7d4102a90 2024-04-22T06:43:18.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.772+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4018730 con 0x7ff7d4102a90 2024-04-22T06:43:18.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.772+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d419e3f0 con 0x7ff7d4102a90 2024-04-22T06:43:18.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.772+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d41a5c70 con 0x7ff7d4102a90 2024-04-22T06:43:18.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.773+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d4069730 con 0x7ff7d4102a90 2024-04-22T06:43:18.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.774+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff7c400f450 con 0x7ff7d4102a90 2024-04-22T06:43:18.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.774+0000 7ff7d17fa700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff7bc0a3d50 0x7ff7bc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:18.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.775+0000 7ff7d37fe700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff7bc0a3d50 0x7ff7bc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:18.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.775+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7ff7c40d05f0 con 0x7ff7d4102a90 2024-04-22T06:43:18.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.775+0000 7ff7d37fe700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff7bc0a3d50 0x7ff7bc0a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006010 tx=0x7ff7c801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.779+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7c4014030 con 0x7ff7d4102a90 2024-04-22T06:43:18.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.995+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff7d419a480 con 0x7ff7d4102a90 2024-04-22T06:43:18.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.996+0000 7ff7d17fa700 1 -- 172.21.15.79:0/3087726323 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff7d419a480 con 0x7ff7d4102a90 2024-04-22T06:43:18.994 INFO:teuthology.orchestra.run.smithi079.stdout:197568495667 2024-04-22T06:43:18.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff7bc0a3d50 msgr2=0x7ff7bc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff7bc0a3d50 0x7ff7bc0a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006010 tx=0x7ff7c801a040 comp rx=0 tx=0).stop 2024-04-22T06:43:18.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 msgr2=0x7ff7d4199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:18.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4199140 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c9b0 tx=0x7ff7c400cd70 comp rx=0 tx=0).stop 2024-04-22T06:43:18.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 shutdown_connections 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7d4108a80 0x7ff7d419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7ff7bc0a3d50 0x7ff7bc0a6210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7d4102a90 0x7ff7d4199140 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 --2- 172.21.15.79:0/3087726323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d4103450 0x7ff7d4199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 >> 172.21.15.79:0/3087726323 conn(0x7ff7d4079bf0 msgr2=0x7ff7d41015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 shutdown_connections 2024-04-22T06:43:18.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:18.998+0000 7ff7da8fb700 1 -- 172.21.15.79:0/3087726323 wait complete. 2024-04-22T06:43:19.187 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:18 smithi079 ceph-mon[27946]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:43:19.187 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1090571298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:43:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:18 smithi193 ceph-mon[31815]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:43:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1090571298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:43:19.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:18 smithi190 ceph-mon[31421]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:43:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1090571298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:43:20.110 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:19 smithi079 ceph-mon[27946]: pgmap v318: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:20.111 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:19 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3087726323' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:43:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:19 smithi193 ceph-mon[31815]: pgmap v318: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:20.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:19 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3087726323' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:43:20.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:19 smithi190 ceph-mon[31421]: pgmap v318: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:19 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3087726323' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:43:21.974 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332137 got 219043332140 for osd.7 2024-04-22T06:43:21.975 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:22.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:21 smithi193 ceph-mon[31815]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-22T06:43:22.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:21 smithi190 ceph-mon[31421]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-22T06:43:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:21 smithi079 ceph-mon[27946]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-22T06:43:22.812 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495667 for osd.6 2024-04-22T06:43:22.812 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:23.108 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:22 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:23.163 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:22 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:23.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:22 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:23.482 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:23.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.765+0000 7f3ee0985700 1 -- 172.21.15.79:0/3312439171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc1022f0 msgr2=0x7f3edc10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:23.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.765+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3312439171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc1022f0 0x7f3edc10e7e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400b0a0 tx=0x7f3ed400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 -- 172.21.15.79:0/3312439171 shutdown_connections 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3312439171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc1022f0 0x7f3edc10e7e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3312439171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3edc107980 0x7f3edc107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3312439171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3edc101930 0x7f3edc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 -- 172.21.15.79:0/3312439171 >> 172.21.15.79:0/3312439171 conn(0x7f3edc0788b0 msgr2=0x7f3edc078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 -- 172.21.15.79:0/3312439171 shutdown_connections 2024-04-22T06:43:23.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.766+0000 7f3ee0985700 1 -- 172.21.15.79:0/3312439171 wait complete. 2024-04-22T06:43:23.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 Processor -- start 2024-04-22T06:43:23.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 -- start start 2024-04-22T06:43:23.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 0x7f3edc199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:23.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3edc1022f0 0x7f3edc19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:23.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3edc107980 0x7f3edc1a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:23.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111af0 con 0x7f3edc101930 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111970 con 0x7f3edc107980 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ee0985700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc111c70 con 0x7f3edc1022f0 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 0x7f3edc199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 0x7f3edc199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47598/0 (socket says 172.21.15.79:47598) 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 -- 172.21.15.79:0/3315055743 learned_addr learned my addr 172.21.15.79:0/3315055743 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3edad9d700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3edc107980 0x7f3edc1a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3ed9d9b700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3edc1022f0 0x7f3edc19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 -- 172.21.15.79:0/3315055743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3edc1022f0 msgr2=0x7f3edc19a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3edc1022f0 0x7f3edc19a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 -- 172.21.15.79:0/3315055743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3edc107980 msgr2=0x7f3edc1a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3edc107980 0x7f3edc1a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 -- 172.21.15.79:0/3315055743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc1a5510 con 0x7f3edc101930 2024-04-22T06:43:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.767+0000 7f3eda59c700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 0x7f3edc199f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00d980 tx=0x7f3ecc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:23.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.768+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc017070 con 0x7f3edc101930 2024-04-22T06:43:23.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.768+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc1a5740 con 0x7f3edc101930 2024-04-22T06:43:23.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.768+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ecc005020 con 0x7f3edc101930 2024-04-22T06:43:23.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.768+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00fa10 con 0x7f3edc101930 2024-04-22T06:43:23.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.768+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1a5c30 con 0x7f3edc101930 2024-04-22T06:43:23.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.769+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc109ea0 con 0x7f3edc101930 2024-04-22T06:43:23.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.770+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ecc004a30 con 0x7f3edc101930 2024-04-22T06:43:23.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.771+0000 7f3ecb7fe700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3ec40a3d50 0x7f3ec40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:23.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.771+0000 7f3ed9d9b700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3ec40a3d50 0x7f3ec40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:23.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.771+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f3ecc0d03d0 con 0x7f3edc101930 2024-04-22T06:43:23.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.771+0000 7f3ed9d9b700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3ec40a3d50 0x7f3ec40a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3edc078fb0 tx=0x7f3ed0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:23.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.776+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ecc020020 con 0x7f3edc101930 2024-04-22T06:43:23.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.979+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f3edc06b5c0 con 0x7f3edc101930 2024-04-22T06:43:23.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.979+0000 7f3ecb7fe700 1 -- 172.21.15.79:0/3315055743 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3ecc09cc00 con 0x7f3edc101930 2024-04-22T06:43:23.978 INFO:teuthology.orchestra.run.smithi079.stdout:253403070505 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3ec40a3d50 msgr2=0x7f3ec40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3ec40a3d50 0x7f3ec40a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3edc078fb0 tx=0x7f3ed0009260 comp rx=0 tx=0).stop 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 msgr2=0x7f3edc199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 0x7f3edc199f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00d980 tx=0x7f3ecc00dc90 comp rx=0 tx=0).stop 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 shutdown_connections 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3edc107980 0x7f3edc1a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f3ec40a3d50 0x7f3ec40a6210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3edc101930 0x7f3edc199f50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 --2- 172.21.15.79:0/3315055743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3edc1022f0 0x7f3edc19a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 >> 172.21.15.79:0/3315055743 conn(0x7f3edc0788b0 msgr2=0x7f3edc0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 shutdown_connections 2024-04-22T06:43:23.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:23.981+0000 7f3ee0985700 1 -- 172.21.15.79:0/3315055743 wait complete. 2024-04-22T06:43:24.008 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:23 smithi079 ceph-mon[27946]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:24.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:23 smithi193 ceph-mon[31815]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:24.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:23 smithi190 ceph-mon[31421]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.437+0000 7f0eddac3700 1 -- 172.21.15.79:0/2666835088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8108a50 msgr2=0x7f0ed8108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.437+0000 7f0eddac3700 1 --2- 172.21.15.79:0/2666835088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8108a50 0x7f0ed8108e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009990 tx=0x7f0ec8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 -- 172.21.15.79:0/2666835088 shutdown_connections 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 --2- 172.21.15.79:0/2666835088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed8103450 0x7f0ed810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 --2- 172.21.15.79:0/2666835088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8108a50 0x7f0ed8108e30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 --2- 172.21.15.79:0/2666835088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ed8102a90 0x7f0ed8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 -- 172.21.15.79:0/2666835088 >> 172.21.15.79:0/2666835088 conn(0x7f0ed8079ba0 msgr2=0x7f0ed8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:24.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 -- 172.21.15.79:0/2666835088 shutdown_connections 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 -- 172.21.15.79:0/2666835088 wait complete. 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 Processor -- start 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.438+0000 7f0eddac3700 1 -- start start 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ed8102a90 0x7f0ed8199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 0x7f0ed8199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed8108a50 0x7f0ed819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8112c00 con 0x7f0ed8103450 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8112a80 con 0x7f0ed8108a50 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8112d80 con 0x7f0ed8102a90 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 0x7f0ed8199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:24.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 0x7f0ed8199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47616/0 (socket says 172.21.15.79:47616) 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 -- 172.21.15.79:0/3002287681 learned_addr learned my addr 172.21.15.79:0/3002287681 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0ed77fe700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ed8102a90 0x7f0ed8199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0ed7fff700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed8108a50 0x7f0ed819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 -- 172.21.15.79:0/3002287681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ed8102a90 msgr2=0x7f0ed8199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ed8102a90 0x7f0ed8199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 -- 172.21.15.79:0/3002287681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed8108a50 msgr2=0x7f0ed819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:24.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed8108a50 0x7f0ed819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 -- 172.21.15.79:0/3002287681 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed819e140 con 0x7f0ed8103450 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eceffd700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 0x7f0ed8199670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000d980 tx=0x7f0ec000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0017070 con 0x7f0ed8103450 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed819e430 con 0x7f0ed8103450 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec0005020 con 0x7f0ed8103450 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000fa10 con 0x7f0ed8103450 2024-04-22T06:43:24.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.439+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed81a5cb0 con 0x7f0ed8103450 2024-04-22T06:43:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.440+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed8069730 con 0x7f0ed8103450 2024-04-22T06:43:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.441+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0ec0004a30 con 0x7f0ed8103450 2024-04-22T06:43:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.442+0000 7f0ed57fa700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0ec40a3cd0 0x7f0ec40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.442+0000 7f0ed77fe700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0ec40a3cd0 0x7f0ec40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.442+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f0ec00d0420 con 0x7f0ed8103450 2024-04-22T06:43:24.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.442+0000 7f0ed77fe700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0ec40a3cd0 0x7f0ec40a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b5c0 tx=0x7f0ec8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:24.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.446+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ec0020020 con 0x7f0ed8103450 2024-04-22T06:43:24.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.653+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f0ed802df80 con 0x7f0ed8103450 2024-04-22T06:43:24.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.653+0000 7f0ed57fa700 1 -- 172.21.15.79:0/3002287681 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f0ec009ccb0 con 0x7f0ed8103450 2024-04-22T06:43:24.652 INFO:teuthology.orchestra.run.smithi079.stdout:309237645342 2024-04-22T06:43:24.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.655+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0ec40a3cd0 msgr2=0x7f0ec40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:24.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.655+0000 7f0eddac3700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0ec40a3cd0 0x7f0ec40a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b5c0 tx=0x7f0ec8005960 comp rx=0 tx=0).stop 2024-04-22T06:43:24.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.655+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 msgr2=0x7f0ed8199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:24.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.655+0000 7f0eddac3700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 0x7f0ed8199670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000d980 tx=0x7f0ec000dc90 comp rx=0 tx=0).stop 2024-04-22T06:43:24.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 shutdown_connections 2024-04-22T06:43:24.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed8108a50 0x7f0ed819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ed8102a90 0x7f0ed8199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f0ec40a3cd0 0x7f0ec40a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 --2- 172.21.15.79:0/3002287681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0ed8103450 0x7f0ed8199670 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:24.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 >> 172.21.15.79:0/3002287681 conn(0x7f0ed8079ba0 msgr2=0x7f0ed8101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:24.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 shutdown_connections 2024-04-22T06:43:24.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.656+0000 7f0eddac3700 1 -- 172.21.15.79:0/3002287681 wait complete. 2024-04-22T06:43:24.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:24 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3315055743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:43:24.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:24.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:24.964 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:24.965 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:24 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:24.965 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:24 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3002287681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:43:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:24 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3315055743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:43:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:24 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:24 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3002287681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:43:25.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:24 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3315055743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:43:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:24 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:24 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3002287681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:43:26.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:25 smithi079 ceph-mon[27946]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:26.082 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:25 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3169739586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:43:26.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:25 smithi193 ceph-mon[31815]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:26.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:25 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3169739586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:43:26.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:25 smithi190 ceph-mon[31421]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:26.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:25 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3169739586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:43:26.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645336 got 309237645342 for osd.10 2024-04-22T06:43:26.272 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:27.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 -- 172.21.15.79:0/3983216955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06b980 msgr2=0x7f4cac10f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3983216955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06b980 0x7f4cac10f8f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400b0a0 tx=0x7f4ca400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 -- 172.21.15.79:0/3983216955 shutdown_connections 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3983216955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06b980 0x7f4cac10f8f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3983216955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac108a20 0x7f4cac108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3983216955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06afe0 0x7f4cac06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 -- 172.21.15.79:0/3983216955 >> 172.21.15.79:0/3983216955 conn(0x7f4cac0788b0 msgr2=0x7f4cac078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 -- 172.21.15.79:0/3983216955 shutdown_connections 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.086+0000 7f4cb2106700 1 -- 172.21.15.79:0/3983216955 wait complete. 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 Processor -- start 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 -- start start 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 0x7f4cac101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06b980 0x7f4cac101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac108a20 0x7f4cac102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac112ca0 con 0x7f4cac06afe0 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac112b20 con 0x7f4cac06b980 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cb2106700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac112e20 con 0x7f4cac108a20 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cab7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 0x7f4cac101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cab7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 0x7f4cac101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47650/0 (socket says 172.21.15.79:47650) 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cab7fe700 1 -- 172.21.15.79:0/3286425334 learned_addr learned my addr 172.21.15.79:0/3286425334 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4cabfff700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac108a20 0x7f4cac102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.087+0000 7f4caaffd700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06b980 0x7f4cac101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cab7fe700 1 -- 172.21.15.79:0/3286425334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac108a20 msgr2=0x7f4cac102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cab7fe700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac108a20 0x7f4cac102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cab7fe700 1 -- 172.21.15.79:0/3286425334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06b980 msgr2=0x7f4cac101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cab7fe700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06b980 0x7f4cac101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cab7fe700 1 -- 172.21.15.79:0/3286425334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac1a7760 con 0x7f4cac06afe0 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4caaffd700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06b980 0x7f4cac101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cab7fe700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 0x7f4cac101a00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00b530 tx=0x7f4c9c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c015020 con 0x7f4cac06afe0 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c9c011ce0 con 0x7f4cac06afe0 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cac1a7a50 con 0x7f4cac06afe0 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c010e40 con 0x7f4cac06afe0 2024-04-22T06:43:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.088+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cac1a7f90 con 0x7f4cac06afe0 2024-04-22T06:43:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.089+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cac10b020 con 0x7f4cac06afe0 2024-04-22T06:43:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.090+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4c9c018020 con 0x7f4cac06afe0 2024-04-22T06:43:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.091+0000 7f4ca8ff9700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c940a3d50 0x7f4c940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.091+0000 7f4caaffd700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c940a3d50 0x7f4c940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.091+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f4c9c0d01b0 con 0x7f4cac06afe0 2024-04-22T06:43:27.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.091+0000 7f4caaffd700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c940a3d50 0x7f4c940a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4cac078f20 tx=0x7f4ca0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:27.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.094+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c9c01f020 con 0x7f4cac06afe0 2024-04-22T06:43:27.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.296+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4cac10aa00 con 0x7f4cac06afe0 2024-04-22T06:43:27.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.297+0000 7f4ca8ff9700 1 -- 172.21.15.79:0/3286425334 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4c9c09cb50 con 0x7f4cac06afe0 2024-04-22T06:43:27.296 INFO:teuthology.orchestra.run.smithi079.stdout:141733920830 2024-04-22T06:43:27.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c940a3d50 msgr2=0x7f4c940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:27.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c940a3d50 0x7f4c940a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4cac078f20 tx=0x7f4ca0008040 comp rx=0 tx=0).stop 2024-04-22T06:43:27.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 msgr2=0x7f4cac101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:27.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 0x7f4cac101a00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00b530 tx=0x7f4c9c00b840 comp rx=0 tx=0).stop 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 shutdown_connections 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac108a20 0x7f4cac102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c940a3d50 0x7f4c940a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4cac06afe0 0x7f4cac101a00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 --2- 172.21.15.79:0/3286425334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cac06b980 0x7f4cac101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 >> 172.21.15.79:0/3286425334 conn(0x7f4cac0788b0 msgr2=0x7f4cac0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 shutdown_connections 2024-04-22T06:43:27.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.299+0000 7f4cb2106700 1 -- 172.21.15.79:0/3286425334 wait complete. 2024-04-22T06:43:27.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:27 smithi079 ceph-mon[27946]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:27.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3286425334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:43:28.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 -- 172.21.15.79:0/1309272897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c1022f0 msgr2=0x7f196c10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 --2- 172.21.15.79:0/1309272897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c1022f0 0x7f196c10e7e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f196400b0a0 tx=0x7f196400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 -- 172.21.15.79:0/1309272897 shutdown_connections 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 --2- 172.21.15.79:0/1309272897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c1022f0 0x7f196c10e7e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 --2- 172.21.15.79:0/1309272897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f196c107980 0x7f196c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 --2- 172.21.15.79:0/1309272897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f196c101930 0x7f196c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.015+0000 7f1973252700 1 -- 172.21.15.79:0/1309272897 >> 172.21.15.79:0/1309272897 conn(0x7f196c0788b0 msgr2=0x7f196c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 -- 172.21.15.79:0/1309272897 shutdown_connections 2024-04-22T06:43:28.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 -- 172.21.15.79:0/1309272897 wait complete. 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 Processor -- start 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 -- start start 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f196c101930 0x7f196c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f196c1022f0 0x7f196c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c107980 0x7f196c19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f196c111ad0 con 0x7f196c107980 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f196c111950 con 0x7f196c101930 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.016+0000 7f1973252700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f196c111c50 con 0x7f196c1022f0 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c107980 0x7f196c19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:28.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c107980 0x7f196c19d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47690/0 (socket says 172.21.15.79:47690) 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 -- 172.21.15.79:0/2202556319 learned_addr learned my addr 172.21.15.79:0/2202556319 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f196bfff700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f196c1022f0 0x7f196c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f1970fee700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f196c101930 0x7f196c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 -- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f196c1022f0 msgr2=0x7f196c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f196c1022f0 0x7f196c1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 -- 172.21.15.79:0/2202556319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f196c101930 msgr2=0x7f196c199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f196c101930 0x7f196c199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 -- 172.21.15.79:0/2202556319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c19e0a0 con 0x7f196c107980 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f1970fee700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f196c101930 0x7f196c199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:28.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f19717ef700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c107980 0x7f196c19d960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f19640062a0 tx=0x7f1964004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964021070 con 0x7f196c107980 2024-04-22T06:43:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1964003c00 con 0x7f196c107980 2024-04-22T06:43:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.017+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964004830 con 0x7f196c107980 2024-04-22T06:43:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.018+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196c19e330 con 0x7f196c107980 2024-04-22T06:43:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.018+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f196c1a5c10 con 0x7f196c107980 2024-04-22T06:43:28.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.019+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f196c0631d0 con 0x7f196c107980 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.020+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1964007a00 con 0x7f196c107980 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.020+0000 7f1969ffb700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f195c0a3d50 0x7f195c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.020+0000 7f1970fee700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f195c0a3d50 0x7f195c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.020+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f196400f020 con 0x7f196c107980 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.021+0000 7f1969ffb700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] conn(0x7f195c0ad890 0x7f195c0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.021+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 --> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f195c0b0400 con 0x7f195c0ad890 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.021+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f19640d1db0 con 0x7f196c107980 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.021+0000 7f1970fee700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f195c0a3d50 0x7f195c0a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1960009bc0 tx=0x7f1960009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.021+0000 7f196bfff700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] conn(0x7f195c0ad890 0x7f195c0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.021+0000 7f196bfff700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] conn(0x7f195c0ad890 0x7f195c0afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:28.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.022+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== osd.5 v2:172.21.15.193:6808/636159812 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f195c0b0400 con 0x7f195c0ad890 2024-04-22T06:43:28.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.038+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 --> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f196c063650 con 0x7f195c0ad890 2024-04-22T06:43:28.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1969ffb700 1 -- 172.21.15.79:0/2202556319 <== osd.5 v2:172.21.15.193:6808/636159812 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f196c063650 con 0x7f195c0ad890 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] conn(0x7f195c0ad890 msgr2=0x7f195c0afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] conn(0x7f195c0ad890 0x7f195c0afd30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f195c0a3d50 msgr2=0x7f195c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f195c0a3d50 0x7f195c0a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1960009bc0 tx=0x7f1960009260 comp rx=0 tx=0).stop 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c107980 msgr2=0x7f196c19d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 --2- 172.21.15.79:0/2202556319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f196c107980 0x7f196c19d960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f19640062a0 tx=0x7f1964004d30 comp rx=0 tx=0).stop 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f19717ef700 1 -- 172.21.15.79:0/2202556319 reap_dead start 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 shutdown_connections 2024-04-22T06:43:28.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.039+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 >> 172.21.15.79:0/2202556319 conn(0x7f196c0788b0 msgr2=0x7f196c06b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:28.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.040+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 shutdown_connections 2024-04-22T06:43:28.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.040+0000 7f1973252700 1 -- 172.21.15.79:0/2202556319 wait complete. 2024-04-22T06:43:28.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:27 smithi193 ceph-mon[31815]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:28.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3286425334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:43:28.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:27 smithi190 ceph-mon[31421]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:28.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3286425334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:43:28.700 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920830 for osd.4 2024-04-22T06:43:28.700 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2198791538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:43:29.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2198791538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:43:29.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2198791538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:43:29.423 INFO:teuthology.orchestra.run.smithi079.stdout:163208757305 2024-04-22T06:43:29.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T06:43:29.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 -- 172.21.15.79:0/1469365182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 msgr2=0x7f8eec10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 --2- 172.21.15.79:0/1469365182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec10f900 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009990 tx=0x7f8ee8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 -- 172.21.15.79:0/1469365182 shutdown_connections 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 --2- 172.21.15.79:0/1469365182 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec10f900 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 --2- 172.21.15.79:0/1469365182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eec108a30 0x7f8eec108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 --2- 172.21.15.79:0/1469365182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec06a750 0x7f8eec06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 -- 172.21.15.79:0/1469365182 >> 172.21.15.79:0/1469365182 conn(0x7f8eec0788f0 msgr2=0x7f8eec078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 -- 172.21.15.79:0/1469365182 shutdown_connections 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.727+0000 7f8ef431a700 1 -- 172.21.15.79:0/1469365182 wait complete. 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 Processor -- start 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 -- start start 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eec06a750 0x7f8eec101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec108a30 0x7f8eec102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec112c10 con 0x7f8eec06b0f0 2024-04-22T06:43:29.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec112a90 con 0x7f8eec06a750 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef431a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec112d90 con 0x7f8eec108a30 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47714/0 (socket says 172.21.15.79:47714) 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 -- 172.21.15.79:0/3077760265 learned_addr learned my addr 172.21.15.79:0/3077760265 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef20b6700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eec06a750 0x7f8eec101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef28b7700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec108a30 0x7f8eec102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 -- 172.21.15.79:0/3077760265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec108a30 msgr2=0x7f8eec102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec108a30 0x7f8eec102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 -- 172.21.15.79:0/3077760265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eec06a750 msgr2=0x7f8eec101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eec06a750 0x7f8eec101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef18b5700 1 -- 172.21.15.79:0/3077760265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eec1a7630 con 0x7f8eec06b0f0 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.728+0000 7f8ef28b7700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec108a30 0x7f8eec102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.729+0000 7f8ef18b5700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec101f50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00d980 tx=0x7f8edc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:29.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.729+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc017070 con 0x7f8eec06b0f0 2024-04-22T06:43:29.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.729+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eec1a7920 con 0x7f8eec06b0f0 2024-04-22T06:43:29.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.729+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8edc005020 con 0x7f8eec06b0f0 2024-04-22T06:43:29.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.729+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc009b50 con 0x7f8eec06b0f0 2024-04-22T06:43:29.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.729+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eec1a7e60 con 0x7f8eec06b0f0 2024-04-22T06:43:29.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.730+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eec069730 con 0x7f8eec06b0f0 2024-04-22T06:43:29.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.731+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8edc0051d0 con 0x7f8eec06b0f0 2024-04-22T06:43:29.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.732+0000 7f8ee37fe700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ed80a3cd0 0x7f8ed80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.732+0000 7f8ef20b6700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ed80a3cd0 0x7f8ed80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.732+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f8edc0d04a0 con 0x7f8eec06b0f0 2024-04-22T06:43:29.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.733+0000 7f8ef20b6700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ed80a3cd0 0x7f8ed80a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4009550 tx=0x7f8ee40091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:29.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.739+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8edc020020 con 0x7f8eec06b0f0 2024-04-22T06:43:29.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.943+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8eec02df80 con 0x7f8eec06b0f0 2024-04-22T06:43:29.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:29 smithi079 ceph-mon[27946]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:29.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.944+0000 7f8ee37fe700 1 -- 172.21.15.79:0/3077760265 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8edc020020 con 0x7f8eec06b0f0 2024-04-22T06:43:29.943 INFO:teuthology.orchestra.run.smithi079.stdout:107374182467 2024-04-22T06:43:29.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ed80a3cd0 msgr2=0x7f8ed80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ed80a3cd0 0x7f8ed80a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4009550 tx=0x7f8ee40091b0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 msgr2=0x7f8eec101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec101f50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00d980 tx=0x7f8edc00dc90 comp rx=0 tx=0).stop 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 shutdown_connections 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8eec108a30 0x7f8eec102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eec06a750 0x7f8eec101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ed80a3cd0 0x7f8ed80a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 --2- 172.21.15.79:0/3077760265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8eec06b0f0 0x7f8eec101f50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 >> 172.21.15.79:0/3077760265 conn(0x7f8eec0788f0 msgr2=0x7f8eec0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 shutdown_connections 2024-04-22T06:43:29.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:29.946+0000 7f8ef431a700 1 -- 172.21.15.79:0/3077760265 wait complete. 2024-04-22T06:43:30.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:29 smithi190 ceph-mon[31421]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:30.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 -- 172.21.15.79:0/1398223337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880102a90 msgr2=0x7f5880102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 --2- 172.21.15.79:0/1398223337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880102a90 0x7f5880102f10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5874009990 tx=0x7f5874009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 -- 172.21.15.79:0/1398223337 shutdown_connections 2024-04-22T06:43:30.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 --2- 172.21.15.79:0/1398223337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880103450 0x7f588010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 --2- 172.21.15.79:0/1398223337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5880108a50 0x7f5880108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 --2- 172.21.15.79:0/1398223337 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880102a90 0x7f5880102f10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 -- 172.21.15.79:0/1398223337 >> 172.21.15.79:0/1398223337 conn(0x7f5880079ba0 msgr2=0x7f5880079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 -- 172.21.15.79:0/1398223337 shutdown_connections 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.849+0000 7f5884e91700 1 -- 172.21.15.79:0/1398223337 wait complete. 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 Processor -- start 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 -- start start 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5880102a90 0x7f58801991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880103450 0x7f58801996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 0x7f588019da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5880112c80 con 0x7f5880108a50 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5880112b00 con 0x7f5880103450 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f5884e91700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5880112e00 con 0x7f5880102a90 2024-04-22T06:43:30.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.850+0000 7f587ed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 0x7f588019da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 0x7f588019da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47628/0 (socket says 172.21.15.79:47628) 2024-04-22T06:43:30.178 INFO:teuthology.orchestra.run.smithi079.stdout:287762808868 2024-04-22T06:43:30.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 -- 172.21.15.79:0/3169739586 learned_addr learned my addr 172.21.15.79:0/3169739586 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:30.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587e59c700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5880102a90 0x7f58801991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587dd9b700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880103450 0x7f58801996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 -- 172.21.15.79:0/3169739586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5880102a90 msgr2=0x7f58801991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5880102a90 0x7f58801991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 -- 172.21.15.79:0/3169739586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880103450 msgr2=0x7f58801996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880103450 0x7f58801996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 -- 172.21.15.79:0/3169739586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588019e1c0 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f587ed9d700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 0x7f588019da80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f587800b970 tx=0x7f587800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587800d860 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588019e4b0 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f587800e430 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5878017660 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.851+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58801a5d30 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.852+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5880069730 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.853+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f587800da00 con 0x7f5880108a50 2024-04-22T06:43:30.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.854+0000 7f586f7fe700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58680a3cd0 0x7f58680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.854+0000 7f587e59c700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58680a3cd0 0x7f58680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.854+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f58780d0370 con 0x7f5880108a50 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.854+0000 7f587e59c700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58680a3cd0 0x7f58680a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5870004630 tx=0x7f587000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:24.858+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5878013030 con 0x7f5880108a50 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.066+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f588019a510 con 0x7f5880108a50 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.066+0000 7f586f7fe700 1 -- 172.21.15.79:0/3169739586 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f587809cc00 con 0x7f5880108a50 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58680a3cd0 msgr2=0x7f58680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58680a3cd0 0x7f58680a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5870004630 tx=0x7f587000b3c0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 msgr2=0x7f588019da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 0x7f588019da80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f587800b970 tx=0x7f587800bc80 comp rx=0 tx=0).stop 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 shutdown_connections 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58680a3cd0 0x7f58680a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5880108a50 0x7f588019da80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5880102a90 0x7f58801991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 --2- 172.21.15.79:0/3169739586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5880103450 0x7f58801996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 >> 172.21.15.79:0/3169739586 conn(0x7f5880079ba0 msgr2=0x7f5880101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:30.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 shutdown_connections 2024-04-22T06:43:30.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:25.068+0000 7f5884e91700 1 -- 172.21.15.79:0/3169739586 wait complete. 2024-04-22T06:43:30.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:29 smithi193 ceph-mon[31815]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:30.917 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070501 got 253403070505 for osd.8 2024-04-22T06:43:30.918 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:31.094 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:30 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3077760265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:43:31.115 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:30 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3077760265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:43:31.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:30 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3077760265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:43:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 -- 172.21.15.79:0/3084973719 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 msgr2=0x7fe2b0101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/3084973719 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0101df0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009990 tx=0x7fe2a0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 -- 172.21.15.79:0/3084973719 shutdown_connections 2024-04-22T06:43:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/3084973719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2b0102330 0x7fe2b010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/3084973719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b0107920 0x7fe2b0107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/3084973719 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0101df0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 -- 172.21.15.79:0/3084973719 >> 172.21.15.79:0/3084973719 conn(0x7fe2b0078910 msgr2=0x7fe2b0078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 -- 172.21.15.79:0/3084973719 shutdown_connections 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 -- 172.21.15.79:0/3084973719 wait complete. 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 Processor -- start 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.307+0000 7fe2b74c4700 1 -- start start 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2b0102330 0x7fe2b01997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b0107920 0x7fe2b019db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0111b60 con 0x7fe2b0101970 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01119e0 con 0x7fe2b0102330 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0111ce0 con 0x7fe2b0107920 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:31.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47720/0 (socket says 172.21.15.79:47720) 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 -- 172.21.15.79:0/2903723616 learned_addr learned my addr 172.21.15.79:0/2903723616 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b4a5f700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2b0102330 0x7fe2b01997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5a61700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b0107920 0x7fe2b019db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 -- 172.21.15.79:0/2903723616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b0107920 msgr2=0x7fe2b019db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b0107920 0x7fe2b019db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 -- 172.21.15.79:0/2903723616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2b0102330 msgr2=0x7fe2b01997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2b0102330 0x7fe2b01997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 -- 172.21.15.79:0/2903723616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b019e240 con 0x7fe2b0101970 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b5260700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0199280 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00b530 tx=0x7fe2ac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac015020 con 0x7fe2b0101970 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b019e530 con 0x7fe2b0101970 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2ac011ce0 con 0x7fe2b0101970 2024-04-22T06:43:31.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac010e40 con 0x7fe2b0101970 2024-04-22T06:43:31.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.308+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b01a5db0 con 0x7fe2b0101970 2024-04-22T06:43:31.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.309+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b0069730 con 0x7fe2b0101970 2024-04-22T06:43:31.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.311+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe2ac018020 con 0x7fe2b0101970 2024-04-22T06:43:31.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.311+0000 7fe2a67fc700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe29c0a3cd0 0x7fe29c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:31.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.311+0000 7fe2b4a5f700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe29c0a3cd0 0x7fe29c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:31.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.311+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fe2ac0cfcd0 con 0x7fe2b0101970 2024-04-22T06:43:31.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.312+0000 7fe2b4a5f700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe29c0a3cd0 0x7fe29c0a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000b5c0 tx=0x7fe2a0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:31.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.316+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2ac01f020 con 0x7fe2b0101970 2024-04-22T06:43:31.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.519+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe2b019a590 con 0x7fe2b0101970 2024-04-22T06:43:31.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.519+0000 7fe2a67fc700 1 -- 172.21.15.79:0/2903723616 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe2ac09c670 con 0x7fe2b0101970 2024-04-22T06:43:31.518 INFO:teuthology.orchestra.run.smithi079.stdout:55834574926 2024-04-22T06:43:31.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.521+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe29c0a3cd0 msgr2=0x7fe29c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:31.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.521+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe29c0a3cd0 0x7fe29c0a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000b5c0 tx=0x7fe2a0005960 comp rx=0 tx=0).stop 2024-04-22T06:43:31.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.521+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 msgr2=0x7fe2b0199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:31.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.521+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0199280 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00b530 tx=0x7fe2ac00b840 comp rx=0 tx=0).stop 2024-04-22T06:43:31.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 shutdown_connections 2024-04-22T06:43:31.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b0107920 0x7fe2b019db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fe29c0a3cd0 0x7fe29c0a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2b0101970 0x7fe2b0199280 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 --2- 172.21.15.79:0/2903723616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe2b0102330 0x7fe2b01997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:31.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 >> 172.21.15.79:0/2903723616 conn(0x7fe2b0078910 msgr2=0x7fe2b006b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:31.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 shutdown_connections 2024-04-22T06:43:31.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:31.522+0000 7fe2b74c4700 1 -- 172.21.15.79:0/2903723616 wait complete. 2024-04-22T06:43:32.175 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:31 smithi079 ceph-mon[27946]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:32.175 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2903723616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:43:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:31 smithi193 ceph-mon[31815]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:32.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2903723616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:43:32.202 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574919 got 55834574926 for osd.1 2024-04-22T06:43:32.202 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:31 smithi190 ceph-mon[31421]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:32.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2903723616' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:43:32.566 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808863 got 287762808868 for osd.9 2024-04-22T06:43:32.566 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:32.969 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182467 for osd.3 2024-04-22T06:43:32.970 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:33.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 -- 172.21.15.79:0/1401365841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 msgr2=0x7f8f00107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:33.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 --2- 172.21.15.79:0/1401365841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f00107d70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009990 tx=0x7f8ef4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 -- 172.21.15.79:0/1401365841 shutdown_connections 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 --2- 172.21.15.79:0/1401365841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 0x7f8f0010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 --2- 172.21.15.79:0/1401365841 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f00107d70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 --2- 172.21.15.79:0/1401365841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 0x7f8f00101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 -- 172.21.15.79:0/1401365841 >> 172.21.15.79:0/1401365841 conn(0x7f8f000788f0 msgr2=0x7f8f00078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 -- 172.21.15.79:0/1401365841 shutdown_connections 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 -- 172.21.15.79:0/1401365841 wait complete. 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.923+0000 7f8f05218700 1 Processor -- start 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 -- start start 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 0x7f8f00199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 0x7f8f001997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:33.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f0019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8f00111bd0 con 0x7f8f00107990 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8f00111a50 con 0x7f8f00101940 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8f05218700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f00111d50 con 0x7f8f00102300 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f0019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f0019db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47678/0 (socket says 172.21.15.79:47678) 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 -- 172.21.15.79:0/2198791538 learned_addr learned my addr 172.21.15.79:0/2198791538 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8efed9d700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 0x7f8f00199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8efe59c700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 0x7f8f001997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 -- 172.21.15.79:0/2198791538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 msgr2=0x7f8f001997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 0x7f8f001997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 -- 172.21.15.79:0/2198791538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 msgr2=0x7f8f00199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 0x7f8f00199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8eff59e700 1 -- 172.21.15.79:0/2198791538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0019e250 con 0x7f8f00107990 2024-04-22T06:43:33.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8efe59c700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 0x7f8f001997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.924+0000 7f8efed9d700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 0x7f8f00199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.925+0000 7f8eff59e700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f0019db10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800b970 tx=0x7f8ef800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.925+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8019070 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.925+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef8014440 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.925+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0019e540 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.925+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef800ce60 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.925+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f001a5dc0 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.926+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f00069730 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.927+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ef801c070 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.928+0000 7f8ee7fff700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee80a3cd0 0x7f8ee80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.928+0000 7f8efed9d700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee80a3cd0 0x7f8ee80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.928+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f8ef80cfd80 con 0x7f8f00107990 2024-04-22T06:43:33.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.929+0000 7f8efed9d700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee80a3cd0 0x7f8ee80a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4006010 tx=0x7f8ef400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:27.932+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ef809c720 con 0x7f8f00107990 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.144+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8f001a5ff0 con 0x7f8f00107990 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.144+0000 7f8ee7fff700 1 -- 172.21.15.79:0/2198791538 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8ef809c020 con 0x7f8f00107990 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stdout:77309411400 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee80a3cd0 msgr2=0x7f8ee80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee80a3cd0 0x7f8ee80a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4006010 tx=0x7f8ef400b540 comp rx=0 tx=0).stop 2024-04-22T06:43:33.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 msgr2=0x7f8f0019db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f0019db10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800b970 tx=0x7f8ef800bd30 comp rx=0 tx=0).stop 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 shutdown_connections 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f8ee80a3cd0 0x7f8ee80a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8f00107990 0x7f8f0019db10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f00101940 0x7f8f00199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 --2- 172.21.15.79:0/2198791538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f00102300 0x7f8f001997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 >> 172.21.15.79:0/2198791538 conn(0x7f8f000788f0 msgr2=0x7f8f0006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 shutdown_connections 2024-04-22T06:43:33.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:28.147+0000 7f8f05218700 1 -- 172.21.15.79:0/2198791538 wait complete. 2024-04-22T06:43:33.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411400 for osd.2 2024-04-22T06:43:33.829 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:34.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:33 smithi193 ceph-mon[31815]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:34.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:33 smithi190 ceph-mon[31421]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:34.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:33 smithi079 ceph-mon[27946]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:34.355 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 -- 172.21.15.79:0/1961740853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240108a50 msgr2=0x7f2240108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 --2- 172.21.15.79:0/1961740853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240108a50 0x7f2240108e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2234009990 tx=0x7f2234009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 -- 172.21.15.79:0/1961740853 shutdown_connections 2024-04-22T06:43:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 --2- 172.21.15.79:0/1961740853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2240103450 0x7f224010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 --2- 172.21.15.79:0/1961740853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240108a50 0x7f2240108e30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 --2- 172.21.15.79:0/1961740853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2240102a90 0x7f2240102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 -- 172.21.15.79:0/1961740853 >> 172.21.15.79:0/1961740853 conn(0x7f2240079ba0 msgr2=0x7f2240079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 -- 172.21.15.79:0/1961740853 shutdown_connections 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.831+0000 7f2246a6d700 1 -- 172.21.15.79:0/1961740853 wait complete. 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 Processor -- start 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 -- start start 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2240102a90 0x7f224019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 0x7f224019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2240108a50 0x7f224019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2240112c60 con 0x7f2240103450 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2240112ae0 con 0x7f2240108a50 2024-04-22T06:43:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2246a6d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2240112de0 con 0x7f2240102a90 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 0x7f224019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 0x7f224019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39734/0 (socket says 172.21.15.79:39734) 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 -- 172.21.15.79:0/58858990 learned_addr learned my addr 172.21.15.79:0/58858990 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f2244809700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2240102a90 0x7f224019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f224500a700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2240108a50 0x7f224019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 -- 172.21.15.79:0/58858990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2240102a90 msgr2=0x7f224019b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2240102a90 0x7f224019b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 -- 172.21.15.79:0/58858990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2240108a50 msgr2=0x7f224019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2240108a50 0x7f224019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 -- 172.21.15.79:0/58858990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22401a03a0 con 0x7f2240103450 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.832+0000 7f223ffff700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 0x7f224019b8d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f222c00b530 tx=0x7f222c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.833+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c015020 con 0x7f2240103450 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.833+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22401a0690 con 0x7f2240103450 2024-04-22T06:43:34.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.833+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f222c011ce0 con 0x7f2240103450 2024-04-22T06:43:34.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.833+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c010dd0 con 0x7f2240103450 2024-04-22T06:43:34.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.833+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22401a7f10 con 0x7f2240103450 2024-04-22T06:43:34.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.834+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2240069730 con 0x7f2240103450 2024-04-22T06:43:34.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.835+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f222c018020 con 0x7f2240103450 2024-04-22T06:43:34.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.835+0000 7f223dffb700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22300a3cd0 0x7f22300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:34.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.835+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f222c0cfb60 con 0x7f2240103450 2024-04-22T06:43:34.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.835+0000 7f2244809700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22300a3cd0 0x7f22300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:34.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.836+0000 7f2244809700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22300a3cd0 0x7f22300a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f223400b5c0 tx=0x7f2234005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:34.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:34.840+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f222c01f020 con 0x7f2240103450 2024-04-22T06:43:35.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.046+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f22401a8140 con 0x7f2240103450 2024-04-22T06:43:35.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.046+0000 7f223dffb700 1 -- 172.21.15.79:0/58858990 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f222c09c500 con 0x7f2240103450 2024-04-22T06:43:35.045 INFO:teuthology.orchestra.run.smithi079.stdout:34359738452 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22300a3cd0 msgr2=0x7f22300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22300a3cd0 0x7f22300a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f223400b5c0 tx=0x7f2234005960 comp rx=0 tx=0).stop 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 msgr2=0x7f224019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 0x7f224019b8d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f222c00b530 tx=0x7f222c00b840 comp rx=0 tx=0).stop 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 shutdown_connections 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2240108a50 0x7f224019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2240102a90 0x7f224019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f22300a3cd0 0x7f22300a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 --2- 172.21.15.79:0/58858990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2240103450 0x7f224019b8d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.048+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 >> 172.21.15.79:0/58858990 conn(0x7f2240079ba0 msgr2=0x7f224010df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.049+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 shutdown_connections 2024-04-22T06:43:35.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:35.049+0000 7f2246a6d700 1 -- 172.21.15.79:0/58858990 wait complete. 2024-04-22T06:43:36.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:35 smithi193 ceph-mon[31815]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:36.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:35 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/58858990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:43:36.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:35 smithi190 ceph-mon[31421]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:36.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:35 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/58858990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:43:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:35 smithi079 ceph-mon[27946]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/58858990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:43:37.297 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738445 got 34359738452 for osd.0 2024-04-22T06:43:37.297 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:37.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 -- 172.21.15.79:0/91736387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc101dc0 msgr2=0x7f7dbc10f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:37.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 --2- 172.21.15.79:0/91736387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc101dc0 0x7f7dbc10f830 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7db400b0a0 tx=0x7f7db400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:37.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 -- 172.21.15.79:0/91736387 shutdown_connections 2024-04-22T06:43:37.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 --2- 172.21.15.79:0/91736387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc101dc0 0x7f7dbc10f830 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:37.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 --2- 172.21.15.79:0/91736387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dbc107b20 0x7f7dbc107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:37.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 --2- 172.21.15.79:0/91736387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dbc101400 0x7f7dbc101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:37.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 -- 172.21.15.79:0/91736387 >> 172.21.15.79:0/91736387 conn(0x7f7dbc076c60 msgr2=0x7f7dbc077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:37.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 -- 172.21.15.79:0/91736387 shutdown_connections 2024-04-22T06:43:37.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.835+0000 7f7dc1359700 1 -- 172.21.15.79:0/91736387 wait complete. 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 Processor -- start 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 -- start start 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dbc101400 0x7f7dbc199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dbc101dc0 0x7f7dbc199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 0x7f7dbc19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc112cb0 con 0x7f7dbc107b20 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc112b30 con 0x7f7dbc101400 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dc1359700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7dbc112e30 con 0x7f7dbc101dc0 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 0x7f7dbc19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 0x7f7dbc19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39750/0 (socket says 172.21.15.79:39750) 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 -- 172.21.15.79:0/1503431913 learned_addr learned my addr 172.21.15.79:0/1503431913 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbaffd700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dbc101400 0x7f7dbc199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:37.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dba7fc700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dbc101dc0 0x7f7dbc199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 -- 172.21.15.79:0/1503431913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dbc101dc0 msgr2=0x7f7dbc199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dbc101dc0 0x7f7dbc199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 -- 172.21.15.79:0/1503431913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dbc101400 msgr2=0x7f7dbc199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dbc101400 0x7f7dbc199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 -- 172.21.15.79:0/1503431913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dbc19e0e0 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.836+0000 7f7dbb7fe700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 0x7f7dbc19d9a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7db40062a0 tx=0x7f7db4004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.837+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4021070 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.837+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7db4007d00 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.837+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dbc19e370 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.837+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db401d450 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.837+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dbc1a5c50 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.838+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dbc069730 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.839+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7db4004020 con 0x7f7dbc107b20 2024-04-22T06:43:37.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.839+0000 7f7da3fff700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7da40a3d50 0x7f7da40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:37.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.839+0000 7f7dbaffd700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7da40a3d50 0x7f7da40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:37.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.839+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f7db400f020 con 0x7f7dbc107b20 2024-04-22T06:43:37.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.840+0000 7f7dbaffd700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7da40a3d50 0x7f7da40a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7dac006fd0 tx=0x7f7dac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:37.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:37.844+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7db402a020 con 0x7f7dbc107b20 2024-04-22T06:43:37.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:37 smithi079 ceph-mon[27946]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.059+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7dbc02df80 con 0x7f7dbc107b20 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.059+0000 7f7da3fff700 1 -- 172.21.15.79:0/1503431913 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7db409e550 con 0x7f7dbc107b20 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi079.stdout:163208757307 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7da40a3d50 msgr2=0x7f7da40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7da40a3d50 0x7f7da40a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7dac006fd0 tx=0x7f7dac008040 comp rx=0 tx=0).stop 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 msgr2=0x7f7dbc19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 0x7f7dbc19d9a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7db40062a0 tx=0x7f7db4004ed0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 shutdown_connections 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f7da40a3d50 0x7f7da40a6210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7dbc107b20 0x7f7dbc19d9a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dbc101400 0x7f7dbc199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 --2- 172.21.15.79:0/1503431913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dbc101dc0 0x7f7dbc199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.061+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 >> 172.21.15.79:0/1503431913 conn(0x7f7dbc076c60 msgr2=0x7f7dbc0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.062+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 shutdown_connections 2024-04-22T06:43:38.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:38.062+0000 7f7dc1359700 1 -- 172.21.15.79:0/1503431913 wait complete. 2024-04-22T06:43:38.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:37 smithi193 ceph-mon[31815]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:38.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:37 smithi190 ceph-mon[31421]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:38.777 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757307 for osd.5 2024-04-22T06:43:38.777 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:38.777 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T06:43:38.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T06:43:39.022 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:39.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:38 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1503431913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:43:39.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:38 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:39.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:38 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:39.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:38 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1503431913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:43:39.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:38 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:39.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:38 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:39.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:38 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1503431913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:43:39.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:38 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:39.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:38 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:40.171 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:39 smithi079 ceph-mon[27946]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:40.171 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:40.171 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:40.171 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:39 smithi079 ceph-mon[27946]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:40.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:39 smithi193 ceph-mon[31815]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:40.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:40.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:40.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:39 smithi193 ceph-mon[31815]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:40.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:39 smithi190 ceph-mon[31421]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:40.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:40.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:40.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:39 smithi190 ceph-mon[31421]: from='mgr.14152 172.21.15.79:0/3145096918' entity='mgr.x' 2024-04-22T06:43:40.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3800859427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 msgr2=0x7fc5c0111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:40.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3800859427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c0111830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800b0a0 tx=0x7fc5b800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3800859427 shutdown_connections 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3800859427 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c0111830 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3800859427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5c0074c70 0x7fc5c0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3800859427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5c0075590 0x7fc5c010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3800859427 >> 172.21.15.79:0/3800859427 conn(0x7fc5c0079bf0 msgr2=0x7fc5c007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3800859427 shutdown_connections 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.739+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3800859427 wait complete. 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 Processor -- start 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 -- start start 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5c0074c70 0x7fc5c019b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5c0075590 0x7fc5c019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:40.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0114ec0 con 0x7fc5c010f440 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0114d40 con 0x7fc5c0075590 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.740+0000 7fc5c4bab700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0115040 con 0x7fc5c0074c70 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c019fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39780/0 (socket says 172.21.15.79:39780) 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 -- 172.21.15.79:0/3580128228 learned_addr learned my addr 172.21.15.79:0/3580128228 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bdd9b700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5c0075590 0x7fc5c019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5be59c700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5c0074c70 0x7fc5c019b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 -- 172.21.15.79:0/3580128228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5c0074c70 msgr2=0x7fc5c019b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5c0074c70 0x7fc5c019b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 -- 172.21.15.79:0/3580128228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5c0075590 msgr2=0x7fc5c019b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5c0075590 0x7fc5c019b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 -- 172.21.15.79:0/3580128228 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c01a03f0 con 0x7fc5c010f440 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5bed9d700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c019fcb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8004c80 tx=0x7fc5b8004cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8021070 con 0x7fc5c010f440 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.741+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5b8007b80 con 0x7fc5c010f440 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.742+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c01a0680 con 0x7fc5c010f440 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.742+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8005040 con 0x7fc5c010f440 2024-04-22T06:43:40.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.742+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c01a7f60 con 0x7fc5c010f440 2024-04-22T06:43:40.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.743+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c0076390 con 0x7fc5c010f440 2024-04-22T06:43:40.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.744+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc5b8024070 con 0x7fc5c010f440 2024-04-22T06:43:40.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.744+0000 7fc5b77fe700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5ac0a3cd0 0x7fc5ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:40.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.744+0000 7fc5be59c700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5ac0a3cd0 0x7fc5ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:40.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.744+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc5b80d1500 con 0x7fc5c010f440 2024-04-22T06:43:40.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.745+0000 7fc5be59c700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5ac0a3cd0 0x7fc5ac0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009a10 tx=0x7fc5b0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:40.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.749+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5b809de20 con 0x7fc5c010f440 2024-04-22T06:43:40.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.963+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc5c019c740 con 0x7fc5ac0a3cd0 2024-04-22T06:43:40.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.971+0000 7fc5b77fe700 1 -- 172.21.15.79:0/3580128228 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258930 (secure 0 0 0) 0x7fc5c019c740 con 0x7fc5ac0a3cd0 2024-04-22T06:43:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5ac0a3cd0 msgr2=0x7fc5ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5ac0a3cd0 0x7fc5ac0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009a10 tx=0x7fc5b0009190 comp rx=0 tx=0).stop 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 msgr2=0x7fc5c019fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c019fcb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8004c80 tx=0x7fc5b8004cb0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 shutdown_connections 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc5ac0a3cd0 0x7fc5ac0a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc5c010f440 0x7fc5c019fcb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5c0074c70 0x7fc5c019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 --2- 172.21.15.79:0/3580128228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5c0075590 0x7fc5c019b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:40.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 >> 172.21.15.79:0/3580128228 conn(0x7fc5c0079bf0 msgr2=0x7fc5c0100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:40.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 shutdown_connections 2024-04-22T06:43:40.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:40.974+0000 7fc5c4bab700 1 -- 172.21.15.79:0/3580128228 wait complete. 2024-04-22T06:43:40.976 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:43:40.983 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:40 smithi079 ceph-mon[27946]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:41.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:40 smithi193 ceph-mon[31815]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:41.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:40 smithi190 ceph-mon[31421]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:41.641 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-22T06:43:40.663528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653835,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":584,"num_read_kb":409,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381580,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560244,"statfs":{"total":1151940427776,"available":1151549689856,"internally_reserved":0,"allocated":11833344,"data_stored":7428885,"data_compressed":112743,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809965+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:43:17.809965+0000","last_peered":"2024-04-22T06:43:17.809965+0000","last_clean":"2024-04-22T06:43:17.809965+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:43:17.809965+0000","last_undegraded":"2024-04-22T06:43:17.809965+0000","last_fullsized":"2024-04-22T06:43:17.809965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804244+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:43:17.804244+0000","last_peered":"2024-04-22T06:43:17.804244+0000","last_clean":"2024-04-22T06:43:17.804244+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:43:17.804244+0000","last_undegraded":"2024-04-22T06:43:17.804244+0000","last_fullsized":"2024-04-22T06:43:17.804244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804277+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:43:17.804277+0000","last_peered":"2024-04-22T06:43:17.804277+0000","last_clean":"2024-04-22T06:43:17.804277+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:43:17.804277+0000","last_undegraded":"2024-04-22T06:43:17.804277+0000","last_fullsized":"2024-04-22T06:43:17.804277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069465+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:43:18.069465+0000","last_peered":"2024-04-22T06:43:18.069465+0000","last_clean":"2024-04-22T06:43:18.069465+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:43:18.069465+0000","last_undegraded":"2024-04-22T06:43:18.069465+0000","last_fullsized":"2024-04-22T06:43:18.069465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807365+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:43:17.807365+0000","last_peered":"2024-04-22T06:43:17.807365+0000","last_clean":"2024-04-22T06:43:17.807365+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:43:17.807365+0000","last_undegraded":"2024-04-22T06:43:17.807365+0000","last_fullsized":"2024-04-22T06:43:17.807365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808320+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:43:17.808320+0000","last_peered":"2024-04-22T06:43:17.808320+0000","last_clean":"2024-04-22T06:43:17.808320+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:43:17.808320+0000","last_undegraded":"2024-04-22T06:43:17.808320+0000","last_fullsized":"2024-04-22T06:43:17.808320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805099+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:43:17.805099+0000","last_peered":"2024-04-22T06:43:17.805099+0000","last_clean":"2024-04-22T06:43:17.805099+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:43:17.805099+0000","last_undegraded":"2024-04-22T06:43:17.805099+0000","last_fullsized":"2024-04-22T06:43:17.805099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811215+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:43:17.811215+0000","last_peered":"2024-04-22T06:43:17.811215+0000","last_clean":"2024-04-22T06:43:17.811215+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:43:17.811215+0000","last_undegraded":"2024-04-22T06:43:17.811215+0000","last_fullsized":"2024-04-22T06:43:17.811215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083475+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:43:18.083475+0000","last_peered":"2024-04-22T06:43:18.083475+0000","last_clean":"2024-04-22T06:43:18.083475+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:43:18.083475+0000","last_undegraded":"2024-04-22T06:43:18.083475+0000","last_fullsized":"2024-04-22T06:43:18.083475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.814042+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:43:17.814042+0000","last_peered":"2024-04-22T06:43:17.814042+0000","last_clean":"2024-04-22T06:43:17.814042+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:43:17.814042+0000","last_undegraded":"2024-04-22T06:43:17.814042+0000","last_fullsized":"2024-04-22T06:43:17.814042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805476+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:43:17.805476+0000","last_peered":"2024-04-22T06:43:17.805476+0000","last_clean":"2024-04-22T06:43:17.805476+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:43:17.805476+0000","last_undegraded":"2024-04-22T06:43:17.805476+0000","last_fullsized":"2024-04-22T06:43:17.805476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069415+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:43:18.069415+0000","last_peered":"2024-04-22T06:43:18.069415+0000","last_clean":"2024-04-22T06:43:18.069415+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:43:18.069415+0000","last_undegraded":"2024-04-22T06:43:18.069415+0000","last_fullsized":"2024-04-22T06:43:18.069415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896503+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:43:17.896503+0000","last_peered":"2024-04-22T06:43:17.896503+0000","last_clean":"2024-04-22T06:43:17.896503+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:43:17.896503+0000","last_undegraded":"2024-04-22T06:43:17.896503+0000","last_fullsized":"2024-04-22T06:43:17.896503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804005+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:43:17.804005+0000","last_peered":"2024-04-22T06:43:17.804005+0000","last_clean":"2024-04-22T06:43:17.804005+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:43:17.804005+0000","last_undegraded":"2024-04-22T06:43:17.804005+0000","last_fullsized":"2024-04-22T06:43:17.804005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804002+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:43:17.804002+0000","last_peered":"2024-04-22T06:43:17.804002+0000","last_clean":"2024-04-22T06:43:17.804002+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:43:17.804002+0000","last_undegraded":"2024-04-22T06:43:17.804002+0000","last_fullsized":"2024-04-22T06:43:17.804002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.082949+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:43:18.082949+0000","last_peered":"2024-04-22T06:43:18.082949+0000","last_clean":"2024-04-22T06:43:18.082949+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:43:18.082949+0000","last_undegraded":"2024-04-22T06:43:18.082949+0000","last_fullsized":"2024-04-22T06:43:18.082949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.810078+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:43:17.810078+0000","last_peered":"2024-04-22T06:43:17.810078+0000","last_clean":"2024-04-22T06:43:17.810078+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:43:17.810078+0000","last_undegraded":"2024-04-22T06:43:17.810078+0000","last_fullsized":"2024-04-22T06:43:17.810078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805972+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:43:17.805972+0000","last_peered":"2024-04-22T06:43:17.805972+0000","last_clean":"2024-04-22T06:43:17.805972+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:43:17.805972+0000","last_undegraded":"2024-04-22T06:43:17.805972+0000","last_fullsized":"2024-04-22T06:43:17.805972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812666+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:43:17.812666+0000","last_peered":"2024-04-22T06:43:17.812666+0000","last_clean":"2024-04-22T06:43:17.812666+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:43:17.812666+0000","last_undegraded":"2024-04-22T06:43:17.812666+0000","last_fullsized":"2024-04-22T06:43:17.812666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.806012+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:43:17.806012+0000","last_peered":"2024-04-22T06:43:17.806012+0000","last_clean":"2024-04-22T06:43:17.806012+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:43:17.806012+0000","last_undegraded":"2024-04-22T06:43:17.806012+0000","last_fullsized":"2024-04-22T06:43:17.806012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811670+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:43:17.811670+0000","last_peered":"2024-04-22T06:43:17.811670+0000","last_clean":"2024-04-22T06:43:17.811670+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:43:17.811670+0000","last_undegraded":"2024-04-22T06:43:17.811670+0000","last_fullsized":"2024-04-22T06:43:17.811670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811771+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:43:17.811771+0000","last_peered":"2024-04-22T06:43:17.811771+0000","last_clean":"2024-04-22T06:43:17.811771+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:43:17.811771+0000","last_undegraded":"2024-04-22T06:43:17.811771+0000","last_fullsized":"2024-04-22T06:43:17.811771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804279+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:43:17.804279+0000","last_peered":"2024-04-22T06:43:17.804279+0000","last_clean":"2024-04-22T06:43:17.804279+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:43:17.804279+0000","last_undegraded":"2024-04-22T06:43:17.804279+0000","last_fullsized":"2024-04-22T06:43:17.804279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809241+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:43:17.809241+0000","last_peered":"2024-04-22T06:43:17.809241+0000","last_clean":"2024-04-22T06:43:17.809241+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:43:17.809241+0000","last_undegraded":"2024-04-22T06:43:17.809241+0000","last_fullsized":"2024-04-22T06:43:17.809241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083100+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:43:18.083100+0000","last_peered":"2024-04-22T06:43:18.083100+0000","last_clean":"2024-04-22T06:43:18.083100+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:43:18.083100+0000","last_undegraded":"2024-04-22T06:43:18.083100+0000","last_fullsized":"2024-04-22T06:43:18.083100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808735+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:43:17.808735+0000","last_peered":"2024-04-22T06:43:17.808735+0000","last_clean":"2024-04-22T06:43:17.808735+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:43:17.808735+0000","last_undegraded":"2024-04-22T06:43:17.808735+0000","last_fullsized":"2024-04-22T06:43:17.808735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.084628+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:43:18.084628+0000","last_peered":"2024-04-22T06:43:18.084628+0000","last_clean":"2024-04-22T06:43:18.084628+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:43:18.084628+0000","last_undegraded":"2024-04-22T06:43:18.084628+0000","last_fullsized":"2024-04-22T06:43:18.084628+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:37.820527+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:43:37.820527+0000","last_peered":"2024-04-22T06:43:37.820527+0000","last_clean":"2024-04-22T06:43:37.820527+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:43:37.820527+0000","last_undegraded":"2024-04-22T06:43:37.820527+0000","last_fullsized":"2024-04-22T06:43:37.820527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808182+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:43:17.808182+0000","last_peered":"2024-04-22T06:43:17.808182+0000","last_clean":"2024-04-22T06:43:17.808182+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:43:17.808182+0000","last_undegraded":"2024-04-22T06:43:17.808182+0000","last_fullsized":"2024-04-22T06:43:17.808182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808123+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:43:17.808123+0000","last_peered":"2024-04-22T06:43:17.808123+0000","last_clean":"2024-04-22T06:43:17.808123+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:43:17.808123+0000","last_undegraded":"2024-04-22T06:43:17.808123+0000","last_fullsized":"2024-04-22T06:43:17.808123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808954+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:43:17.808954+0000","last_peered":"2024-04-22T06:43:17.808954+0000","last_clean":"2024-04-22T06:43:17.808954+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:43:17.808954+0000","last_undegraded":"2024-04-22T06:43:17.808954+0000","last_fullsized":"2024-04-22T06:43:17.808954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896284+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:43:17.896284+0000","last_peered":"2024-04-22T06:43:17.896284+0000","last_clean":"2024-04-22T06:43:17.896284+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:43:17.896284+0000","last_undegraded":"2024-04-22T06:43:17.896284+0000","last_fullsized":"2024-04-22T06:43:17.896284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811322+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:43:17.811322+0000","last_peered":"2024-04-22T06:43:17.811322+0000","last_clean":"2024-04-22T06:43:17.811322+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:43:17.811322+0000","last_undegraded":"2024-04-22T06:43:17.811322+0000","last_fullsized":"2024-04-22T06:43:17.811322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808483+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:43:17.808483+0000","last_peered":"2024-04-22T06:43:17.808483+0000","last_clean":"2024-04-22T06:43:17.808483+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:43:17.808483+0000","last_undegraded":"2024-04-22T06:43:17.808483+0000","last_fullsized":"2024-04-22T06:43:17.808483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805795+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:43:17.805795+0000","last_peered":"2024-04-22T06:43:17.805795+0000","last_clean":"2024-04-22T06:43:17.805795+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:43:17.805795+0000","last_undegraded":"2024-04-22T06:43:17.805795+0000","last_fullsized":"2024-04-22T06:43:17.805795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:38.821349+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:43:38.821349+0000","last_peered":"2024-04-22T06:43:38.821349+0000","last_clean":"2024-04-22T06:43:38.821349+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:43:38.821349+0000","last_undegraded":"2024-04-22T06:43:38.821349+0000","last_fullsized":"2024-04-22T06:43:38.821349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083307+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:43:18.083307+0000","last_peered":"2024-04-22T06:43:18.083307+0000","last_clean":"2024-04-22T06:43:18.083307+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:43:18.083307+0000","last_undegraded":"2024-04-22T06:43:18.083307+0000","last_fullsized":"2024-04-22T06:43:18.083307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805520+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:43:17.805520+0000","last_peered":"2024-04-22T06:43:17.805520+0000","last_clean":"2024-04-22T06:43:17.805520+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:43:17.805520+0000","last_undegraded":"2024-04-22T06:43:17.805520+0000","last_fullsized":"2024-04-22T06:43:17.805520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804252+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:43:17.804252+0000","last_peered":"2024-04-22T06:43:17.804252+0000","last_clean":"2024-04-22T06:43:17.804252+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:43:17.804252+0000","last_undegraded":"2024-04-22T06:43:17.804252+0000","last_fullsized":"2024-04-22T06:43:17.804252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807998+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:43:17.807998+0000","last_peered":"2024-04-22T06:43:17.807998+0000","last_clean":"2024-04-22T06:43:17.807998+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:43:17.807998+0000","last_undegraded":"2024-04-22T06:43:17.807998+0000","last_fullsized":"2024-04-22T06:43:17.807998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083271+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:43:18.083271+0000","last_peered":"2024-04-22T06:43:18.083271+0000","last_clean":"2024-04-22T06:43:18.083271+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:43:18.083271+0000","last_undegraded":"2024-04-22T06:43:18.083271+0000","last_fullsized":"2024-04-22T06:43:18.083271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'26","reported_seq":196,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:39.822752+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:43:39.822752+0000","last_peered":"2024-04-22T06:43:39.822752+0000","last_clean":"2024-04-22T06:43:39.822752+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:43:39.822752+0000","last_undegraded":"2024-04-22T06:43:39.822752+0000","last_fullsized":"2024-04-22T06:43:39.822752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58841,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":132,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808949+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:43:17.808949+0000","last_peered":"2024-04-22T06:43:17.808949+0000","last_clean":"2024-04-22T06:43:17.808949+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:43:17.808949+0000","last_undegraded":"2024-04-22T06:43:17.808949+0000","last_fullsized":"2024-04-22T06:43:17.808949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812268+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:43:17.812268+0000","last_peered":"2024-04-22T06:43:17.812268+0000","last_clean":"2024-04-22T06:43:17.812268+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:43:17.812268+0000","last_undegraded":"2024-04-22T06:43:17.812268+0000","last_fullsized":"2024-04-22T06:43:17.812268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896362+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:43:17.896362+0000","last_peered":"2024-04-22T06:43:17.896362+0000","last_clean":"2024-04-22T06:43:17.896362+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:43:17.896362+0000","last_undegraded":"2024-04-22T06:43:17.896362+0000","last_fullsized":"2024-04-22T06:43:17.896362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807652+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:43:17.807652+0000","last_peered":"2024-04-22T06:43:17.807652+0000","last_clean":"2024-04-22T06:43:17.807652+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:43:17.807652+0000","last_undegraded":"2024-04-22T06:43:17.807652+0000","last_fullsized":"2024-04-22T06:43:17.807652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805442+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:43:17.805442+0000","last_peered":"2024-04-22T06:43:17.805442+0000","last_clean":"2024-04-22T06:43:17.805442+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:43:17.805442+0000","last_undegraded":"2024-04-22T06:43:17.805442+0000","last_fullsized":"2024-04-22T06:43:17.805442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:34.816445+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:43:34.816445+0000","last_peered":"2024-04-22T06:43:34.816445+0000","last_clean":"2024-04-22T06:43:34.816445+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:43:34.816445+0000","last_undegraded":"2024-04-22T06:43:34.816445+0000","last_fullsized":"2024-04-22T06:43:34.816445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805027+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:43:17.805027+0000","last_peered":"2024-04-22T06:43:17.805027+0000","last_clean":"2024-04-22T06:43:17.805027+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:43:17.805027+0000","last_undegraded":"2024-04-22T06:43:17.805027+0000","last_fullsized":"2024-04-22T06:43:17.805027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809785+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:43:17.809785+0000","last_peered":"2024-04-22T06:43:17.809785+0000","last_clean":"2024-04-22T06:43:17.809785+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:43:17.809785+0000","last_undegraded":"2024-04-22T06:43:17.809785+0000","last_fullsized":"2024-04-22T06:43:17.809785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808638+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:43:17.808638+0000","last_peered":"2024-04-22T06:43:17.808638+0000","last_clean":"2024-04-22T06:43:17.808638+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:43:17.808638+0000","last_undegraded":"2024-04-22T06:43:17.808638+0000","last_fullsized":"2024-04-22T06:43:17.808638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809643+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:43:17.809643+0000","last_peered":"2024-04-22T06:43:17.809643+0000","last_clean":"2024-04-22T06:43:17.809643+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:43:17.809643+0000","last_undegraded":"2024-04-22T06:43:17.809643+0000","last_fullsized":"2024-04-22T06:43:17.809643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809509+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:43:17.809509+0000","last_peered":"2024-04-22T06:43:17.809509+0000","last_clean":"2024-04-22T06:43:17.809509+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:43:17.809509+0000","last_undegraded":"2024-04-22T06:43:17.809509+0000","last_fullsized":"2024-04-22T06:43:17.809509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805908+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:43:17.805908+0000","last_peered":"2024-04-22T06:43:17.805908+0000","last_clean":"2024-04-22T06:43:17.805908+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:43:17.805908+0000","last_undegraded":"2024-04-22T06:43:17.805908+0000","last_fullsized":"2024-04-22T06:43:17.805908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808325+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:43:17.808325+0000","last_peered":"2024-04-22T06:43:17.808325+0000","last_clean":"2024-04-22T06:43:17.808325+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:43:17.808325+0000","last_undegraded":"2024-04-22T06:43:17.808325+0000","last_fullsized":"2024-04-22T06:43:17.808325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805230+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:43:17.805230+0000","last_peered":"2024-04-22T06:43:17.805230+0000","last_clean":"2024-04-22T06:43:17.805230+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:43:17.805230+0000","last_undegraded":"2024-04-22T06:43:17.805230+0000","last_fullsized":"2024-04-22T06:43:17.805230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808340+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:43:17.808340+0000","last_peered":"2024-04-22T06:43:17.808340+0000","last_clean":"2024-04-22T06:43:17.808340+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:43:17.808340+0000","last_undegraded":"2024-04-22T06:43:17.808340+0000","last_fullsized":"2024-04-22T06:43:17.808340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.897820+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:43:17.897820+0000","last_peered":"2024-04-22T06:43:17.897820+0000","last_clean":"2024-04-22T06:43:17.897820+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:43:17.897820+0000","last_undegraded":"2024-04-22T06:43:17.897820+0000","last_fullsized":"2024-04-22T06:43:17.897820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:34.815877+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:43:34.815877+0000","last_peered":"2024-04-22T06:43:34.815877+0000","last_clean":"2024-04-22T06:43:34.815877+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:43:34.815877+0000","last_undegraded":"2024-04-22T06:43:34.815877+0000","last_fullsized":"2024-04-22T06:43:34.815877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811485+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:43:17.811485+0000","last_peered":"2024-04-22T06:43:17.811485+0000","last_clean":"2024-04-22T06:43:17.811485+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:43:17.811485+0000","last_undegraded":"2024-04-22T06:43:17.811485+0000","last_fullsized":"2024-04-22T06:43:17.811485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069443+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:43:18.069443+0000","last_peered":"2024-04-22T06:43:18.069443+0000","last_clean":"2024-04-22T06:43:18.069443+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:43:18.069443+0000","last_undegraded":"2024-04-22T06:43:18.069443+0000","last_fullsized":"2024-04-22T06:43:18.069443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805111+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:43:17.805111+0000","last_peered":"2024-04-22T06:43:17.805111+0000","last_clean":"2024-04-22T06:43:17.805111+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:43:17.805111+0000","last_undegraded":"2024-04-22T06:43:17.805111+0000","last_fullsized":"2024-04-22T06:43:17.805111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896171+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:43:17.896171+0000","last_peered":"2024-04-22T06:43:17.896171+0000","last_clean":"2024-04-22T06:43:17.896171+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:43:17.896171+0000","last_undegraded":"2024-04-22T06:43:17.896171+0000","last_fullsized":"2024-04-22T06:43:17.896171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805229+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:43:17.805229+0000","last_peered":"2024-04-22T06:43:17.805229+0000","last_clean":"2024-04-22T06:43:17.805229+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:43:17.805229+0000","last_undegraded":"2024-04-22T06:43:17.805229+0000","last_fullsized":"2024-04-22T06:43:17.805229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809050+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:43:17.809050+0000","last_peered":"2024-04-22T06:43:17.809050+0000","last_clean":"2024-04-22T06:43:17.809050+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:43:17.809050+0000","last_undegraded":"2024-04-22T06:43:17.809050+0000","last_fullsized":"2024-04-22T06:43:17.809050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896578+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:43:17.896578+0000","last_peered":"2024-04-22T06:43:17.896578+0000","last_clean":"2024-04-22T06:43:17.896578+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:43:17.896578+0000","last_undegraded":"2024-04-22T06:43:17.896578+0000","last_fullsized":"2024-04-22T06:43:17.896578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.806040+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:43:17.806040+0000","last_peered":"2024-04-22T06:43:17.806040+0000","last_clean":"2024-04-22T06:43:17.806040+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:43:17.806040+0000","last_undegraded":"2024-04-22T06:43:17.806040+0000","last_fullsized":"2024-04-22T06:43:17.806040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811726+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:43:17.811726+0000","last_peered":"2024-04-22T06:43:17.811726+0000","last_clean":"2024-04-22T06:43:17.811726+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:43:17.811726+0000","last_undegraded":"2024-04-22T06:43:17.811726+0000","last_fullsized":"2024-04-22T06:43:17.811726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069685+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:43:18.069685+0000","last_peered":"2024-04-22T06:43:18.069685+0000","last_clean":"2024-04-22T06:43:18.069685+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:43:18.069685+0000","last_undegraded":"2024-04-22T06:43:18.069685+0000","last_fullsized":"2024-04-22T06:43:18.069685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804148+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:43:17.804148+0000","last_peered":"2024-04-22T06:43:17.804148+0000","last_clean":"2024-04-22T06:43:17.804148+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:43:17.804148+0000","last_undegraded":"2024-04-22T06:43:17.804148+0000","last_fullsized":"2024-04-22T06:43:17.804148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083030+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:43:18.083030+0000","last_peered":"2024-04-22T06:43:18.083030+0000","last_clean":"2024-04-22T06:43:18.083030+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:43:18.083030+0000","last_undegraded":"2024-04-22T06:43:18.083030+0000","last_fullsized":"2024-04-22T06:43:18.083030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807981+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:43:17.807981+0000","last_peered":"2024-04-22T06:43:17.807981+0000","last_clean":"2024-04-22T06:43:17.807981+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:43:17.807981+0000","last_undegraded":"2024-04-22T06:43:17.807981+0000","last_fullsized":"2024-04-22T06:43:17.807981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811318+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:43:17.811318+0000","last_peered":"2024-04-22T06:43:17.811318+0000","last_clean":"2024-04-22T06:43:17.811318+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:43:17.811318+0000","last_undegraded":"2024-04-22T06:43:17.811318+0000","last_fullsized":"2024-04-22T06:43:17.811318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807447+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:43:17.807447+0000","last_peered":"2024-04-22T06:43:17.807447+0000","last_clean":"2024-04-22T06:43:17.807447+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:43:17.807447+0000","last_undegraded":"2024-04-22T06:43:17.807447+0000","last_fullsized":"2024-04-22T06:43:17.807447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.817770+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:43:17.817770+0000","last_peered":"2024-04-22T06:43:17.817770+0000","last_clean":"2024-04-22T06:43:17.817770+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:43:17.817770+0000","last_undegraded":"2024-04-22T06:43:17.817770+0000","last_fullsized":"2024-04-22T06:43:17.817770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811529+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:43:17.811529+0000","last_peered":"2024-04-22T06:43:17.811529+0000","last_clean":"2024-04-22T06:43:17.811529+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:43:17.811529+0000","last_undegraded":"2024-04-22T06:43:17.811529+0000","last_fullsized":"2024-04-22T06:43:17.811529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.821217+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:43:17.821217+0000","last_peered":"2024-04-22T06:43:17.821217+0000","last_clean":"2024-04-22T06:43:17.821217+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:43:17.821217+0000","last_undegraded":"2024-04-22T06:43:17.821217+0000","last_fullsized":"2024-04-22T06:43:17.821217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805313+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:43:17.805313+0000","last_peered":"2024-04-22T06:43:17.805313+0000","last_clean":"2024-04-22T06:43:17.805313+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:43:17.805313+0000","last_undegraded":"2024-04-22T06:43:17.805313+0000","last_fullsized":"2024-04-22T06:43:17.805313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812035+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:43:17.812035+0000","last_peered":"2024-04-22T06:43:17.812035+0000","last_clean":"2024-04-22T06:43:17.812035+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:43:17.812035+0000","last_undegraded":"2024-04-22T06:43:17.812035+0000","last_fullsized":"2024-04-22T06:43:17.812035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805532+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:43:17.805532+0000","last_peered":"2024-04-22T06:43:17.805532+0000","last_clean":"2024-04-22T06:43:17.805532+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:43:17.805532+0000","last_undegraded":"2024-04-22T06:43:17.805532+0000","last_fullsized":"2024-04-22T06:43:17.805532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807695+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:43:17.807695+0000","last_peered":"2024-04-22T06:43:17.807695+0000","last_clean":"2024-04-22T06:43:17.807695+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:43:17.807695+0000","last_undegraded":"2024-04-22T06:43:17.807695+0000","last_fullsized":"2024-04-22T06:43:17.807695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805339+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:43:17.805339+0000","last_peered":"2024-04-22T06:43:17.805339+0000","last_clean":"2024-04-22T06:43:17.805339+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:43:17.805339+0000","last_undegraded":"2024-04-22T06:43:17.805339+0000","last_fullsized":"2024-04-22T06:43:17.805339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896491+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:43:17.896491+0000","last_peered":"2024-04-22T06:43:17.896491+0000","last_clean":"2024-04-22T06:43:17.896491+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:43:17.896491+0000","last_undegraded":"2024-04-22T06:43:17.896491+0000","last_fullsized":"2024-04-22T06:43:17.896491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805672+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:43:17.805672+0000","last_peered":"2024-04-22T06:43:17.805672+0000","last_clean":"2024-04-22T06:43:17.805672+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:43:17.805672+0000","last_undegraded":"2024-04-22T06:43:17.805672+0000","last_fullsized":"2024-04-22T06:43:17.805672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.814239+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:43:17.814239+0000","last_peered":"2024-04-22T06:43:17.814239+0000","last_clean":"2024-04-22T06:43:17.814239+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:43:17.814239+0000","last_undegraded":"2024-04-22T06:43:17.814239+0000","last_fullsized":"2024-04-22T06:43:17.814239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.814744+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:43:17.814744+0000","last_peered":"2024-04-22T06:43:17.814744+0000","last_clean":"2024-04-22T06:43:17.814744+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:43:17.814744+0000","last_undegraded":"2024-04-22T06:43:17.814744+0000","last_fullsized":"2024-04-22T06:43:17.814744+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812343+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:43:17.812343+0000","last_peered":"2024-04-22T06:43:17.812343+0000","last_clean":"2024-04-22T06:43:17.812343+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:43:17.812343+0000","last_undegraded":"2024-04-22T06:43:17.812343+0000","last_fullsized":"2024-04-22T06:43:17.812343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804143+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:43:17.804143+0000","last_peered":"2024-04-22T06:43:17.804143+0000","last_clean":"2024-04-22T06:43:17.804143+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:43:17.804143+0000","last_undegraded":"2024-04-22T06:43:17.804143+0000","last_fullsized":"2024-04-22T06:43:17.804143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896436+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:43:17.896436+0000","last_peered":"2024-04-22T06:43:17.896436+0000","last_clean":"2024-04-22T06:43:17.896436+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:43:17.896436+0000","last_undegraded":"2024-04-22T06:43:17.896436+0000","last_fullsized":"2024-04-22T06:43:17.896436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809303+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:43:17.809303+0000","last_peered":"2024-04-22T06:43:17.809303+0000","last_clean":"2024-04-22T06:43:17.809303+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:43:17.809303+0000","last_undegraded":"2024-04-22T06:43:17.809303+0000","last_fullsized":"2024-04-22T06:43:17.809303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812864+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:43:17.812864+0000","last_peered":"2024-04-22T06:43:17.812864+0000","last_clean":"2024-04-22T06:43:17.812864+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:43:17.812864+0000","last_undegraded":"2024-04-22T06:43:17.812864+0000","last_fullsized":"2024-04-22T06:43:17.812864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811699+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:43:17.811699+0000","last_peered":"2024-04-22T06:43:17.811699+0000","last_clean":"2024-04-22T06:43:17.811699+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:43:17.811699+0000","last_undegraded":"2024-04-22T06:43:17.811699+0000","last_fullsized":"2024-04-22T06:43:17.811699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812891+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:43:17.812891+0000","last_peered":"2024-04-22T06:43:17.812891+0000","last_clean":"2024-04-22T06:43:17.812891+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:43:17.812891+0000","last_undegraded":"2024-04-22T06:43:17.812891+0000","last_fullsized":"2024-04-22T06:43:17.812891+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811369+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:43:17.811369+0000","last_peered":"2024-04-22T06:43:17.811369+0000","last_clean":"2024-04-22T06:43:17.811369+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:43:17.811369+0000","last_undegraded":"2024-04-22T06:43:17.811369+0000","last_fullsized":"2024-04-22T06:43:17.811369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083107+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:43:18.083107+0000","last_peered":"2024-04-22T06:43:18.083107+0000","last_clean":"2024-04-22T06:43:18.083107+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:43:18.083107+0000","last_undegraded":"2024-04-22T06:43:18.083107+0000","last_fullsized":"2024-04-22T06:43:18.083107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896446+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:43:17.896446+0000","last_peered":"2024-04-22T06:43:17.896446+0000","last_clean":"2024-04-22T06:43:17.896446+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:43:17.896446+0000","last_undegraded":"2024-04-22T06:43:17.896446+0000","last_fullsized":"2024-04-22T06:43:17.896446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809029+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:43:17.809029+0000","last_peered":"2024-04-22T06:43:17.809029+0000","last_clean":"2024-04-22T06:43:17.809029+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:43:17.809029+0000","last_undegraded":"2024-04-22T06:43:17.809029+0000","last_fullsized":"2024-04-22T06:43:17.809029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63233,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":448,"num_read_kb":290,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":189699,"data_compressed":5622,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":1106910,"data_compressed":37581,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":1048674,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495671,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1003520,"data_stored":514456,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":516197,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":456650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":1047967,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":58841,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":60582,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":60810,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:43:41.645 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T06:43:41.920 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:41.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:41 smithi079 ceph-mon[27946]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:42.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:41 smithi193 ceph-mon[31815]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:42.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:41 smithi190 ceph-mon[31421]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:43.056 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:42 smithi079 ceph-mon[27946]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:42 smithi193 ceph-mon[31815]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:43.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:42 smithi190 ceph-mon[31421]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:43.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.593+0000 7fc2dc394700 1 -- 172.21.15.79:0/3914242394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 msgr2=0x7fc2d410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.593+0000 7fc2dc394700 1 --2- 172.21.15.79:0/3914242394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d410f8b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00b0a0 tx=0x7fc2cc00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.593+0000 7fc2dc394700 1 -- 172.21.15.79:0/3914242394 shutdown_connections 2024-04-22T06:43:43.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.593+0000 7fc2dc394700 1 --2- 172.21.15.79:0/3914242394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d410f8b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.593+0000 7fc2dc394700 1 --2- 172.21.15.79:0/3914242394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4108a50 0x7fc2d4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.593+0000 7fc2dc394700 1 --2- 172.21.15.79:0/3914242394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d4102a90 0x7fc2d4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- 172.21.15.79:0/3914242394 >> 172.21.15.79:0/3914242394 conn(0x7fc2d4079ba0 msgr2=0x7fc2d4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- 172.21.15.79:0/3914242394 shutdown_connections 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- 172.21.15.79:0/3914242394 wait complete. 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 Processor -- start 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- start start 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4102a90 0x7fc2d4199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d4199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d4108a50 0x7fc2d419db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4112ce0 con 0x7fc2d4103450 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4112b60 con 0x7fc2d4108a50 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2dc394700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4112e60 con 0x7fc2d4102a90 2024-04-22T06:43:43.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2d992f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d4199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.594+0000 7fc2d992f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d4199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58062/0 (socket says 172.21.15.79:58062) 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 -- 172.21.15.79:0/1528525116 learned_addr learned my addr 172.21.15.79:0/1528525116 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2da931700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d4108a50 0x7fc2d419db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2da130700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4102a90 0x7fc2d4199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 -- 172.21.15.79:0/1528525116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4102a90 msgr2=0x7fc2d4199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4102a90 0x7fc2d4199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 -- 172.21.15.79:0/1528525116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d4108a50 msgr2=0x7fc2d419db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d4108a50 0x7fc2d419db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 -- 172.21.15.79:0/1528525116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d419e240 con 0x7fc2d4103450 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2da130700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4102a90 0x7fc2d4199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2d992f700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d4199770 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400b530 tx=0x7fc2c400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4015020 con 0x7fc2d4103450 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2c4011ce0 con 0x7fc2d4103450 2024-04-22T06:43:43.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d419e530 con 0x7fc2d4103450 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4010d20 con 0x7fc2d4103450 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.595+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d41a5db0 con 0x7fc2d4103450 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.596+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d4069730 con 0x7fc2d4103450 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.597+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc2c4018020 con 0x7fc2d4103450 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.598+0000 7fc2cb7fe700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2c00a3cd0 0x7fc2c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.598+0000 7fc2da130700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2c00a3cd0 0x7fc2c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.598+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc2c40d01f0 con 0x7fc2d4103450 2024-04-22T06:43:43.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.598+0000 7fc2da130700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2c00a3cd0 0x7fc2c00a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0007900 tx=0x7fc2d0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:43.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.601+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2c401f020 con 0x7fc2d4103450 2024-04-22T06:43:43.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.796+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2d4062e80 con 0x7fc2c00a3cd0 2024-04-22T06:43:43.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.803+0000 7fc2cb7fe700 1 -- 172.21.15.79:0/1528525116 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258930 (secure 0 0 0) 0x7fc2d4062e80 con 0x7fc2c00a3cd0 2024-04-22T06:43:43.805 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2c00a3cd0 msgr2=0x7fc2c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2c00a3cd0 0x7fc2c00a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0007900 tx=0x7fc2d0008040 comp rx=0 tx=0).stop 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 msgr2=0x7fc2d4199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d4199770 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400b530 tx=0x7fc2c400b8f0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 shutdown_connections 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2d4108a50 0x7fc2d419db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2d4102a90 0x7fc2d4199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc2c00a3cd0 0x7fc2c00a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 --2- 172.21.15.79:0/1528525116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc2d4103450 0x7fc2d4199770 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.806+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 >> 172.21.15.79:0/1528525116 conn(0x7fc2d4079ba0 msgr2=0x7fc2d41017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.807+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 shutdown_connections 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:43.807+0000 7fc2dc394700 1 -- 172.21.15.79:0/1528525116 wait complete. 2024-04-22T06:43:43.807 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:43:44.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:43 smithi193 ceph-mon[31815]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:44.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:43 smithi190 ceph-mon[31421]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:44.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:43 smithi079 ceph-mon[27946]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:44.605 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-04-22T06:43:42.664014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653835,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":594,"num_read_kb":414,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381580,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560244,"statfs":{"total":1151940427776,"available":1151549689856,"internally_reserved":0,"allocated":11833344,"data_stored":7428885,"data_compressed":112743,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003207"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809965+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:43:17.809965+0000","last_peered":"2024-04-22T06:43:17.809965+0000","last_clean":"2024-04-22T06:43:17.809965+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:43:17.809965+0000","last_undegraded":"2024-04-22T06:43:17.809965+0000","last_fullsized":"2024-04-22T06:43:17.809965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804244+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:43:17.804244+0000","last_peered":"2024-04-22T06:43:17.804244+0000","last_clean":"2024-04-22T06:43:17.804244+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:43:17.804244+0000","last_undegraded":"2024-04-22T06:43:17.804244+0000","last_fullsized":"2024-04-22T06:43:17.804244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804277+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:43:17.804277+0000","last_peered":"2024-04-22T06:43:17.804277+0000","last_clean":"2024-04-22T06:43:17.804277+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:43:17.804277+0000","last_undegraded":"2024-04-22T06:43:17.804277+0000","last_fullsized":"2024-04-22T06:43:17.804277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069465+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:43:18.069465+0000","last_peered":"2024-04-22T06:43:18.069465+0000","last_clean":"2024-04-22T06:43:18.069465+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:43:18.069465+0000","last_undegraded":"2024-04-22T06:43:18.069465+0000","last_fullsized":"2024-04-22T06:43:18.069465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807365+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:43:17.807365+0000","last_peered":"2024-04-22T06:43:17.807365+0000","last_clean":"2024-04-22T06:43:17.807365+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:43:17.807365+0000","last_undegraded":"2024-04-22T06:43:17.807365+0000","last_fullsized":"2024-04-22T06:43:17.807365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808320+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:43:17.808320+0000","last_peered":"2024-04-22T06:43:17.808320+0000","last_clean":"2024-04-22T06:43:17.808320+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:43:17.808320+0000","last_undegraded":"2024-04-22T06:43:17.808320+0000","last_fullsized":"2024-04-22T06:43:17.808320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805099+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:43:17.805099+0000","last_peered":"2024-04-22T06:43:17.805099+0000","last_clean":"2024-04-22T06:43:17.805099+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:43:17.805099+0000","last_undegraded":"2024-04-22T06:43:17.805099+0000","last_fullsized":"2024-04-22T06:43:17.805099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811215+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:43:17.811215+0000","last_peered":"2024-04-22T06:43:17.811215+0000","last_clean":"2024-04-22T06:43:17.811215+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:43:17.811215+0000","last_undegraded":"2024-04-22T06:43:17.811215+0000","last_fullsized":"2024-04-22T06:43:17.811215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083475+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:43:18.083475+0000","last_peered":"2024-04-22T06:43:18.083475+0000","last_clean":"2024-04-22T06:43:18.083475+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:43:18.083475+0000","last_undegraded":"2024-04-22T06:43:18.083475+0000","last_fullsized":"2024-04-22T06:43:18.083475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.814042+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:43:17.814042+0000","last_peered":"2024-04-22T06:43:17.814042+0000","last_clean":"2024-04-22T06:43:17.814042+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:43:17.814042+0000","last_undegraded":"2024-04-22T06:43:17.814042+0000","last_fullsized":"2024-04-22T06:43:17.814042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805476+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:43:17.805476+0000","last_peered":"2024-04-22T06:43:17.805476+0000","last_clean":"2024-04-22T06:43:17.805476+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:43:17.805476+0000","last_undegraded":"2024-04-22T06:43:17.805476+0000","last_fullsized":"2024-04-22T06:43:17.805476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069415+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:43:18.069415+0000","last_peered":"2024-04-22T06:43:18.069415+0000","last_clean":"2024-04-22T06:43:18.069415+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:43:18.069415+0000","last_undegraded":"2024-04-22T06:43:18.069415+0000","last_fullsized":"2024-04-22T06:43:18.069415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896503+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:43:17.896503+0000","last_peered":"2024-04-22T06:43:17.896503+0000","last_clean":"2024-04-22T06:43:17.896503+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:43:17.896503+0000","last_undegraded":"2024-04-22T06:43:17.896503+0000","last_fullsized":"2024-04-22T06:43:17.896503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804005+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:43:17.804005+0000","last_peered":"2024-04-22T06:43:17.804005+0000","last_clean":"2024-04-22T06:43:17.804005+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:43:17.804005+0000","last_undegraded":"2024-04-22T06:43:17.804005+0000","last_fullsized":"2024-04-22T06:43:17.804005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804002+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:43:17.804002+0000","last_peered":"2024-04-22T06:43:17.804002+0000","last_clean":"2024-04-22T06:43:17.804002+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:43:17.804002+0000","last_undegraded":"2024-04-22T06:43:17.804002+0000","last_fullsized":"2024-04-22T06:43:17.804002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.082949+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:43:18.082949+0000","last_peered":"2024-04-22T06:43:18.082949+0000","last_clean":"2024-04-22T06:43:18.082949+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:43:18.082949+0000","last_undegraded":"2024-04-22T06:43:18.082949+0000","last_fullsized":"2024-04-22T06:43:18.082949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.810078+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:43:17.810078+0000","last_peered":"2024-04-22T06:43:17.810078+0000","last_clean":"2024-04-22T06:43:17.810078+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:43:17.810078+0000","last_undegraded":"2024-04-22T06:43:17.810078+0000","last_fullsized":"2024-04-22T06:43:17.810078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805972+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:43:17.805972+0000","last_peered":"2024-04-22T06:43:17.805972+0000","last_clean":"2024-04-22T06:43:17.805972+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:43:17.805972+0000","last_undegraded":"2024-04-22T06:43:17.805972+0000","last_fullsized":"2024-04-22T06:43:17.805972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812666+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:43:17.812666+0000","last_peered":"2024-04-22T06:43:17.812666+0000","last_clean":"2024-04-22T06:43:17.812666+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:43:17.812666+0000","last_undegraded":"2024-04-22T06:43:17.812666+0000","last_fullsized":"2024-04-22T06:43:17.812666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.806012+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:43:17.806012+0000","last_peered":"2024-04-22T06:43:17.806012+0000","last_clean":"2024-04-22T06:43:17.806012+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:43:17.806012+0000","last_undegraded":"2024-04-22T06:43:17.806012+0000","last_fullsized":"2024-04-22T06:43:17.806012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811670+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:43:17.811670+0000","last_peered":"2024-04-22T06:43:17.811670+0000","last_clean":"2024-04-22T06:43:17.811670+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:43:17.811670+0000","last_undegraded":"2024-04-22T06:43:17.811670+0000","last_fullsized":"2024-04-22T06:43:17.811670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811771+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:43:17.811771+0000","last_peered":"2024-04-22T06:43:17.811771+0000","last_clean":"2024-04-22T06:43:17.811771+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:43:17.811771+0000","last_undegraded":"2024-04-22T06:43:17.811771+0000","last_fullsized":"2024-04-22T06:43:17.811771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804279+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:43:17.804279+0000","last_peered":"2024-04-22T06:43:17.804279+0000","last_clean":"2024-04-22T06:43:17.804279+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:43:17.804279+0000","last_undegraded":"2024-04-22T06:43:17.804279+0000","last_fullsized":"2024-04-22T06:43:17.804279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809241+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:43:17.809241+0000","last_peered":"2024-04-22T06:43:17.809241+0000","last_clean":"2024-04-22T06:43:17.809241+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:43:17.809241+0000","last_undegraded":"2024-04-22T06:43:17.809241+0000","last_fullsized":"2024-04-22T06:43:17.809241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083100+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:43:18.083100+0000","last_peered":"2024-04-22T06:43:18.083100+0000","last_clean":"2024-04-22T06:43:18.083100+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:43:18.083100+0000","last_undegraded":"2024-04-22T06:43:18.083100+0000","last_fullsized":"2024-04-22T06:43:18.083100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808735+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:43:17.808735+0000","last_peered":"2024-04-22T06:43:17.808735+0000","last_clean":"2024-04-22T06:43:17.808735+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:43:17.808735+0000","last_undegraded":"2024-04-22T06:43:17.808735+0000","last_fullsized":"2024-04-22T06:43:17.808735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.084628+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:43:18.084628+0000","last_peered":"2024-04-22T06:43:18.084628+0000","last_clean":"2024-04-22T06:43:18.084628+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:43:18.084628+0000","last_undegraded":"2024-04-22T06:43:18.084628+0000","last_fullsized":"2024-04-22T06:43:18.084628+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:37.820527+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:43:37.820527+0000","last_peered":"2024-04-22T06:43:37.820527+0000","last_clean":"2024-04-22T06:43:37.820527+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:43:37.820527+0000","last_undegraded":"2024-04-22T06:43:37.820527+0000","last_fullsized":"2024-04-22T06:43:37.820527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808182+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:43:17.808182+0000","last_peered":"2024-04-22T06:43:17.808182+0000","last_clean":"2024-04-22T06:43:17.808182+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:43:17.808182+0000","last_undegraded":"2024-04-22T06:43:17.808182+0000","last_fullsized":"2024-04-22T06:43:17.808182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808123+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:43:17.808123+0000","last_peered":"2024-04-22T06:43:17.808123+0000","last_clean":"2024-04-22T06:43:17.808123+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:43:17.808123+0000","last_undegraded":"2024-04-22T06:43:17.808123+0000","last_fullsized":"2024-04-22T06:43:17.808123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808954+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:43:17.808954+0000","last_peered":"2024-04-22T06:43:17.808954+0000","last_clean":"2024-04-22T06:43:17.808954+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:43:17.808954+0000","last_undegraded":"2024-04-22T06:43:17.808954+0000","last_fullsized":"2024-04-22T06:43:17.808954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896284+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:43:17.896284+0000","last_peered":"2024-04-22T06:43:17.896284+0000","last_clean":"2024-04-22T06:43:17.896284+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:43:17.896284+0000","last_undegraded":"2024-04-22T06:43:17.896284+0000","last_fullsized":"2024-04-22T06:43:17.896284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811322+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:43:17.811322+0000","last_peered":"2024-04-22T06:43:17.811322+0000","last_clean":"2024-04-22T06:43:17.811322+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:43:17.811322+0000","last_undegraded":"2024-04-22T06:43:17.811322+0000","last_fullsized":"2024-04-22T06:43:17.811322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808483+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:43:17.808483+0000","last_peered":"2024-04-22T06:43:17.808483+0000","last_clean":"2024-04-22T06:43:17.808483+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:43:17.808483+0000","last_undegraded":"2024-04-22T06:43:17.808483+0000","last_fullsized":"2024-04-22T06:43:17.808483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805795+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:43:17.805795+0000","last_peered":"2024-04-22T06:43:17.805795+0000","last_clean":"2024-04-22T06:43:17.805795+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:43:17.805795+0000","last_undegraded":"2024-04-22T06:43:17.805795+0000","last_fullsized":"2024-04-22T06:43:17.805795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:38.821349+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:43:38.821349+0000","last_peered":"2024-04-22T06:43:38.821349+0000","last_clean":"2024-04-22T06:43:38.821349+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:43:38.821349+0000","last_undegraded":"2024-04-22T06:43:38.821349+0000","last_fullsized":"2024-04-22T06:43:38.821349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083307+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:43:18.083307+0000","last_peered":"2024-04-22T06:43:18.083307+0000","last_clean":"2024-04-22T06:43:18.083307+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:43:18.083307+0000","last_undegraded":"2024-04-22T06:43:18.083307+0000","last_fullsized":"2024-04-22T06:43:18.083307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805520+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:43:17.805520+0000","last_peered":"2024-04-22T06:43:17.805520+0000","last_clean":"2024-04-22T06:43:17.805520+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:43:17.805520+0000","last_undegraded":"2024-04-22T06:43:17.805520+0000","last_fullsized":"2024-04-22T06:43:17.805520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804252+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:43:17.804252+0000","last_peered":"2024-04-22T06:43:17.804252+0000","last_clean":"2024-04-22T06:43:17.804252+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:43:17.804252+0000","last_undegraded":"2024-04-22T06:43:17.804252+0000","last_fullsized":"2024-04-22T06:43:17.804252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807998+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:43:17.807998+0000","last_peered":"2024-04-22T06:43:17.807998+0000","last_clean":"2024-04-22T06:43:17.807998+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:43:17.807998+0000","last_undegraded":"2024-04-22T06:43:17.807998+0000","last_fullsized":"2024-04-22T06:43:17.807998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083271+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:43:18.083271+0000","last_peered":"2024-04-22T06:43:18.083271+0000","last_clean":"2024-04-22T06:43:18.083271+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:43:18.083271+0000","last_undegraded":"2024-04-22T06:43:18.083271+0000","last_fullsized":"2024-04-22T06:43:18.083271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'26","reported_seq":196,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:39.822752+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:43:39.822752+0000","last_peered":"2024-04-22T06:43:39.822752+0000","last_clean":"2024-04-22T06:43:39.822752+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:43:39.822752+0000","last_undegraded":"2024-04-22T06:43:39.822752+0000","last_fullsized":"2024-04-22T06:43:39.822752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58841,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":151,"num_read_kb":132,"num_write":27,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808949+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:43:17.808949+0000","last_peered":"2024-04-22T06:43:17.808949+0000","last_clean":"2024-04-22T06:43:17.808949+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:43:17.808949+0000","last_undegraded":"2024-04-22T06:43:17.808949+0000","last_fullsized":"2024-04-22T06:43:17.808949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812268+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:43:17.812268+0000","last_peered":"2024-04-22T06:43:17.812268+0000","last_clean":"2024-04-22T06:43:17.812268+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:43:17.812268+0000","last_undegraded":"2024-04-22T06:43:17.812268+0000","last_fullsized":"2024-04-22T06:43:17.812268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896362+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:43:17.896362+0000","last_peered":"2024-04-22T06:43:17.896362+0000","last_clean":"2024-04-22T06:43:17.896362+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:43:17.896362+0000","last_undegraded":"2024-04-22T06:43:17.896362+0000","last_fullsized":"2024-04-22T06:43:17.896362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807652+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:43:17.807652+0000","last_peered":"2024-04-22T06:43:17.807652+0000","last_clean":"2024-04-22T06:43:17.807652+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:43:17.807652+0000","last_undegraded":"2024-04-22T06:43:17.807652+0000","last_fullsized":"2024-04-22T06:43:17.807652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805442+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:43:17.805442+0000","last_peered":"2024-04-22T06:43:17.805442+0000","last_clean":"2024-04-22T06:43:17.805442+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:43:17.805442+0000","last_undegraded":"2024-04-22T06:43:17.805442+0000","last_fullsized":"2024-04-22T06:43:17.805442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:39.823527+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:43:39.823527+0000","last_peered":"2024-04-22T06:43:39.823527+0000","last_clean":"2024-04-22T06:43:39.823527+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:43:39.823527+0000","last_undegraded":"2024-04-22T06:43:39.823527+0000","last_fullsized":"2024-04-22T06:43:39.823527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805027+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:43:17.805027+0000","last_peered":"2024-04-22T06:43:17.805027+0000","last_clean":"2024-04-22T06:43:17.805027+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:43:17.805027+0000","last_undegraded":"2024-04-22T06:43:17.805027+0000","last_fullsized":"2024-04-22T06:43:17.805027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809785+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:43:17.809785+0000","last_peered":"2024-04-22T06:43:17.809785+0000","last_clean":"2024-04-22T06:43:17.809785+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:43:17.809785+0000","last_undegraded":"2024-04-22T06:43:17.809785+0000","last_fullsized":"2024-04-22T06:43:17.809785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808638+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:43:17.808638+0000","last_peered":"2024-04-22T06:43:17.808638+0000","last_clean":"2024-04-22T06:43:17.808638+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:43:17.808638+0000","last_undegraded":"2024-04-22T06:43:17.808638+0000","last_fullsized":"2024-04-22T06:43:17.808638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809643+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:43:17.809643+0000","last_peered":"2024-04-22T06:43:17.809643+0000","last_clean":"2024-04-22T06:43:17.809643+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:43:17.809643+0000","last_undegraded":"2024-04-22T06:43:17.809643+0000","last_fullsized":"2024-04-22T06:43:17.809643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809509+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:43:17.809509+0000","last_peered":"2024-04-22T06:43:17.809509+0000","last_clean":"2024-04-22T06:43:17.809509+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:43:17.809509+0000","last_undegraded":"2024-04-22T06:43:17.809509+0000","last_fullsized":"2024-04-22T06:43:17.809509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805908+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:43:17.805908+0000","last_peered":"2024-04-22T06:43:17.805908+0000","last_clean":"2024-04-22T06:43:17.805908+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:43:17.805908+0000","last_undegraded":"2024-04-22T06:43:17.805908+0000","last_fullsized":"2024-04-22T06:43:17.805908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808325+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:43:17.808325+0000","last_peered":"2024-04-22T06:43:17.808325+0000","last_clean":"2024-04-22T06:43:17.808325+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:43:17.808325+0000","last_undegraded":"2024-04-22T06:43:17.808325+0000","last_fullsized":"2024-04-22T06:43:17.808325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805230+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:43:17.805230+0000","last_peered":"2024-04-22T06:43:17.805230+0000","last_clean":"2024-04-22T06:43:17.805230+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:43:17.805230+0000","last_undegraded":"2024-04-22T06:43:17.805230+0000","last_fullsized":"2024-04-22T06:43:17.805230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.808340+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:43:17.808340+0000","last_peered":"2024-04-22T06:43:17.808340+0000","last_clean":"2024-04-22T06:43:17.808340+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:43:17.808340+0000","last_undegraded":"2024-04-22T06:43:17.808340+0000","last_fullsized":"2024-04-22T06:43:17.808340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.897820+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:43:17.897820+0000","last_peered":"2024-04-22T06:43:17.897820+0000","last_clean":"2024-04-22T06:43:17.897820+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:43:17.897820+0000","last_undegraded":"2024-04-22T06:43:17.897820+0000","last_fullsized":"2024-04-22T06:43:17.897820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:39.822906+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:43:39.822906+0000","last_peered":"2024-04-22T06:43:39.822906+0000","last_clean":"2024-04-22T06:43:39.822906+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:43:39.822906+0000","last_undegraded":"2024-04-22T06:43:39.822906+0000","last_fullsized":"2024-04-22T06:43:39.822906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811485+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:43:17.811485+0000","last_peered":"2024-04-22T06:43:17.811485+0000","last_clean":"2024-04-22T06:43:17.811485+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:43:17.811485+0000","last_undegraded":"2024-04-22T06:43:17.811485+0000","last_fullsized":"2024-04-22T06:43:17.811485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069443+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:43:18.069443+0000","last_peered":"2024-04-22T06:43:18.069443+0000","last_clean":"2024-04-22T06:43:18.069443+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:43:18.069443+0000","last_undegraded":"2024-04-22T06:43:18.069443+0000","last_fullsized":"2024-04-22T06:43:18.069443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805111+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:43:17.805111+0000","last_peered":"2024-04-22T06:43:17.805111+0000","last_clean":"2024-04-22T06:43:17.805111+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:43:17.805111+0000","last_undegraded":"2024-04-22T06:43:17.805111+0000","last_fullsized":"2024-04-22T06:43:17.805111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896171+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:43:17.896171+0000","last_peered":"2024-04-22T06:43:17.896171+0000","last_clean":"2024-04-22T06:43:17.896171+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:43:17.896171+0000","last_undegraded":"2024-04-22T06:43:17.896171+0000","last_fullsized":"2024-04-22T06:43:17.896171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805229+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:43:17.805229+0000","last_peered":"2024-04-22T06:43:17.805229+0000","last_clean":"2024-04-22T06:43:17.805229+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:43:17.805229+0000","last_undegraded":"2024-04-22T06:43:17.805229+0000","last_fullsized":"2024-04-22T06:43:17.805229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809050+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:43:17.809050+0000","last_peered":"2024-04-22T06:43:17.809050+0000","last_clean":"2024-04-22T06:43:17.809050+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:43:17.809050+0000","last_undegraded":"2024-04-22T06:43:17.809050+0000","last_fullsized":"2024-04-22T06:43:17.809050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896578+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:43:17.896578+0000","last_peered":"2024-04-22T06:43:17.896578+0000","last_clean":"2024-04-22T06:43:17.896578+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:43:17.896578+0000","last_undegraded":"2024-04-22T06:43:17.896578+0000","last_fullsized":"2024-04-22T06:43:17.896578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.806040+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:43:17.806040+0000","last_peered":"2024-04-22T06:43:17.806040+0000","last_clean":"2024-04-22T06:43:17.806040+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:43:17.806040+0000","last_undegraded":"2024-04-22T06:43:17.806040+0000","last_fullsized":"2024-04-22T06:43:17.806040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811726+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:43:17.811726+0000","last_peered":"2024-04-22T06:43:17.811726+0000","last_clean":"2024-04-22T06:43:17.811726+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:43:17.811726+0000","last_undegraded":"2024-04-22T06:43:17.811726+0000","last_fullsized":"2024-04-22T06:43:17.811726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.069685+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:43:18.069685+0000","last_peered":"2024-04-22T06:43:18.069685+0000","last_clean":"2024-04-22T06:43:18.069685+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:43:18.069685+0000","last_undegraded":"2024-04-22T06:43:18.069685+0000","last_fullsized":"2024-04-22T06:43:18.069685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804148+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:43:17.804148+0000","last_peered":"2024-04-22T06:43:17.804148+0000","last_clean":"2024-04-22T06:43:17.804148+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:43:17.804148+0000","last_undegraded":"2024-04-22T06:43:17.804148+0000","last_fullsized":"2024-04-22T06:43:17.804148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083030+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:43:18.083030+0000","last_peered":"2024-04-22T06:43:18.083030+0000","last_clean":"2024-04-22T06:43:18.083030+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:43:18.083030+0000","last_undegraded":"2024-04-22T06:43:18.083030+0000","last_fullsized":"2024-04-22T06:43:18.083030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807981+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:43:17.807981+0000","last_peered":"2024-04-22T06:43:17.807981+0000","last_clean":"2024-04-22T06:43:17.807981+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:43:17.807981+0000","last_undegraded":"2024-04-22T06:43:17.807981+0000","last_fullsized":"2024-04-22T06:43:17.807981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811318+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:43:17.811318+0000","last_peered":"2024-04-22T06:43:17.811318+0000","last_clean":"2024-04-22T06:43:17.811318+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:43:17.811318+0000","last_undegraded":"2024-04-22T06:43:17.811318+0000","last_fullsized":"2024-04-22T06:43:17.811318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807447+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:43:17.807447+0000","last_peered":"2024-04-22T06:43:17.807447+0000","last_clean":"2024-04-22T06:43:17.807447+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:43:17.807447+0000","last_undegraded":"2024-04-22T06:43:17.807447+0000","last_fullsized":"2024-04-22T06:43:17.807447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.817770+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:43:17.817770+0000","last_peered":"2024-04-22T06:43:17.817770+0000","last_clean":"2024-04-22T06:43:17.817770+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:43:17.817770+0000","last_undegraded":"2024-04-22T06:43:17.817770+0000","last_fullsized":"2024-04-22T06:43:17.817770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811529+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:43:17.811529+0000","last_peered":"2024-04-22T06:43:17.811529+0000","last_clean":"2024-04-22T06:43:17.811529+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:43:17.811529+0000","last_undegraded":"2024-04-22T06:43:17.811529+0000","last_fullsized":"2024-04-22T06:43:17.811529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.821217+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:43:17.821217+0000","last_peered":"2024-04-22T06:43:17.821217+0000","last_clean":"2024-04-22T06:43:17.821217+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:43:17.821217+0000","last_undegraded":"2024-04-22T06:43:17.821217+0000","last_fullsized":"2024-04-22T06:43:17.821217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805313+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:43:17.805313+0000","last_peered":"2024-04-22T06:43:17.805313+0000","last_clean":"2024-04-22T06:43:17.805313+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:43:17.805313+0000","last_undegraded":"2024-04-22T06:43:17.805313+0000","last_fullsized":"2024-04-22T06:43:17.805313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812035+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:43:17.812035+0000","last_peered":"2024-04-22T06:43:17.812035+0000","last_clean":"2024-04-22T06:43:17.812035+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:43:17.812035+0000","last_undegraded":"2024-04-22T06:43:17.812035+0000","last_fullsized":"2024-04-22T06:43:17.812035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805532+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:43:17.805532+0000","last_peered":"2024-04-22T06:43:17.805532+0000","last_clean":"2024-04-22T06:43:17.805532+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:43:17.805532+0000","last_undegraded":"2024-04-22T06:43:17.805532+0000","last_fullsized":"2024-04-22T06:43:17.805532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.807695+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:43:17.807695+0000","last_peered":"2024-04-22T06:43:17.807695+0000","last_clean":"2024-04-22T06:43:17.807695+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:43:17.807695+0000","last_undegraded":"2024-04-22T06:43:17.807695+0000","last_fullsized":"2024-04-22T06:43:17.807695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805339+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:43:17.805339+0000","last_peered":"2024-04-22T06:43:17.805339+0000","last_clean":"2024-04-22T06:43:17.805339+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:43:17.805339+0000","last_undegraded":"2024-04-22T06:43:17.805339+0000","last_fullsized":"2024-04-22T06:43:17.805339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896491+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:43:17.896491+0000","last_peered":"2024-04-22T06:43:17.896491+0000","last_clean":"2024-04-22T06:43:17.896491+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:43:17.896491+0000","last_undegraded":"2024-04-22T06:43:17.896491+0000","last_fullsized":"2024-04-22T06:43:17.896491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.805672+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:43:17.805672+0000","last_peered":"2024-04-22T06:43:17.805672+0000","last_clean":"2024-04-22T06:43:17.805672+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:43:17.805672+0000","last_undegraded":"2024-04-22T06:43:17.805672+0000","last_fullsized":"2024-04-22T06:43:17.805672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.814239+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:43:17.814239+0000","last_peered":"2024-04-22T06:43:17.814239+0000","last_clean":"2024-04-22T06:43:17.814239+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:43:17.814239+0000","last_undegraded":"2024-04-22T06:43:17.814239+0000","last_fullsized":"2024-04-22T06:43:17.814239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.814744+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:43:17.814744+0000","last_peered":"2024-04-22T06:43:17.814744+0000","last_clean":"2024-04-22T06:43:17.814744+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:43:17.814744+0000","last_undegraded":"2024-04-22T06:43:17.814744+0000","last_fullsized":"2024-04-22T06:43:17.814744+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812343+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:43:17.812343+0000","last_peered":"2024-04-22T06:43:17.812343+0000","last_clean":"2024-04-22T06:43:17.812343+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:43:17.812343+0000","last_undegraded":"2024-04-22T06:43:17.812343+0000","last_fullsized":"2024-04-22T06:43:17.812343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.804143+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:43:17.804143+0000","last_peered":"2024-04-22T06:43:17.804143+0000","last_clean":"2024-04-22T06:43:17.804143+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:43:17.804143+0000","last_undegraded":"2024-04-22T06:43:17.804143+0000","last_fullsized":"2024-04-22T06:43:17.804143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896436+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:43:17.896436+0000","last_peered":"2024-04-22T06:43:17.896436+0000","last_clean":"2024-04-22T06:43:17.896436+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:43:17.896436+0000","last_undegraded":"2024-04-22T06:43:17.896436+0000","last_fullsized":"2024-04-22T06:43:17.896436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809303+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:43:17.809303+0000","last_peered":"2024-04-22T06:43:17.809303+0000","last_clean":"2024-04-22T06:43:17.809303+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:43:17.809303+0000","last_undegraded":"2024-04-22T06:43:17.809303+0000","last_fullsized":"2024-04-22T06:43:17.809303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812864+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:43:17.812864+0000","last_peered":"2024-04-22T06:43:17.812864+0000","last_clean":"2024-04-22T06:43:17.812864+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:43:17.812864+0000","last_undegraded":"2024-04-22T06:43:17.812864+0000","last_fullsized":"2024-04-22T06:43:17.812864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811699+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:43:17.811699+0000","last_peered":"2024-04-22T06:43:17.811699+0000","last_clean":"2024-04-22T06:43:17.811699+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:43:17.811699+0000","last_undegraded":"2024-04-22T06:43:17.811699+0000","last_fullsized":"2024-04-22T06:43:17.811699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.812891+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:43:17.812891+0000","last_peered":"2024-04-22T06:43:17.812891+0000","last_clean":"2024-04-22T06:43:17.812891+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:43:17.812891+0000","last_undegraded":"2024-04-22T06:43:17.812891+0000","last_fullsized":"2024-04-22T06:43:17.812891+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.811369+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:43:17.811369+0000","last_peered":"2024-04-22T06:43:17.811369+0000","last_clean":"2024-04-22T06:43:17.811369+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:43:17.811369+0000","last_undegraded":"2024-04-22T06:43:17.811369+0000","last_fullsized":"2024-04-22T06:43:17.811369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:18.083107+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:43:18.083107+0000","last_peered":"2024-04-22T06:43:18.083107+0000","last_clean":"2024-04-22T06:43:18.083107+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:43:18.083107+0000","last_undegraded":"2024-04-22T06:43:18.083107+0000","last_fullsized":"2024-04-22T06:43:18.083107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.896446+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:43:17.896446+0000","last_peered":"2024-04-22T06:43:17.896446+0000","last_clean":"2024-04-22T06:43:17.896446+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:43:17.896446+0000","last_undegraded":"2024-04-22T06:43:17.896446+0000","last_fullsized":"2024-04-22T06:43:17.896446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:17.809029+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:43:17.809029+0000","last_peered":"2024-04-22T06:43:17.809029+0000","last_clean":"2024-04-22T06:43:17.809029+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:43:17.809029+0000","last_undegraded":"2024-04-22T06:43:17.809029+0000","last_fullsized":"2024-04-22T06:43:17.809029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63233,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":458,"num_read_kb":295,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":189699,"data_compressed":5622,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":456197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":1106910,"data_compressed":37581,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":456353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":1048674,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":946176,"data_stored":455822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1003520,"data_stored":514456,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":516197,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":456133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":456650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":1047967,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34066,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":58841,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":60582,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":60810,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:43:44.610 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T06:43:44.610 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T06:43:44.610 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T06:43:44.610 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T06:43:44.853 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:45.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:44 smithi193 ceph-mon[31815]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:45.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:44 smithi190 ceph-mon[31421]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:45.208 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:44 smithi079 ceph-mon[27946]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:43:46.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3462260315 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 msgr2=0x7fdec010f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3462260315 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec010f8c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800b0a0 tx=0x7fdeb800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3462260315 shutdown_connections 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3462260315 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec010f8c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3462260315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdec0107930 0x7fdec0107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3462260315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdec0102a90 0x7fdec0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3462260315 >> 172.21.15.79:0/3462260315 conn(0x7fdec0079bf0 msgr2=0x7fdec007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3462260315 shutdown_connections 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.542+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3462260315 wait complete. 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 Processor -- start 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 -- start start 2024-04-22T06:43:46.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdec0102a90 0x7fdec0199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec0199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdec0107930 0x7fdec019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0112cc0 con 0x7fdec0103430 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0112b40 con 0x7fdec0102a90 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec6b2e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0112e40 con 0x7fdec0107930 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdebffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec0199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdebffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec0199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58080/0 (socket says 172.21.15.79:58080) 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdebffff700 1 -- 172.21.15.79:0/3375492683 learned_addr learned my addr 172.21.15.79:0/3375492683 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.543+0000 7fdec50cb700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdec0107930 0x7fdec019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdec48ca700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdec0102a90 0x7fdec0199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebffff700 1 -- 172.21.15.79:0/3375492683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdec0107930 msgr2=0x7fdec019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebffff700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdec0107930 0x7fdec019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebffff700 1 -- 172.21.15.79:0/3375492683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdec0102a90 msgr2=0x7fdec0199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebffff700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdec0102a90 0x7fdec0199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebffff700 1 -- 172.21.15.79:0/3375492683 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec019e260 con 0x7fdec0103430 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebffff700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec0199790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00b530 tx=0x7fdeac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac015020 con 0x7fdec0103430 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdeac011ce0 con 0x7fdec0103430 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec019e550 con 0x7fdec0103430 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac010e40 con 0x7fdec0103430 2024-04-22T06:43:46.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.544+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec01a5dd0 con 0x7fdec0103430 2024-04-22T06:43:46.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.545+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec0069730 con 0x7fdec0103430 2024-04-22T06:43:46.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.546+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdeac018020 con 0x7fdec0103430 2024-04-22T06:43:46.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.547+0000 7fdebdffb700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdeb00a3d50 0x7fdeb00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.547+0000 7fdec48ca700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdeb00a3d50 0x7fdeb00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.547+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fdeac0cfc80 con 0x7fdec0103430 2024-04-22T06:43:46.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.547+0000 7fdec48ca700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdeb00a3d50 0x7fdeb00a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4006fd0 tx=0x7fdeb4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:46.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.552+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdeac01f020 con 0x7fdec0103430 2024-04-22T06:43:46.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.834+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdec002df80 con 0x7fdec0103430 2024-04-22T06:43:46.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.834+0000 7fdebdffb700 1 -- 172.21.15.79:0/3375492683 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fdeac09c620 con 0x7fdec0103430 2024-04-22T06:43:46.833 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:46.833 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdeb00a3d50 msgr2=0x7fdeb00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdeb00a3d50 0x7fdeb00a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4006fd0 tx=0x7fdeb4008040 comp rx=0 tx=0).stop 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 msgr2=0x7fdec0199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec0199790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00b530 tx=0x7fdeac00b840 comp rx=0 tx=0).stop 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 shutdown_connections 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdec0107930 0x7fdec019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdec0102a90 0x7fdec0199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdeb00a3d50 0x7fdeb00a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 --2- 172.21.15.79:0/3375492683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdec0103430 0x7fdec0199790 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.836+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 >> 172.21.15.79:0/3375492683 conn(0x7fdec0079bf0 msgr2=0x7fdec0100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:46.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.837+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 shutdown_connections 2024-04-22T06:43:46.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:46.837+0000 7fdec6b2e700 1 -- 172.21.15.79:0/3375492683 wait complete. 2024-04-22T06:43:47.510 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T06:43:47.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 -- 172.21.15.79:0/990726969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 msgr2=0x7f349010a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 --2- 172.21.15.79:0/990726969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349010a3e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3488009ca0 tx=0x7f348802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 -- 172.21.15.79:0/990726969 shutdown_connections 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 --2- 172.21.15.79:0/990726969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349010a3e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 --2- 172.21.15.79:0/990726969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 0x7f3490103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 --2- 172.21.15.79:0/990726969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3490101fa0 0x7f34901023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.616+0000 7f3497c6d640 1 -- 172.21.15.79:0/990726969 >> 172.21.15.79:0/990726969 conn(0x7f34900fd530 msgr2=0x7f34900ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 -- 172.21.15.79:0/990726969 shutdown_connections 2024-04-22T06:43:47.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 -- 172.21.15.79:0/990726969 wait complete. 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 Processor -- start 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 -- start start 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3490101fa0 0x7f34901981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 0x7f3490198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f349010db60 con 0x7f3490103b50 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f349010d9e0 con 0x7f3490101fa0 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3497c6d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f349010dce0 con 0x7f3490103190 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f349620a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f349620a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58096/0 (socket says 172.21.15.79:58096) 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f349620a640 1 -- 172.21.15.79:0/3904352476 learned_addr learned my addr 172.21.15.79:0/3904352476 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.617+0000 7f3495208640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 0x7f3490198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f349620a640 1 -- 172.21.15.79:0/3904352476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 msgr2=0x7f3490198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f349620a640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 0x7f3490198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f349620a640 1 -- 172.21.15.79:0/3904352476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3490101fa0 msgr2=0x7f34901981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f349620a640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3490101fa0 0x7f34901981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f349620a640 1 -- 172.21.15.79:0/3904352476 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349019fe00 con 0x7f3490103b50 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f3495208640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 0x7f3490198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f349620a640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349019f790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f348802c700 tx=0x7f3488034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3488039040 con 0x7f3490103b50 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3488044070 con 0x7f3490103b50 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34880042a0 con 0x7f3490103b50 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34901a0090 con 0x7f3490103b50 2024-04-22T06:43:47.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.618+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34901a05d0 con 0x7f3490103b50 2024-04-22T06:43:47.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.619+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3490109340 con 0x7f3490103b50 2024-04-22T06:43:47.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.620+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3488003970 con 0x7f3490103b50 2024-04-22T06:43:47.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.621+0000 7f3486ffd640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f347c0a3cc0 0x7f347c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.621+0000 7f3495a09640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f347c0a3cc0 0x7f347c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.621+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f34880f3380 con 0x7f3490103b50 2024-04-22T06:43:47.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.621+0000 7f3495a09640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f347c0a3cc0 0x7f347c0a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3490102ff0 tx=0x7f348c006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:47.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.625+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34880bfd20 con 0x7f3490103b50 2024-04-22T06:43:47.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.731+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f34880bbc90 con 0x7f3490103b50 2024-04-22T06:43:47.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:47 smithi079 ceph-mon[27946]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:47 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3375492683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:43:47.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.810+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3490062b50 con 0x7f3490103b50 2024-04-22T06:43:47.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.812+0000 7f3486ffd640 1 -- 172.21.15.79:0/3904352476 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22650 (secure 0 0 0) 0x7f3488077180 con 0x7f3490103b50 2024-04-22T06:43:47.810 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:47.810 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":87,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:43:17.795510+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:43:47.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f347c0a3cc0 msgr2=0x7f347c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f347c0a3cc0 0x7f347c0a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3490102ff0 tx=0x7f348c006c90 comp rx=0 tx=0).stop 2024-04-22T06:43:47.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 msgr2=0x7f349019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349019f790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f348802c700 tx=0x7f3488034760 comp rx=0 tx=0).stop 2024-04-22T06:43:47.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 shutdown_connections 2024-04-22T06:43:47.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f347c0a3cc0 0x7f347c0a6180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3490103b50 0x7f349019f790 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3490103190 0x7f3490198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 --2- 172.21.15.79:0/3904352476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3490101fa0 0x7f34901981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 >> 172.21.15.79:0/3904352476 conn(0x7f34900fd530 msgr2=0x7f3490106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:47.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 shutdown_connections 2024-04-22T06:43:47.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.814+0000 7f3497c6d640 1 -- 172.21.15.79:0/3904352476 wait complete. 2024-04-22T06:43:47.824 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:43:47.824 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 -- 172.21.15.79:0/1553639895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 msgr2=0x7f693810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 --2- 172.21.15.79:0/1553639895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693810a3e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6930009ca0 tx=0x7f693002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 -- 172.21.15.79:0/1553639895 shutdown_connections 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 --2- 172.21.15.79:0/1553639895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693810a3e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 --2- 172.21.15.79:0/1553639895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 0x7f6938103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 --2- 172.21.15.79:0/1553639895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6938101fa0 0x7f69381023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 -- 172.21.15.79:0/1553639895 >> 172.21.15.79:0/1553639895 conn(0x7f69380fd530 msgr2=0x7f69380ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:47.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 -- 172.21.15.79:0/1553639895 shutdown_connections 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.927+0000 7f693f081640 1 -- 172.21.15.79:0/1553639895 wait complete. 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 Processor -- start 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 -- start start 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6938101fa0 0x7f6938198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 0x7f69381986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f693810daa0 con 0x7f6938103b50 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f693810d920 con 0x7f6938101fa0 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693f081640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f693810dc20 con 0x7f6938103190 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58118/0 (socket says 172.21.15.79:58118) 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 -- 172.21.15.79:0/3990149354 learned_addr learned my addr 172.21.15.79:0/3990149354 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:47.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f6937fff640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 0x7f69381986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693ce1d640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6938101fa0 0x7f6938198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 -- 172.21.15.79:0/3990149354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 msgr2=0x7f69381986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 0x7f69381986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 -- 172.21.15.79:0/3990149354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6938101fa0 msgr2=0x7f6938198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6938101fa0 0x7f6938198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f693d61e640 1 -- 172.21.15.79:0/3990149354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693819fd90 con 0x7f6938103b50 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.928+0000 7f6937fff640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 0x7f69381986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.929+0000 7f693d61e640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693819f720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6930032520 tx=0x7f69300347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.929+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930039040 con 0x7f6938103b50 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.929+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69381a0020 con 0x7f6938103b50 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.929+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6930044070 con 0x7f6938103b50 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.929+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69300042a0 con 0x7f6938103b50 2024-04-22T06:43:47.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.929+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69381a0560 con 0x7f6938103b50 2024-04-22T06:43:47.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.930+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6938109340 con 0x7f6938103b50 2024-04-22T06:43:47.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.931+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6930003970 con 0x7f6938103b50 2024-04-22T06:43:47.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.931+0000 7f6935ffb640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69280a3bb0 0x7f69280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:47.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.932+0000 7f693ce1d640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69280a3bb0 0x7f69280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:47.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.932+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f69300f27e0 con 0x7f6938103b50 2024-04-22T06:43:47.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.932+0000 7f693ce1d640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69280a3bb0 0x7f69280a6070 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f692c005fd0 tx=0x7f692c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:47.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:47.936+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69300333a0 con 0x7f6938103b50 2024-04-22T06:43:48.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.116+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6938062b50 con 0x7f6938103b50 2024-04-22T06:43:48.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.117+0000 7f6935ffb640 1 -- 172.21.15.79:0/3990149354 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f69300bf1d0 con 0x7f6938103b50 2024-04-22T06:43:48.115 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:43:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69280a3bb0 msgr2=0x7f69280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69280a3bb0 0x7f69280a6070 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f692c005fd0 tx=0x7f692c005850 comp rx=0 tx=0).stop 2024-04-22T06:43:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 msgr2=0x7f693819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693819f720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6930032520 tx=0x7f69300347b0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 shutdown_connections 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69280a3bb0 0x7f69280a6070 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6938103b50 0x7f693819f720 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6938103190 0x7f69381986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 --2- 172.21.15.79:0/3990149354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6938101fa0 0x7f6938198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 >> 172.21.15.79:0/3990149354 conn(0x7f69380fd530 msgr2=0x7f6938106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 shutdown_connections 2024-04-22T06:43:48.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.119+0000 7f693f081640 1 -- 172.21.15.79:0/3990149354 wait complete. 2024-04-22T06:43:48.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:43:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:47 smithi193 ceph-mon[31815]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:47 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3375492683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:43:48.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:47 smithi190 ceph-mon[31421]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:43:48.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:47 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3375492683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:43:48.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 -- 172.21.15.79:0/2609903942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194101fa0 msgr2=0x7f01941023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 --2- 172.21.15.79:0/2609903942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194101fa0 0x7f01941023c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0188009900 tx=0x7f018802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 -- 172.21.15.79:0/2609903942 shutdown_connections 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 --2- 172.21.15.79:0/2609903942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0194103190 0x7f0194103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 --2- 172.21.15.79:0/2609903942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0194103b50 0x7f019410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 --2- 172.21.15.79:0/2609903942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194101fa0 0x7f01941023c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 -- 172.21.15.79:0/2609903942 >> 172.21.15.79:0/2609903942 conn(0x7f01940fd530 msgr2=0x7f01940ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 -- 172.21.15.79:0/2609903942 shutdown_connections 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 -- 172.21.15.79:0/2609903942 wait complete. 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.230+0000 7f019acba640 1 Processor -- start 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 -- start start 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0194101fa0 0x7f01941981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 0x7f01941986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0194103b50 0x7f019419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f019410dae0 con 0x7f0194103190 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f019410d960 con 0x7f0194103b50 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f019410dc60 con 0x7f0194101fa0 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 0x7f01941986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 0x7f01941986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58142/0 (socket says 172.21.15.79:58142) 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 -- 172.21.15.79:0/1511313217 learned_addr learned my addr 172.21.15.79:0/1511313217 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0198a56640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0194101fa0 0x7f01941981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 -- 172.21.15.79:0/1511313217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0194101fa0 msgr2=0x7f01941981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0199257640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0194103b50 0x7f019419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0194101fa0 0x7f01941981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 -- 172.21.15.79:0/1511313217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0194103b50 msgr2=0x7f019419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0194103b50 0x7f019419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 -- 172.21.15.79:0/1511313217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019419fdc0 con 0x7f0194103190 2024-04-22T06:43:48.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0193fff640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 0x7f01941986e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f018000c920 tx=0x7f018000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:48.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180007c70 con 0x7f0194103190 2024-04-22T06:43:48.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01941a00b0 con 0x7f0194103190 2024-04-22T06:43:48.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0180007e10 con 0x7f0194103190 2024-04-22T06:43:48.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.231+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180018820 con 0x7f0194103190 2024-04-22T06:43:48.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.232+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01941a0650 con 0x7f0194103190 2024-04-22T06:43:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.233+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0194109390 con 0x7f0194103190 2024-04-22T06:43:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.234+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f018000f450 con 0x7f0194103190 2024-04-22T06:43:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.234+0000 7f0191ffb640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f01840a3bb0 0x7f01840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.234+0000 7f0198a56640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f01840a3bb0 0x7f01840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.234+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f01800d0500 con 0x7f0194103190 2024-04-22T06:43:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.235+0000 7f0198a56640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f01840a3bb0 0x7f01840a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0188009230 tx=0x7f0188005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:48.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.239+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0180014030 con 0x7f0194103190 2024-04-22T06:43:48.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.415+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0194062b50 con 0x7f0194103190 2024-04-22T06:43:48.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.415+0000 7f0191ffb640 1 -- 172.21.15.79:0/1511313217 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f018009cef0 con 0x7f0194103190 2024-04-22T06:43:48.511 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f01840a3bb0 msgr2=0x7f01840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f01840a3bb0 0x7f01840a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0188009230 tx=0x7f0188005fd0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 msgr2=0x7f01941986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 0x7f01941986e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f018000c920 tx=0x7f018000cde0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 shutdown_connections 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f01840a3bb0 0x7f01840a6070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0194103b50 0x7f019419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0194103190 0x7f01941986e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 --2- 172.21.15.79:0/1511313217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0194101fa0 0x7f01941981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 >> 172.21.15.79:0/1511313217 conn(0x7f01940fd530 msgr2=0x7f0194106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:48.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 shutdown_connections 2024-04-22T06:43:48.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.417+0000 7f019acba640 1 -- 172.21.15.79:0/1511313217 wait complete. 2024-04-22T06:43:48.514 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:43:48.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 -- 172.21.15.79:0/1422670079 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 msgr2=0x7fb214102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/1422670079 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb214102850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb210009900 tx=0x7fb21002bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 -- 172.21.15.79:0/1422670079 shutdown_connections 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/1422670079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb214101b20 0x7fb2141093c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/1422670079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101160 0x7fb2141015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/1422670079 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb214102850 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 -- 172.21.15.79:0/1422670079 >> 172.21.15.79:0/1422670079 conn(0x7fb2140fbb90 msgr2=0x7fb2140fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 -- 172.21.15.79:0/1422670079 shutdown_connections 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.617+0000 7fb21c6f2640 1 -- 172.21.15.79:0/1422670079 wait complete. 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 Processor -- start 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 -- start start 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb214101160 0x7fb214198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101b20 0x7fb214198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb21419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb21410c9b0 con 0x7fb214102430 2024-04-22T06:43:48.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb21410c830 con 0x7fb214101160 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb21410cb30 con 0x7fb214101b20 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb21419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb21419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58150/0 (socket says 172.21.15.79:58150) 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 -- 172.21.15.79:0/3954706815 learned_addr learned my addr 172.21.15.79:0/3954706815 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb219c8d640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101b20 0x7fb214198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21a48e640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb214101160 0x7fb214198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 -- 172.21.15.79:0/3954706815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101b20 msgr2=0x7fb214198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101b20 0x7fb214198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 -- 172.21.15.79:0/3954706815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb214101160 msgr2=0x7fb214198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb214101160 0x7fb214198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 -- 172.21.15.79:0/3954706815 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21419fe20 con 0x7fb214102430 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb219c8d640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101b20 0x7fb214198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21ac8f640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb21419f7b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00cce0 tx=0x7fb20c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:48.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00f450 con 0x7fb214102430 2024-04-22T06:43:48.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2141a0110 con 0x7fb214102430 2024-04-22T06:43:48.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb20c00faa0 con 0x7fb214102430 2024-04-22T06:43:48.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c018b70 con 0x7fb214102430 2024-04-22T06:43:48.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.618+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2141a0660 con 0x7fb214102430 2024-04-22T06:43:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.619+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2141a0920 con 0x7fb214102430 2024-04-22T06:43:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.620+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb20c018d10 con 0x7fb214102430 2024-04-22T06:43:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.621+0000 7fb20b7fe640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb2000a3bb0 0x7fb2000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.621+0000 7fb21a48e640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb2000a3bb0 0x7fb2000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.621+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb20c0d0920 con 0x7fb214102430 2024-04-22T06:43:48.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.621+0000 7fb21a48e640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb2000a3bb0 0x7fb2000a6070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb210006010 tx=0x7fb210005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:48.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.626+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb20c014030 con 0x7fb214102430 2024-04-22T06:43:48.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.806+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2141a0af0 con 0x7fb214102430 2024-04-22T06:43:48.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.806+0000 7fb20b7fe640 1 -- 172.21.15.79:0/3954706815 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fb20c09d2f0 con 0x7fb214102430 2024-04-22T06:43:48.805 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb2000a3bb0 msgr2=0x7fb2000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb2000a3bb0 0x7fb2000a6070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb210006010 tx=0x7fb210005be0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 msgr2=0x7fb21419f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb21419f7b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00cce0 tx=0x7fb20c00a620 comp rx=0 tx=0).stop 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 shutdown_connections 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb2000a3bb0 0x7fb2000a6070 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb214102430 0x7fb21419f7b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb214101b20 0x7fb214198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 --2- 172.21.15.79:0/3954706815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb214101160 0x7fb214198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 >> 172.21.15.79:0/3954706815 conn(0x7fb2140fbb90 msgr2=0x7fb2140fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:48.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 shutdown_connections 2024-04-22T06:43:48.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.808+0000 7fb21c6f2640 1 -- 172.21.15.79:0/3954706815 wait complete. 2024-04-22T06:43:48.818 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:43:48.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 -- 172.21.15.79:0/542159412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248102a00 msgr2=0x7fd248104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 --2- 172.21.15.79:0/542159412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248102a00 0x7fd248104df0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd244009900 tx=0x7fd24402bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 -- 172.21.15.79:0/542159412 shutdown_connections 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 --2- 172.21.15.79:0/542159412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd248107c60 0x7fd24810a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 --2- 172.21.15.79:0/542159412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248105330 0x7fd248107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 --2- 172.21.15.79:0/542159412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248102a00 0x7fd248104df0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 -- 172.21.15.79:0/542159412 >> 172.21.15.79:0/542159412 conn(0x7fd2480fc410 msgr2=0x7fd2480fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 -- 172.21.15.79:0/542159412 shutdown_connections 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 -- 172.21.15.79:0/542159412 wait complete. 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.917+0000 7fd24fd52640 1 Processor -- start 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 -- start start 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248102a00 0x7fd248198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd248105330 0x7fd248198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 0x7fd24819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd24810dab0 con 0x7fd248107c60 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd24810d930 con 0x7fd248105330 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd24810dc30 con 0x7fd248102a00 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 0x7fd24819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 0x7fd24819f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58156/0 (socket says 172.21.15.79:58156) 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 -- 172.21.15.79:0/3546994084 learned_addr learned my addr 172.21.15.79:0/3546994084 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24daee640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248102a00 0x7fd248198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24d2ed640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd248105330 0x7fd248198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 -- 172.21.15.79:0/3546994084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248102a00 msgr2=0x7fd248198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248102a00 0x7fd248198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 -- 172.21.15.79:0/3546994084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd248105330 msgr2=0x7fd248198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd248105330 0x7fd248198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 -- 172.21.15.79:0/3546994084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24819fd70 con 0x7fd248107c60 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24e2ef640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 0x7fd24819f700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd24000cce0 tx=0x7fd24000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:48.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24000f450 con 0x7fd248107c60 2024-04-22T06:43:48.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd24000faa0 con 0x7fd248107c60 2024-04-22T06:43:48.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2481a0060 con 0x7fd248107c60 2024-04-22T06:43:48.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240018b70 con 0x7fd248107c60 2024-04-22T06:43:48.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.918+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2481a0560 con 0x7fd248107c60 2024-04-22T06:43:48.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.919+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2480fdf00 con 0x7fd248107c60 2024-04-22T06:43:48.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.920+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd240018d10 con 0x7fd248107c60 2024-04-22T06:43:48.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.921+0000 7fd23effd640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd2340a3bb0 0x7fd2340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:48.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.921+0000 7fd24daee640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd2340a3bb0 0x7fd2340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:48.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.921+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd2400d07c0 con 0x7fd248107c60 2024-04-22T06:43:48.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.921+0000 7fd24daee640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd2340a3bb0 0x7fd2340a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd244006010 tx=0x7fd2440095a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:48.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:48.925+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd240014030 con 0x7fd248107c60 2024-04-22T06:43:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:48 smithi079 ceph-mon[27946]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:43:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:48 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3904352476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:48 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3990149354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:48 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1511313217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:49.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.141+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd248062b50 con 0x7fd248107c60 2024-04-22T06:43:49.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.142+0000 7fd23effd640 1 -- 172.21.15.79:0/3546994084 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fd24009d1b0 con 0x7fd248107c60 2024-04-22T06:43:49.140 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:49.141 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:42:28.714055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24436,24440,24467]},"id":1}]} 2024-04-22T06:43:49.141 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:43:49.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd2340a3bb0 msgr2=0x7fd2340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd2340a3bb0 0x7fd2340a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd244006010 tx=0x7fd2440095a0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 msgr2=0x7fd24819f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 0x7fd24819f700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd24000cce0 tx=0x7fd24000a620 comp rx=0 tx=0).stop 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 shutdown_connections 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd2340a3bb0 0x7fd2340a6070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd248105330 0x7fd248198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd248107c60 0x7fd24819f700 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 --2- 172.21.15.79:0/3546994084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248102a00 0x7fd248198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 >> 172.21.15.79:0/3546994084 conn(0x7fd2480fc410 msgr2=0x7fd248103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 shutdown_connections 2024-04-22T06:43:49.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.144+0000 7fd24fd52640 1 -- 172.21.15.79:0/3546994084 wait complete. 2024-04-22T06:43:49.155 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:43:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:48 smithi193 ceph-mon[31815]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:43:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:48 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3904352476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:48 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3990149354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:48 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1511313217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:49.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:48 smithi190 ceph-mon[31421]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:43:49.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:48 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3904352476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:49.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:48 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3990149354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:49.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:48 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1511313217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:49.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- 172.21.15.79:0/2832149180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 msgr2=0x7f21e81012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- 172.21.15.79:0/2832149180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81012a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009900 tx=0x7f21d802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- 172.21.15.79:0/2832149180 shutdown_connections 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- 172.21.15.79:0/2832149180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e8101fd0 0x7f21e8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- 172.21.15.79:0/2832149180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8102990 0x7f21e81092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- 172.21.15.79:0/2832149180 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81012a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- 172.21.15.79:0/2832149180 >> 172.21.15.79:0/2832149180 conn(0x7f21e80fc410 msgr2=0x7f21e80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- 172.21.15.79:0/2832149180 shutdown_connections 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- 172.21.15.79:0/2832149180 wait complete. 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 Processor -- start 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- start start 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e8101fd0 0x7f21e8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8102990 0x7f21e819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f21e810db20 con 0x7f21e8100e80 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21e810d9a0 con 0x7f21e8101fd0 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.255+0000 7f21f048c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f21e810dca0 con 0x7f21e8102990 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58180/0 (socket says 172.21.15.79:58180) 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 -- 172.21.15.79:0/1365537320 learned_addr learned my addr 172.21.15.79:0/1365537320 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21eda27640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e8101fd0 0x7f21e8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 -- 172.21.15.79:0/1365537320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8102990 msgr2=0x7f21e819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8102990 0x7f21e819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 -- 172.21.15.79:0/1365537320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e8101fd0 msgr2=0x7f21e8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e8101fd0 0x7f21e8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 -- 172.21.15.79:0/1365537320 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e819fde0 con 0x7f21e8100e80 2024-04-22T06:43:49.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21ee228640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81981c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f21d8006010 tx=0x7f21d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d8043020 con 0x7f21e8100e80 2024-04-22T06:43:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d8004aa0 con 0x7f21e8100e80 2024-04-22T06:43:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e81a0070 con 0x7f21e8100e80 2024-04-22T06:43:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d8003c70 con 0x7f21e8100e80 2024-04-22T06:43:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.256+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e81a05c0 con 0x7f21e8100e80 2024-04-22T06:43:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.257+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e81081b0 con 0x7f21e8100e80 2024-04-22T06:43:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.258+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f21d8046020 con 0x7f21e8100e80 2024-04-22T06:43:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.259+0000 7f21df7fe640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21d40a3bb0 0x7f21d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.259+0000 7f21eda27640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21d40a3bb0 0x7f21d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.259+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f21d80f2870 con 0x7f21e8100e80 2024-04-22T06:43:49.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.259+0000 7f21eda27640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21d40a3bb0 0x7f21d40a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005fd0 tx=0x7f21e4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.263+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21d804d020 con 0x7f21e8100e80 2024-04-22T06:43:49.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.441+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21e8062bf0 con 0x7f21e8100e80 2024-04-22T06:43:49.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.442+0000 7f21df7fe640 1 -- 172.21.15.79:0/1365537320 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22650 (secure 0 0 0) 0x7f21d80bf260 con 0x7f21e8100e80 2024-04-22T06:43:49.441 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":87,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:43:17.795510+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:43:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21d40a3bb0 msgr2=0x7f21d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21d40a3bb0 0x7f21d40a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005fd0 tx=0x7f21e4005850 comp rx=0 tx=0).stop 2024-04-22T06:43:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 msgr2=0x7f21e81981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81981c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f21d8006010 tx=0x7f21d80042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 shutdown_connections 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f21d40a3bb0 0x7f21d40a6070 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e8101fd0 0x7f21e8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21e8102990 0x7f21e819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 --2- 172.21.15.79:0/1365537320 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21e8100e80 0x7f21e81981c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 >> 172.21.15.79:0/1365537320 conn(0x7f21e80fc410 msgr2=0x7f21e8105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 shutdown_connections 2024-04-22T06:43:49.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.445+0000 7f21f048c640 1 -- 172.21.15.79:0/1365537320 wait complete. 2024-04-22T06:43:49.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:43:49.455 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- 172.21.15.79:0/965726498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4100e60 msgr2=0x7fdcd4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- 172.21.15.79:0/965726498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4100e60 0x7fdcd4101280 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009900 tx=0x7fdcc402bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- 172.21.15.79:0/965726498 shutdown_connections 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- 172.21.15.79:0/965726498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd4102a00 0x7fdcd4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- 172.21.15.79:0/965726498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd4102040 0x7fdcd41024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- 172.21.15.79:0/965726498 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4100e60 0x7fdcd4101280 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- 172.21.15.79:0/965726498 >> 172.21.15.79:0/965726498 conn(0x7fdcd40fc420 msgr2=0x7fdcd40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- 172.21.15.79:0/965726498 shutdown_connections 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- 172.21.15.79:0/965726498 wait complete. 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 Processor -- start 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- start start 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd4100e60 0x7fdcd410b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd4102040 0x7fdcd4109ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 0x7fdcd410a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd410db30 con 0x7fdcd4102a00 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd410d9b0 con 0x7fdcd4102040 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcdc231640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd410dcb0 con 0x7fdcd4100e60 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcda7ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 0x7fdcd410a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcda7ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 0x7fdcd410a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58208/0 (socket says 172.21.15.79:58208) 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.559+0000 7fdcda7ce640 1 -- 172.21.15.79:0/512358116 learned_addr learned my addr 172.21.15.79:0/512358116 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcd9fcd640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd4100e60 0x7fdcd410b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcda7ce640 1 -- 172.21.15.79:0/512358116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd4100e60 msgr2=0x7fdcd410b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcda7ce640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd4100e60 0x7fdcd410b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcda7ce640 1 -- 172.21.15.79:0/512358116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd4102040 msgr2=0x7fdcd4109ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcda7ce640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd4102040 0x7fdcd4109ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcda7ce640 1 -- 172.21.15.79:0/512358116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd410ac70 con 0x7fdcd4102a00 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcda7ce640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 0x7fdcd410a3e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00eaa0 tx=0x7fdccc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00cdb0 con 0x7fdcd4102a00 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdccc005020 con 0x7fdcd4102a00 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd41a2360 con 0x7fdcd4102a00 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0116d0 con 0x7fdcd4102a00 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.560+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41a28b0 con 0x7fdcd4102a00 2024-04-22T06:43:49.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.561+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd41082f0 con 0x7fdcd4102a00 2024-04-22T06:43:49.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.562+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdccc0051d0 con 0x7fdcd4102a00 2024-04-22T06:43:49.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.562+0000 7fdccaffd640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdcc00a3c30 0x7fdcc00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.563+0000 7fdcd9fcd640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdcc00a3c30 0x7fdcc00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.563+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fdccc0d06a0 con 0x7fdcd4102a00 2024-04-22T06:43:49.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.563+0000 7fdcd9fcd640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdcc00a3c30 0x7fdcc00a60f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4006010 tx=0x7fdcc4005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.567+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdccc016030 con 0x7fdcd4102a00 2024-04-22T06:43:49.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.746+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdcd4062b50 con 0x7fdcd4102a00 2024-04-22T06:43:49.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.746+0000 7fdccaffd640 1 -- 172.21.15.79:0/512358116 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fdccc09d090 con 0x7fdcd4102a00 2024-04-22T06:43:49.745 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:43:49.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.748+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdcc00a3c30 msgr2=0x7fdcc00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.748+0000 7fdcdc231640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdcc00a3c30 0x7fdcc00a60f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4006010 tx=0x7fdcc4005be0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.748+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 msgr2=0x7fdcd410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.748+0000 7fdcdc231640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 0x7fdcd410a3e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00eaa0 tx=0x7fdccc00ef60 comp rx=0 tx=0).stop 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 shutdown_connections 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdcc00a3c30 0x7fdcc00a60f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcd4102040 0x7fdcd4109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcd4102a00 0x7fdcd410a3e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 --2- 172.21.15.79:0/512358116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdcd4100e60 0x7fdcd410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 >> 172.21.15.79:0/512358116 conn(0x7fdcd40fc420 msgr2=0x7fdcd41054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 shutdown_connections 2024-04-22T06:43:49.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.749+0000 7fdcdc231640 1 -- 172.21.15.79:0/512358116 wait complete. 2024-04-22T06:43:49.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 -- 172.21.15.79:0/3591047821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514101fe0 msgr2=0x7f6514102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 --2- 172.21.15.79:0/3591047821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514101fe0 0x7f6514102460 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6508009900 tx=0x7f650802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 -- 172.21.15.79:0/3591047821 shutdown_connections 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 --2- 172.21.15.79:0/3591047821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65141029a0 0x7f65141092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 --2- 172.21.15.79:0/3591047821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514101fe0 0x7f6514102460 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 --2- 172.21.15.79:0/3591047821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6514100e90 0x7f65141012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 -- 172.21.15.79:0/3591047821 >> 172.21.15.79:0/3591047821 conn(0x7f65140fc400 msgr2=0x7f65140fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 -- 172.21.15.79:0/3591047821 shutdown_connections 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 -- 172.21.15.79:0/3591047821 wait complete. 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.862+0000 7f651a06d640 1 Processor -- start 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 -- start start 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 0x7f6514197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6514101fe0 0x7f65141976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65141029a0 0x7f651419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f651410ca40 con 0x7f6514100e90 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f651410c8c0 con 0x7f65141029a0 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f651410cbc0 con 0x7f6514101fe0 2024-04-22T06:43:49.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 0x7f6514197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 0x7f6514197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58218/0 (socket says 172.21.15.79:58218) 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 -- 172.21.15.79:0/319374516 learned_addr learned my addr 172.21.15.79:0/319374516 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6512ffd640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6514101fe0 0x7f65141976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6513fff640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65141029a0 0x7f651419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 -- 172.21.15.79:0/319374516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6514101fe0 msgr2=0x7f65141976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6514101fe0 0x7f65141976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 -- 172.21.15.79:0/319374516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65141029a0 msgr2=0x7f651419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65141029a0 0x7f651419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 -- 172.21.15.79:0/319374516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651419ed90 con 0x7f6514100e90 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6512ffd640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6514101fe0 0x7f65141976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6513fff640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65141029a0 0x7f651419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:49.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f65137fe640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 0x7f6514197170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f650400c890 tx=0x7f650400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504004220 con 0x7f6514100e90 2024-04-22T06:43:49.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651419f080 con 0x7f6514100e90 2024-04-22T06:43:49.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65040043c0 con 0x7f6514100e90 2024-04-22T06:43:49.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504004a00 con 0x7f6514100e90 2024-04-22T06:43:49.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.863+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651419f670 con 0x7f6514100e90 2024-04-22T06:43:49.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.864+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6514108270 con 0x7f6514100e90 2024-04-22T06:43:49.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.865+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f65040079b0 con 0x7f6514100e90 2024-04-22T06:43:49.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.866+0000 7f6510ff9640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f64fc0a3bb0 0x7f64fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.866+0000 7f6512ffd640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f64fc0a3bb0 0x7f64fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.866+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f65040d01e0 con 0x7f6514100e90 2024-04-22T06:43:49.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.866+0000 7f6512ffd640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f64fc0a3bb0 0x7f64fc0a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6508006010 tx=0x7f65080095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:49.870+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6504014030 con 0x7f6514100e90 2024-04-22T06:43:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:49 smithi079 ceph-mon[27946]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:43:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:49 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3954706815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:49 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3546994084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:49.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:49 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1365537320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:50.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.047+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6514062b50 con 0x7f6514100e90 2024-04-22T06:43:50.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.047+0000 7f6510ff9640 1 -- 172.21.15.79:0/319374516 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f650409cbd0 con 0x7f6514100e90 2024-04-22T06:43:50.046 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f64fc0a3bb0 msgr2=0x7f64fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f64fc0a3bb0 0x7f64fc0a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6508006010 tx=0x7f65080095f0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 msgr2=0x7f6514197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 0x7f6514197170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f650400c890 tx=0x7f650400cd50 comp rx=0 tx=0).stop 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 shutdown_connections 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f64fc0a3bb0 0x7f64fc0a6070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65141029a0 0x7f651419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6514101fe0 0x7f65141976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 --2- 172.21.15.79:0/319374516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6514100e90 0x7f6514197170 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 >> 172.21.15.79:0/319374516 conn(0x7f65140fc400 msgr2=0x7f6514105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 shutdown_connections 2024-04-22T06:43:50.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.049+0000 7f651a06d640 1 -- 172.21.15.79:0/319374516 wait complete. 2024-04-22T06:43:50.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 -- 172.21.15.79:0/2926825884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 msgr2=0x7f1314103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 --2- 172.21.15.79:0/2926825884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f1314103620 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1308009900 tx=0x7f130802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 -- 172.21.15.79:0/2926825884 shutdown_connections 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 --2- 172.21.15.79:0/2926825884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314103b60 0x7f131410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 --2- 172.21.15.79:0/2926825884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f1314103620 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 --2- 172.21.15.79:0/2926825884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1314101fb0 0x7f13141023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 -- 172.21.15.79:0/2926825884 >> 172.21.15.79:0/2926825884 conn(0x7f13140fd520 msgr2=0x7f13140ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 -- 172.21.15.79:0/2926825884 shutdown_connections 2024-04-22T06:43:50.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.161+0000 7f1318a5c640 1 -- 172.21.15.79:0/2926825884 wait complete. 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 Processor -- start 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 -- start start 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314101fb0 0x7f13141982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f13141987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1314103b60 0x7f131419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f131410db50 con 0x7f13141031a0 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f131410d9d0 con 0x7f1314101fb0 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1318a5c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f131410dcd0 con 0x7f1314103b60 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f13141987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f13141987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58238/0 (socket says 172.21.15.79:58238) 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 -- 172.21.15.79:0/3291082532 learned_addr learned my addr 172.21.15.79:0/3291082532 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1312d9d640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1314103b60 0x7f131419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f131259c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314101fb0 0x7f13141982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 -- 172.21.15.79:0/3291082532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1314103b60 msgr2=0x7f131419f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1314103b60 0x7f131419f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 -- 172.21.15.79:0/3291082532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314101fb0 msgr2=0x7f13141982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314101fb0 0x7f13141982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 -- 172.21.15.79:0/3291082532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131419fec0 con 0x7f13141031a0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f131259c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314101fb0 0x7f13141982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.162+0000 7f1311d9b640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f13141987e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1308006010 tx=0x7f13080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.163+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308043020 con 0x7f13141031a0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.163+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1308004aa0 con 0x7f13141031a0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.163+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13141a0150 con 0x7f13141031a0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.163+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308003c70 con 0x7f13141031a0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.163+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13141a06a0 con 0x7f13141031a0 2024-04-22T06:43:50.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.164+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1314109370 con 0x7f13141031a0 2024-04-22T06:43:50.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.165+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1308046020 con 0x7f13141031a0 2024-04-22T06:43:50.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.165+0000 7f13037fe640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f12fc0a3bb0 0x7f12fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.165+0000 7f131259c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f12fc0a3bb0 0x7f12fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.165+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f13080f2780 con 0x7f13141031a0 2024-04-22T06:43:50.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.166+0000 7f131259c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f12fc0a3bb0 0x7f12fc0a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1304006fd0 tx=0x7f1304008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:50.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.170+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f130804d020 con 0x7f13141031a0 2024-04-22T06:43:50.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:49 smithi193 ceph-mon[31815]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:43:50.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:49 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3954706815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:50.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:49 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3546994084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:50.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:49 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1365537320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:50.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:49 smithi190 ceph-mon[31421]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:43:50.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:49 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3954706815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:50.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:49 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3546994084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:50.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:49 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1365537320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:50.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.347+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1314062b50 con 0x7f13141031a0 2024-04-22T06:43:50.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.348+0000 7f13037fe640 1 -- 172.21.15.79:0/3291082532 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f13080bf170 con 0x7f13141031a0 2024-04-22T06:43:50.347 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f12fc0a3bb0 msgr2=0x7f12fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f12fc0a3bb0 0x7f12fc0a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1304006fd0 tx=0x7f1304008040 comp rx=0 tx=0).stop 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 msgr2=0x7f13141987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f13141987e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1308006010 tx=0x7f13080042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 shutdown_connections 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f12fc0a3bb0 0x7f12fc0a6070 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1314103b60 0x7f131419f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f13141031a0 0x7f13141987e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 --2- 172.21.15.79:0/3291082532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1314101fb0 0x7f13141982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 >> 172.21.15.79:0/3291082532 conn(0x7f13140fd520 msgr2=0x7f1314106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 shutdown_connections 2024-04-22T06:43:50.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.350+0000 7f1318a5c640 1 -- 172.21.15.79:0/3291082532 wait complete. 2024-04-22T06:43:50.360 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:43:50.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 -- 172.21.15.79:0/1538640639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 msgr2=0x7f9b80103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- 172.21.15.79:0/1538640639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b80103600 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009900 tx=0x7f9b7c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:50.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 -- 172.21.15.79:0/1538640639 shutdown_connections 2024-04-22T06:43:50.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- 172.21.15.79:0/1538640639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 0x7f9b8010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- 172.21.15.79:0/1538640639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b80103600 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- 172.21.15.79:0/1538640639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 0x7f9b801023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 -- 172.21.15.79:0/1538640639 >> 172.21.15.79:0/1538640639 conn(0x7f9b800fd500 msgr2=0x7f9b800ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 -- 172.21.15.79:0/1538640639 shutdown_connections 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 -- 172.21.15.79:0/1538640639 wait complete. 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 Processor -- start 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 -- start start 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 0x7f9b80198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b801986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.459+0000 7f9b88375640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 0x7f9b8019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b88375640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010dad0 con 0x7f9b80103180 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b88375640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d950 con 0x7f9b80101f90 2024-04-22T06:43:50.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b88375640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010dc50 con 0x7f9b80103b40 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b801986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b801986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58268/0 (socket says 172.21.15.79:58268) 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 -- 172.21.15.79:0/2711748217 learned_addr learned my addr 172.21.15.79:0/2711748217 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b86111640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 0x7f9b80198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 -- 172.21.15.79:0/2711748217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 msgr2=0x7f9b8019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b86912640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 0x7f9b8019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 0x7f9b8019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 -- 172.21.15.79:0/2711748217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 msgr2=0x7f9b80198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 0x7f9b80198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 -- 172.21.15.79:0/2711748217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8019fda0 con 0x7f9b80103180 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b86111640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 0x7f9b80198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b86912640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 0x7f9b8019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:50.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b85910640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b801986c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c006010 tx=0x7f9b7c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c043020 con 0x7f9b80103180 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801a0030 con 0x7f9b80103180 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b7c004aa0 con 0x7f9b80103180 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c003c70 con 0x7f9b80103180 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.460+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801a0580 con 0x7f9b80103180 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.461+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b801092b0 con 0x7f9b80103180 2024-04-22T06:43:50.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.462+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9b7c046020 con 0x7f9b80103180 2024-04-22T06:43:50.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.463+0000 7f9b737fe640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9b6c0a3bb0 0x7f9b6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.463+0000 7f9b86111640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9b6c0a3bb0 0x7f9b6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.463+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f9b7c0f2870 con 0x7f9b80103180 2024-04-22T06:43:50.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.463+0000 7f9b86111640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9b6c0a3bb0 0x7f9b6c0a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b80102fe0 tx=0x7f9b74005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:50.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.467+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b7c04d020 con 0x7f9b80103180 2024-04-22T06:43:50.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.682+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b80062b50 con 0x7f9b80103180 2024-04-22T06:43:50.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.683+0000 7f9b737fe640 1 -- 172.21.15.79:0/2711748217 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f9b7c0bf260 con 0x7f9b80103180 2024-04-22T06:43:50.681 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:50.681 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:42:28.714055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24436,24440,24467]},"id":1}]} 2024-04-22T06:43:50.682 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:43:50.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9b6c0a3bb0 msgr2=0x7f9b6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9b6c0a3bb0 0x7f9b6c0a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b80102fe0 tx=0x7f9b74005e70 comp rx=0 tx=0).stop 2024-04-22T06:43:50.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 msgr2=0x7f9b801986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b801986c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c006010 tx=0x7f9b7c0042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 shutdown_connections 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9b6c0a3bb0 0x7f9b6c0a6070 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b80103b40 0x7f9b8019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9b80103180 0x7f9b801986c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 --2- 172.21.15.79:0/2711748217 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b80101f90 0x7f9b80198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 >> 172.21.15.79:0/2711748217 conn(0x7f9b800fd500 msgr2=0x7f9b80106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 shutdown_connections 2024-04-22T06:43:50.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.685+0000 7f9b88375640 1 -- 172.21.15.79:0/2711748217 wait complete. 2024-04-22T06:43:50.694 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 -- 172.21.15.79:0/3010407280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 msgr2=0x7f5c6c1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 --2- 172.21.15.79:0/3010407280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c1092b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009ca0 tx=0x7f5c6402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 -- 172.21.15.79:0/3010407280 shutdown_connections 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 --2- 172.21.15.79:0/3010407280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c6c101fd0 0x7f5c6c102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 --2- 172.21.15.79:0/3010407280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c1092b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 --2- 172.21.15.79:0/3010407280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c6c100e80 0x7f5c6c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 -- 172.21.15.79:0/3010407280 >> 172.21.15.79:0/3010407280 conn(0x7f5c6c0fc410 msgr2=0x7f5c6c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 -- 172.21.15.79:0/3010407280 shutdown_connections 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.795+0000 7f5c7150e640 1 -- 172.21.15.79:0/3010407280 wait complete. 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 Processor -- start 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 -- start start 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c6c100e80 0x7f5c6c198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c6c101fd0 0x7f5c6c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c10db20 con 0x7f5c6c102990 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c10d9a0 con 0x7f5c6c100e80 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c7150e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c10dca0 con 0x7f5c6c101fd0 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58286/0 (socket says 172.21.15.79:58286) 2024-04-22T06:43:50.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 -- 172.21.15.79:0/2548247878 learned_addr learned my addr 172.21.15.79:0/2548247878 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6affd640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c6c100e80 0x7f5c6c198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 -- 172.21.15.79:0/2548247878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c6c101fd0 msgr2=0x7f5c6c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6a7fc640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c6c101fd0 0x7f5c6c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c6c101fd0 0x7f5c6c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 -- 172.21.15.79:0/2548247878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c6c100e80 msgr2=0x7f5c6c198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c6c100e80 0x7f5c6c198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.796+0000 7f5c6b7fe640 1 -- 172.21.15.79:0/2548247878 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c19fd70 con 0x7f5c6c102990 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.797+0000 7f5c6b7fe640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c19f700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5c6402c700 tx=0x7f5c64034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.797+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c64039040 con 0x7f5c6c102990 2024-04-22T06:43:50.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.797+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c64044070 con 0x7f5c6c102990 2024-04-22T06:43:50.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.797+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c1a0000 con 0x7f5c6c102990 2024-04-22T06:43:50.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.797+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c640042a0 con 0x7f5c6c102990 2024-04-22T06:43:50.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.797+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c1a04f0 con 0x7f5c6c102990 2024-04-22T06:43:50.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.798+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6c108190 con 0x7f5c6c102990 2024-04-22T06:43:50.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.799+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5c64003970 con 0x7f5c6c102990 2024-04-22T06:43:50.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.799+0000 7f5c53fff640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5c540a3bb0 0x7f5c540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:50.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.800+0000 7f5c6affd640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5c540a3bb0 0x7f5c540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:50.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.800+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f5c640f3240 con 0x7f5c6c102990 2024-04-22T06:43:50.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.800+0000 7f5c6affd640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5c540a3bb0 0x7f5c540a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c005fd0 tx=0x7f5c5c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:50.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.803+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c640bfc30 con 0x7f5c6c102990 2024-04-22T06:43:50.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.980+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c6c062b50 con 0x7f5c6c102990 2024-04-22T06:43:50.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.982+0000 7f5c53fff640 1 -- 172.21.15.79:0/2548247878 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22650 (secure 0 0 0) 0x7f5c640bf530 con 0x7f5c6c102990 2024-04-22T06:43:50.981 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:50.981 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":87,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:43:17.795510+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:43:50.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5c540a3bb0 msgr2=0x7f5c540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5c540a3bb0 0x7f5c540a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c005fd0 tx=0x7f5c5c00b390 comp rx=0 tx=0).stop 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 msgr2=0x7f5c6c19f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c19f700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5c6402c700 tx=0x7f5c64034760 comp rx=0 tx=0).stop 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 shutdown_connections 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5c540a3bb0 0x7f5c540a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c6c102990 0x7f5c6c19f700 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c6c101fd0 0x7f5c6c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 --2- 172.21.15.79:0/2548247878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c6c100e80 0x7f5c6c198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 >> 172.21.15.79:0/2548247878 conn(0x7f5c6c0fc410 msgr2=0x7f5c6c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 shutdown_connections 2024-04-22T06:43:50.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:50.984+0000 7f5c7150e640 1 -- 172.21.15.79:0/2548247878 wait complete. 2024-04-22T06:43:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:50 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/512358116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:50 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/319374516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:50 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3291082532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:50 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2711748217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:50.994 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 -- 172.21.15.79:0/2585254698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 msgr2=0x7fdfa01012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/2585254698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa01012c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009900 tx=0x7fdf9002bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 -- 172.21.15.79:0/2585254698 shutdown_connections 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/2585254698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 0x7fdfa0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/2585254698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 0x7fdfa01092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/2585254698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa01012c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 -- 172.21.15.79:0/2585254698 >> 172.21.15.79:0/2585254698 conn(0x7fdfa00fc410 msgr2=0x7fdfa00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 -- 172.21.15.79:0/2585254698 shutdown_connections 2024-04-22T06:43:51.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.093+0000 7fdfa50c8640 1 -- 172.21.15.79:0/2585254698 wait complete. 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 Processor -- start 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 -- start start 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 0x7fdfa019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 0x7fdfa019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010c9f0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010c870 con 0x7fdfa0101ff0 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdfa50c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010cb70 con 0x7fdfa01029b0 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdf9ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdf9ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa006b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58310/0 (socket says 172.21.15.79:58310) 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdf9ed9d640 1 -- 172.21.15.79:0/3279857537 learned_addr learned my addr 172.21.15.79:0/3279857537 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdf9e59c640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 0x7fdfa019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.094+0000 7fdf9f59e640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 0x7fdfa019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9ed9d640 1 -- 172.21.15.79:0/3279857537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 msgr2=0x7fdfa019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9ed9d640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 0x7fdfa019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9ed9d640 1 -- 172.21.15.79:0/3279857537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 msgr2=0x7fdfa019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9ed9d640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 0x7fdfa019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9ed9d640 1 -- 172.21.15.79:0/3279857537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa019fe20 con 0x7fdfa0100ea0 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9e59c640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 0x7fdfa019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9f59e640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 0x7fdfa019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf9ed9d640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa006b1f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdf90006010 tx=0x7fdf900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90043020 con 0x7fdfa0100ea0 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa01a0050 con 0x7fdfa0100ea0 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf90004aa0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90003c70 con 0x7fdfa0100ea0 2024-04-22T06:43:51.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.095+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa01a05f0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.096+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa01081b0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.097+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdf90046020 con 0x7fdfa0100ea0 2024-04-22T06:43:51.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.097+0000 7fdf87fff640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdf880a3bb0 0x7fdf880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.098+0000 7fdf9e59c640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdf880a3bb0 0x7fdf880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.098+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fdf900f28b0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.098+0000 7fdf9e59c640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdf880a3bb0 0x7fdf880a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdf940058c0 tx=0x7fdf94005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.102+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf9004d020 con 0x7fdfa0100ea0 2024-04-22T06:43:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:50 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/512358116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:50 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/319374516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:50 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3291082532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:51.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:50 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2711748217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:51.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:50 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/512358116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:50 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/319374516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:50 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3291082532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:51.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:50 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2711748217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:51.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.318+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfa002d5c0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.318+0000 7fdf87fff640 1 -- 172.21.15.79:0/3279857537 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fdf900bf2a0 con 0x7fdfa0100ea0 2024-04-22T06:43:51.317 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:43:51.317 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:42:28.714055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24436,24440,24467]},"id":1}]} 2024-04-22T06:43:51.318 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:43:51.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdf880a3bb0 msgr2=0x7fdf880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdf880a3bb0 0x7fdf880a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdf940058c0 tx=0x7fdf94005850 comp rx=0 tx=0).stop 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 msgr2=0x7fdfa006b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa006b1f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdf90006010 tx=0x7fdf900042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 shutdown_connections 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fdf880a3bb0 0x7fdf880a6070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdfa0101ff0 0x7fdfa019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdfa01029b0 0x7fdfa019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 --2- 172.21.15.79:0/3279857537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfa0100ea0 0x7fdfa006b1f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.320+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 >> 172.21.15.79:0/3279857537 conn(0x7fdfa00fc410 msgr2=0x7fdfa0105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.321+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 shutdown_connections 2024-04-22T06:43:51.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.321+0000 7fdfa50c8640 1 -- 172.21.15.79:0/3279857537 wait complete. 2024-04-22T06:43:51.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:42:24.011937+0000', 'modified': '2024-04-22T06:42:28.714055+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24467, 'mds_1': 24440, 'mds_2': 24436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24406': {'gid': 24406, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3775212644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3775212644}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3775212644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6835/2543246858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 2543246858}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 2543246858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/678626858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 678626858}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 678626858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.193:6839/746968688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 746968688}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 746968688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.190:6837/2879949884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 2879949884}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 2879949884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.190:6839/4154999918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 4154999918}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 4154999918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24467, 'qdb_cluster': [24436, 24440, 24467]} 2024-04-22T06:43:51.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T06:43:51.331 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.430+0000 7fc323b3d640 1 -- 172.21.15.79:0/779138552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c101f90 msgr2=0x7fc31c1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.430+0000 7fc323b3d640 1 --2- 172.21.15.79:0/779138552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c101f90 0x7fc31c1023b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc318009900 tx=0x7fc31802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- 172.21.15.79:0/779138552 shutdown_connections 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 --2- 172.21.15.79:0/779138552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103b40 0x7fc31c10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 --2- 172.21.15.79:0/779138552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc31c103180 0x7fc31c103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 --2- 172.21.15.79:0/779138552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c101f90 0x7fc31c1023b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- 172.21.15.79:0/779138552 >> 172.21.15.79:0/779138552 conn(0x7fc31c0fd500 msgr2=0x7fc31c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- 172.21.15.79:0/779138552 shutdown_connections 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- 172.21.15.79:0/779138552 wait complete. 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 Processor -- start 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- start start 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc31c101f90 0x7fc31c198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103180 0x7fc31c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 0x7fc31c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c10dad0 con 0x7fc31c103b40 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c10d950 con 0x7fc31c103180 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc323b3d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c10dc50 con 0x7fc31c101f90 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc3220da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 0x7fc31c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc3220da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 0x7fc31c19f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58322/0 (socket says 172.21.15.79:58322) 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.431+0000 7fc3220da640 1 -- 172.21.15.79:0/2536150478 learned_addr learned my addr 172.21.15.79:0/2536150478 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3218d9640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc31c101f90 0x7fc31c198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3210d8640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103180 0x7fc31c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3220da640 1 -- 172.21.15.79:0/2536150478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc31c101f90 msgr2=0x7fc31c198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3220da640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc31c101f90 0x7fc31c198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3220da640 1 -- 172.21.15.79:0/2536150478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103180 msgr2=0x7fc31c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3220da640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103180 0x7fc31c198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3220da640 1 -- 172.21.15.79:0/2536150478 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c19fe60 con 0x7fc31c103b40 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3210d8640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103180 0x7fc31c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc3220da640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 0x7fc31c19f7f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc31400cad0 tx=0x7fc31400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc312ffd640 1 -- 172.21.15.79:0/2536150478 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31400ae20 con 0x7fc31c103b40 2024-04-22T06:43:51.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c1a0150 con 0x7fc31c103b40 2024-04-22T06:43:51.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc312ffd640 1 -- 172.21.15.79:0/2536150478 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc314005020 con 0x7fc31c103b40 2024-04-22T06:43:51.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc312ffd640 1 -- 172.21.15.79:0/2536150478 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31400f6b0 con 0x7fc31c103b40 2024-04-22T06:43:51.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.432+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31c1a06a0 con 0x7fc31c103b40 2024-04-22T06:43:51.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.434+0000 7fc312ffd640 1 -- 172.21.15.79:0/2536150478 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc3140051d0 con 0x7fc31c103b40 2024-04-22T06:43:51.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 Processor -- start 2024-04-22T06:43:51.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 -- start start 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116e1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e1830 0x5573116e1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 0x5573116e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557311645b10 con 0x5573116e22d0 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5573115e3240 con 0x5573116e0e40 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc326809740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557311641ad0 con 0x5573116e1830 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc3210d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 0x5573116e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc312ffd640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc3080a3bb0 0x7fc3080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc3210d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 0x5573116e8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58324/0 (socket says 172.21.15.79:58324) 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.435+0000 7fc3210d8640 1 -- 172.21.15.79:0/461187082 learned_addr learned my addr 172.21.15.79:0/461187082 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116e1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3218d9640 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e1830 0x5573116e1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3220da640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc3080a3bb0 0x7fc3080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 -- 172.21.15.79:0/461187082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e1830 msgr2=0x5573116e1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc312ffd640 1 -- 172.21.15.79:0/2536150478 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc3140d0510 con 0x7fc31c103b40 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e1830 0x5573116e1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 -- 172.21.15.79:0/461187082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 msgr2=0x5573116e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116e1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 -- 172.21.15.79:0/461187082 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573113582d0 con 0x5573116e22d0 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3210d8640 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 0x5573116e8eb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00bfd0 tx=0x7fc30c00b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0a8833bcc11e3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc3220da640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc3080a3bb0 0x7fc3080a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc314099170 tx=0x7fc31406e230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc307fff640 1 -- 172.21.15.79:0/461187082 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c004220 con 0x5573116e22d0 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc307fff640 1 -- 172.21.15.79:0/461187082 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c010e60 con 0x5573116e22d0 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.436+0000 7fc307fff640 1 -- 172.21.15.79:0/461187082 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c003be0 con 0x5573116e22d0 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 -- 172.21.15.79:0/461187082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 msgr2=0x5573116e8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 0x5573116e8eb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00bfd0 tx=0x7fc30c00b680 comp rx=0 tx=0).stop 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 -- 172.21.15.79:0/461187082 shutdown_connections 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e22d0 0x5573116e8eb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e1830 0x5573116e1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 --2- 172.21.15.79:0/461187082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 -- 172.21.15.79:0/461187082 >> 172.21.15.79:0/461187082 conn(0x5573116dcf40 msgr2=0x5573116db490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 -- 172.21.15.79:0/461187082 shutdown_connections 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 -- 172.21.15.79:0/461187082 wait complete. 2024-04-22T06:43:51.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 -- start start 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.437+0000 7fc326809740 1 Processor -- start 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116eaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5573116e94b0 con 0x5573116e1830 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5573115ce190 con 0x5573116e0e40 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5573116e9630 con 0x5573116e22d0 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 10 client.0 _ll_register_callbacks cb 0x5573115c98b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58328/0 (socket says 172.21.15.79:58328) 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 -- 172.21.15.79:0/377069282 learned_addr learned my addr 172.21.15.79:0/377069282 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3210d8640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3210d8640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116eaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 -- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 msgr2=0x5573116f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 -- 172.21.15.79:0/377069282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 msgr2=0x5573116eaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116eaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 -- 172.21.15.79:0/377069282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573114226b0 con 0x5573116e1830 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3210d8640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116eaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3218d9640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116eb400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc31c102fe0 tx=0x7fc3180043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fefbaa54807e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3057fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3057fa640 1 -- 172.21.15.79:0/377069282 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318005250 con 0x5573116e1830 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc3057fa640 1 -- 172.21.15.79:0/377069282 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc3180053f0 con 0x5573116e1830 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.438+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc30c010e60 con 0x5573116e1830 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc3057fa640 1 -- 172.21.15.79:0/377069282 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc3180046c0 con 0x5573116e1830 2024-04-22T06:43:51.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 10 client.15042 fetch_fsmap learned FSMap version 21 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5573116f57b0 con 0x5573116e1830 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc3057fa640 1 -- 172.21.15.79:0/377069282 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c010e60 con 0x5573116e1830 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc3057fa640 1 -- 172.21.15.79:0/377069282 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fc318003920 con 0x5573116e1830 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 10 client.15042 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 10 client.15042 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 20 client.15042 populate_metadata read hostname 'smithi079' 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5573116f7e50 0x5573116f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 4 client.15042 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557311653530 con 0x5573116f7e50 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.439+0000 7fc3210d8640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5573116f7e50 0x5573116f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.440+0000 7fc3057fa640 10 client.15042 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:43:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.440+0000 7fc3210d8640 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5573116f7e50 0x5573116f92c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4e2ed4a0908cfd6 server_cookie=c6565491023e00d9 in_seq=0 out_seq=0 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.440+0000 7fc3057fa640 1 -- 172.21.15.79:0/377069282 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557311653530 con 0x5573116f7e50 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.440+0000 7fc3057fa640 10 client.15042 handle_command_reply: tid=0 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 client.15042 shutdown 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 msgr2=0x5573116eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116eb400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc31c102fe0 tx=0x7fc3180043b0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 shutdown_connections 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116eb400 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5573116f7e50 0x5573116f92c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 --2- 172.21.15.79:0/377069282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116eaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc3057fa640 0 client.15042 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 >> 172.21.15.79:0/377069282 conn(0x5573116dcf40 msgr2=0x5573116daa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 shutdown_connections 2024-04-22T06:43:51.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 1 -- 172.21.15.79:0/377069282 wait complete. 2024-04-22T06:43:51.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.441+0000 7fc326809740 20 client.15042 trim_cache size 0 max 16384 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 Processor -- start 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 -- start start 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 0x5573116e1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e1830 0x5573116f2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116eaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5573116f7600 con 0x5573116e0e40 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5573116f7cd0 con 0x5573116e1830 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc326809740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5573116f83a0 con 0x5573116e22d0 2024-04-22T06:43:51.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 0x5573116e1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 0x5573116e1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58336/0 (socket says 172.21.15.79:58336) 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 -- 172.21.15.79:0/1678962653 learned_addr learned my addr 172.21.15.79:0/1678962653 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 -- 172.21.15.79:0/1678962653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 msgr2=0x5573116eaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116eaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 -- 172.21.15.79:0/1678962653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e1830 msgr2=0x5573116f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e1830 0x5573116f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 -- 172.21.15.79:0/1678962653 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573116dcd40 con 0x5573116e0e40 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3210d8640 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 0x5573116e1c10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b8b0 tx=0x7fc30c00b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e54dac9507d0d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3077fe640 1 -- 172.21.15.79:0/1678962653 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c032070 con 0x5573116e0e40 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3077fe640 1 -- 172.21.15.79:0/1678962653 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c02d900 con 0x5573116e0e40 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.459+0000 7fc3077fe640 1 -- 172.21.15.79:0/1678962653 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0052a0 con 0x5573116e0e40 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 -- 172.21.15.79:0/1678962653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 msgr2=0x5573116e1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 0x5573116e1c10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b8b0 tx=0x7fc30c00b680 comp rx=0 tx=0).stop 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 -- 172.21.15.79:0/1678962653 shutdown_connections 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e1830 0x5573116f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116eaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 --2- 172.21.15.79:0/1678962653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e0e40 0x5573116e1c10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 -- 172.21.15.79:0/1678962653 >> 172.21.15.79:0/1678962653 conn(0x5573116dcf40 msgr2=0x5573116e8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 -- 172.21.15.79:0/1678962653 shutdown_connections 2024-04-22T06:43:51.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 -- 172.21.15.79:0/1678962653 wait complete. 2024-04-22T06:43:51.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 -- start start 2024-04-22T06:43:51.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.460+0000 7fc326809740 1 Processor -- start 2024-04-22T06:43:51.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5573116f9140 con 0x5573116e1830 2024-04-22T06:43:51.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5573116f8a70 con 0x5573116e0e40 2024-04-22T06:43:51.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5573116f9810 con 0x5573116e22d0 2024-04-22T06:43:51.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 10 client.0 _ll_register_callbacks cb 0x7fc318005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:51.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58348/0 (socket says 172.21.15.79:58348) 2024-04-22T06:43:51.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 -- 172.21.15.79:0/625952873 learned_addr learned my addr 172.21.15.79:0/625952873 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3210d8640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 -- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 msgr2=0x5573116f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 -- 172.21.15.79:0/625952873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 msgr2=0x5573116ec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116ec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 -- 172.21.15.79:0/625952873 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573116f1d60 con 0x5573116e1830 2024-04-22T06:43:51.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3210d8640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3218d9640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116ecaf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc318009900 tx=0x7fc318005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a955575fa1384c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3057fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:43:51.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3057fa640 1 -- 172.21.15.79:0/625952873 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318059070 con 0x5573116e1830 2024-04-22T06:43:51.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc30c02d900 con 0x5573116e1830 2024-04-22T06:43:51.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3057fa640 1 -- 172.21.15.79:0/625952873 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc3180035f0 con 0x5573116e1830 2024-04-22T06:43:51.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc3057fa640 1 -- 172.21.15.79:0/625952873 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc318065050 con 0x5573116e1830 2024-04-22T06:43:51.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.461+0000 7fc326809740 10 client.15048 fetch_fsmap learned FSMap version 21 2024-04-22T06:43:51.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55731175e5a0 con 0x5573116e1830 2024-04-22T06:43:51.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc3057fa640 1 -- 172.21.15.79:0/625952873 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c02d900 con 0x5573116e1830 2024-04-22T06:43:51.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc3057fa640 1 -- 172.21.15.79:0/625952873 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fc318004b40 con 0x5573116e1830 2024-04-22T06:43:51.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 10 client.15048 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:43:51.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 10 client.15048 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:43:51.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 20 client.15048 populate_metadata read hostname 'smithi079' 2024-04-22T06:43:51.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x557311763230 0x557311761f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 4 client.15048 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "status"}) -- 0x55731164d3a0 con 0x557311763230 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc3210d8640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x557311763230 0x557311761f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc3057fa640 10 client.15048 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.462+0000 7fc3210d8640 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x557311763230 0x557311761f40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c2b12399ec6f2f10 server_cookie=30053b3b8ed34641 in_seq=0 out_seq=0 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc3057fa640 1 -- 172.21.15.79:0/625952873 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55731164d3a0 con 0x557311763230 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc3057fa640 10 client.15048 handle_command_reply: tid=0 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 client.15048 shutdown 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 msgr2=0x5573116ecaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116ecaf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc318009900 tx=0x7fc318005790 comp rx=0 tx=0).stop 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 shutdown_connections 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573116e22d0 0x5573116f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5573116e1830 0x5573116ecaf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x557311763230 0x557311761f40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 --2- 172.21.15.79:0/625952873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5573116e0e40 0x5573116ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 >> 172.21.15.79:0/625952873 conn(0x5573116dcf40 msgr2=0x5573116e8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc3057fa640 0 client.15048 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:43:51.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 shutdown_connections 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 1 -- 172.21.15.79:0/625952873 wait complete. 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc326809740 20 client.15048 trim_cache size 0 max 16384 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 2, 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24436, 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:43:51.475 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:43:51.476 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 83.760295510000006, 2024-04-22T06:43:51.476 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 21, 2024-04-22T06:43:51.476 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 87, 2024-04-22T06:43:51.476 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:43:51.476 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 101.6012494 2024-04-22T06:43:51.476 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:43:51.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc3080a3bb0 msgr2=0x7fc3080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc323b3d640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc3080a3bb0 0x7fc3080a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc314099170 tx=0x7fc31406e230 comp rx=0 tx=0).stop 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 msgr2=0x7fc31c19f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.463+0000 7fc323b3d640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 0x7fc31c19f7f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc31400cad0 tx=0x7fc31400cf90 comp rx=0 tx=0).stop 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 shutdown_connections 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc3080a3bb0 0x7fc3080a6070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc31c103180 0x7fc31c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc31c103b40 0x7fc31c19f7f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 --2- 172.21.15.79:0/2536150478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc31c101f90 0x7fc31c198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 >> 172.21.15.79:0/2536150478 conn(0x7fc31c0fd500 msgr2=0x7fc31c1065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 shutdown_connections 2024-04-22T06:43:51.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.464+0000 7fc323b3d640 1 -- 172.21.15.79:0/2536150478 wait complete. 2024-04-22T06:43:51.479 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 -- 172.21.15.79:0/1305432840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 msgr2=0x7f9a98107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 --2- 172.21.15.79:0/1305432840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98107750 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 -- 172.21.15.79:0/1305432840 shutdown_connections 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 --2- 172.21.15.79:0/1305432840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98107c90 0x7f9a9810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 --2- 172.21.15.79:0/1305432840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98107750 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 --2- 172.21.15.79:0/1305432840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a98102a30 0x7f9a98104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 -- 172.21.15.79:0/1305432840 >> 172.21.15.79:0/1305432840 conn(0x7f9a980fc440 msgr2=0x7f9a980fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 -- 172.21.15.79:0/1305432840 shutdown_connections 2024-04-22T06:43:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.579+0000 7f9a9e115640 1 -- 172.21.15.79:0/1305432840 wait complete. 2024-04-22T06:43:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 Processor -- start 2024-04-22T06:43:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 -- start start 2024-04-22T06:43:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98102a30 0x7f9a98198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a98107c90 0x7f9a9819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810daf0 con 0x7f9a98105360 2024-04-22T06:43:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810d970 con 0x7f9a98107c90 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a9e115640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810dc70 con 0x7f9a98102a30 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58356/0 (socket says 172.21.15.79:58356) 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 -- 172.21.15.79:0/4019508200 learned_addr learned my addr 172.21.15.79:0/4019508200 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a977fe640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98102a30 0x7f9a98198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 -- 172.21.15.79:0/4019508200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98102a30 msgr2=0x7f9a98198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98102a30 0x7f9a98198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 -- 172.21.15.79:0/4019508200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a98107c90 msgr2=0x7f9a9819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a98107c90 0x7f9a9819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a96ffd640 1 -- 172.21.15.79:0/4019508200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9819fe60 con 0x7f9a98105360 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.580+0000 7f9a977fe640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98102a30 0x7f9a98198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.581+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98198780 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c000c00 tx=0x7f9a8c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.581+0000 7f9a94ff9640 1 -- 172.21.15.79:0/4019508200 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c043020 con 0x7f9a98105360 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.581+0000 7f9a94ff9640 1 -- 172.21.15.79:0/4019508200 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a8c004aa0 con 0x7f9a98105360 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.581+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a981a00f0 con 0x7f9a98105360 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.581+0000 7f9a94ff9640 1 -- 172.21.15.79:0/4019508200 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c003d00 con 0x7f9a98105360 2024-04-22T06:43:51.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.581+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a981a0640 con 0x7f9a98105360 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.583+0000 7f9a94ff9640 1 -- 172.21.15.79:0/4019508200 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9a8c046020 con 0x7f9a98105360 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 Processor -- start 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 -- start start 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6a250 0x563249f6a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 0x563249f6b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6b6e0 0x563249f722c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563249ecab10 con 0x563249f6ac40 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563249e68240 con 0x563249f6b6e0 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563249ec6ad0 con 0x563249f6a250 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 0x563249f6b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 0x563249f6b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58368/0 (socket says 172.21.15.79:58368) 2024-04-22T06:43:51.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 -- 172.21.15.79:0/2411460302 learned_addr learned my addr 172.21.15.79:0/2411460302 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a94ff9640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a800a3bb0 0x7f9a800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6b6e0 0x563249f722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a97fff640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a800a3bb0 0x7f9a800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 -- 172.21.15.79:0/2411460302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6a250 msgr2=0x563249f6a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6a250 0x563249f6a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 -- 172.21.15.79:0/2411460302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6b6e0 msgr2=0x563249f722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6b6e0 0x563249f722c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 -- 172.21.15.79:0/2411460302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563249bdd2d0 con 0x563249f6ac40 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6b6e0 0x563249f722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a94ff9640 1 -- 172.21.15.79:0/4019508200 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f9a8c0f2a30 con 0x7f9a98105360 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.584+0000 7f9a977fe640 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 0x563249f6b0c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800cca0 tx=0x7f9a8800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40337c50e7806938 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9a97fff640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a800a3bb0 0x7f9a800a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x563249f61200 tx=0x7f9a900094a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9a7dffb640 1 -- 172.21.15.79:0/2411460302 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88005220 con 0x563249f6ac40 2024-04-22T06:43:51.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9a7dffb640 1 -- 172.21.15.79:0/2411460302 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a88004c50 con 0x563249f6ac40 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9a7dffb640 1 -- 172.21.15.79:0/2411460302 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88010750 con 0x563249f6ac40 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2411460302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 msgr2=0x563249f6b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 0x563249f6b0c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800cca0 tx=0x7f9a8800e720 comp rx=0 tx=0).stop 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2411460302 shutdown_connections 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6b6e0 0x563249f722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6ac40 0x563249f6b0c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2411460302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6a250 0x563249f6a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2411460302 >> 172.21.15.79:0/2411460302 conn(0x563249f61f40 msgr2=0x563249f60490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2411460302 shutdown_connections 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.585+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2411460302 wait complete. 2024-04-22T06:43:51.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 -- start start 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 Processor -- start 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6a250 0x563249f74240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6ac40 0x563249f74780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 0x563249f7b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563249f728c0 con 0x563249f6b6e0 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563249e53190 con 0x563249f6a250 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563249f72a40 con 0x563249f6ac40 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 0x563249f7b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 0x563249f7b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58372/0 (socket says 172.21.15.79:58372) 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9a97fff640 1 -- 172.21.15.79:0/304048443 learned_addr learned my addr 172.21.15.79:0/304048443 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9aa0de7740 10 client.0 _ll_register_callbacks cb 0x563249e4e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.586+0000 7f9a977fe640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6a250 0x563249f74240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6ac40 0x563249f74780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a97fff640 1 -- 172.21.15.79:0/304048443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6ac40 msgr2=0x563249f74780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a97fff640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6ac40 0x563249f74780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a97fff640 1 -- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6a250 msgr2=0x563249f74240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a97fff640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6a250 0x563249f74240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a97fff640 1 -- 172.21.15.79:0/304048443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563249ca76b0 con 0x563249f6b6e0 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6ac40 0x563249f74780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a977fe640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6a250 0x563249f74240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:51.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a97fff640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 0x563249f7b7c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009bf0 tx=0x7f9a90010a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646857f26b2bf130 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a76ffd640 1 -- 172.21.15.79:0/304048443 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90020070 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a76ffd640 1 -- 172.21.15.79:0/304048443 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f9a9000f990 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a88004c50 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a76ffd640 1 -- 172.21.15.79:0/304048443 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a9002c050 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9aa0de7740 10 client.15060 fetch_fsmap learned FSMap version 21 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563249f7ebc0 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a76ffd640 1 -- 172.21.15.79:0/304048443 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a88004c50 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.587+0000 7f9a76ffd640 1 -- 172.21.15.79:0/304048443 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9a90019040 con 0x563249f6b6e0 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9aa0de7740 10 client.15060 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9aa0de7740 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9aa0de7740 20 client.15060 populate_metadata read hostname 'smithi079' 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249f81340 0x563249f827b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9aa0de7740 4 client.15060 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563249ed8530 con 0x563249f81340 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9a977fe640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249f81340 0x563249f827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9a76ffd640 10 client.15060 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.588+0000 7f9a977fe640 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249f81340 0x563249f827b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c002e52b7c1e94df server_cookie=b8448be4d7696966 in_seq=0 out_seq=0 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9a76ffd640 1 -- 172.21.15.79:0/304048443 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563249ed8530 con 0x563249f81340 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9a76ffd640 10 client.15060 handle_command_reply: tid=0 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 client.15060 shutdown 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 msgr2=0x563249f7b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 0x563249f7b7c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009bf0 tx=0x7f9a90010a80 comp rx=0 tx=0).stop 2024-04-22T06:43:51.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 shutdown_connections 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249f81340 0x563249f827b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6b6e0 0x563249f7b7c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6ac40 0x563249f74780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/304048443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6a250 0x563249f74240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9a76ffd640 0 client.15060 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 >> 172.21.15.79:0/304048443 conn(0x563249f61f40 msgr2=0x563249f5fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 shutdown_connections 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 1 -- 172.21.15.79:0/304048443 wait complete. 2024-04-22T06:43:51.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.589+0000 7f9aa0de7740 20 client.15060 trim_cache size 0 max 16384 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 Processor -- start 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 -- start start 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f6e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f7b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f5b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563249f80af0 con 0x563249f6a250 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563249f811c0 con 0x563249f6ac40 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563249f81890 con 0x563249f6b6e0 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9a977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f6e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9a977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f6e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58386/0 (socket says 172.21.15.79:58386) 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.607+0000 7f9a977fe640 1 -- 172.21.15.79:0/2533730001 learned_addr learned my addr 172.21.15.79:0/2533730001 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a977fe640 1 -- 172.21.15.79:0/2533730001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 msgr2=0x563249f5b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a97fff640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f5b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f7b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a977fe640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f5b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a977fe640 1 -- 172.21.15.79:0/2533730001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 msgr2=0x563249f7b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a977fe640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f7b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a977fe640 1 -- 172.21.15.79:0/2533730001 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563249f61230 con 0x563249f6a250 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a97fff640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f5b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f7b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a977fe640 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f6e2a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800ec20 tx=0x7f9a8800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4298d7f3f6b19371 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a7d7fa640 1 -- 172.21.15.79:0/2533730001 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88029110 con 0x563249f6a250 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a7d7fa640 1 -- 172.21.15.79:0/2533730001 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a88029760 con 0x563249f6a250 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9a7d7fa640 1 -- 172.21.15.79:0/2533730001 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88010ab0 con 0x563249f6a250 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2533730001 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 msgr2=0x563249f6e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.608+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f6e2a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800ec20 tx=0x7f9a8800e720 comp rx=0 tx=0).stop 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2533730001 shutdown_connections 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f7b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f5b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/2533730001 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f6e2a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2533730001 >> 172.21.15.79:0/2533730001 conn(0x563249f61f40 msgr2=0x563249f60350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2533730001 shutdown_connections 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- 172.21.15.79:0/2533730001 wait complete. 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- start start 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 Processor -- start 2024-04-22T06:43:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f77010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f77550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f7a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563249f82630 con 0x563249f6a250 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563249f81f60 con 0x563249f6ac40 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563249f82d00 con 0x563249f6b6e0 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9aa0de7740 10 client.0 _ll_register_callbacks cb 0x7f9a9000ef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9a977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f77010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9a977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f77010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58392/0 (socket says 172.21.15.79:58392) 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9a977fe640 1 -- 172.21.15.79:0/3208363976 learned_addr learned my addr 172.21.15.79:0/3208363976 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f77550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.609+0000 7f9a97fff640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f7a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a977fe640 1 -- 172.21.15.79:0/3208363976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 msgr2=0x563249f7a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a977fe640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f7a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a977fe640 1 -- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 msgr2=0x563249f77550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a977fe640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f77550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a977fe640 1 -- 172.21.15.79:0/3208363976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563249f7aa00 con 0x563249f6a250 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a977fe640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f77010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800efe0 tx=0x7f9a8800c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a9346718bca24d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a76ffd640 1 -- 172.21.15.79:0/3208363976 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88036030 con 0x563249f6a250 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a76ffd640 1 -- 172.21.15.79:0/3208363976 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f9a8802e040 con 0x563249f6a250 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a88029760 con 0x563249f6a250 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a76ffd640 1 -- 172.21.15.79:0/3208363976 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a88029910 con 0x563249f6a250 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 10 client.15066 fetch_fsmap learned FSMap version 21 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563249fe72f0 con 0x563249f6a250 2024-04-22T06:43:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a76ffd640 1 -- 172.21.15.79:0/3208363976 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a88029760 con 0x563249f6a250 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9a76ffd640 1 -- 172.21.15.79:0/3208363976 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9a8803b450 con 0x563249f6a250 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 10 client.15066 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 10 client.15066 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 20 client.15066 populate_metadata read hostname 'smithi079' 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249feb790 0x563249fea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 4 client.15066 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.610+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "status"}) -- 0x563249ed23a0 con 0x563249feb790 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249feb790 0x563249fea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9a76ffd640 10 client.15066 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9a96ffd640 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249feb790 0x563249fea4a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65aec625d3ec27f server_cookie=aee5f8403c854317 in_seq=0 out_seq=0 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9a76ffd640 1 -- 172.21.15.79:0/3208363976 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563249ed23a0 con 0x563249feb790 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9a76ffd640 10 client.15066 handle_command_reply: tid=0 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 client.15066 shutdown 2024-04-22T06:43:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 msgr2=0x563249f77010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f77010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800efe0 tx=0x7f9a8800c510 comp rx=0 tx=0).stop 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 shutdown_connections 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x563249feb790 0x563249fea4a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563249f6ac40 0x563249f77550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563249f6b6e0 0x563249f7a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9a76ffd640 0 client.15066 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 --2- 172.21.15.79:0/3208363976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563249f6a250 0x563249f77010 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 >> 172.21.15.79:0/3208363976 conn(0x563249f61f40 msgr2=0x563249f80030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.611+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 shutdown_connections 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9aa0de7740 1 -- 172.21.15.79:0/3208363976 wait complete. 2024-04-22T06:43:51.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9aa0de7740 20 client.15066 trim_cache size 0 max 16384 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 1, 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24440, 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 84.916063984999994, 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 21, 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 87, 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 104.593156433 2024-04-22T06:43:51.619 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:43:51.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a800a3bb0 msgr2=0x7f9a800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a800a3bb0 0x7f9a800a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x563249f61200 tx=0x7f9a900094a0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 msgr2=0x7f9a98198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98198780 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c000c00 tx=0x7f9a8c004270 comp rx=0 tx=0).stop 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 shutdown_connections 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f9a800a3bb0 0x7f9a800a6070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a98107c90 0x7f9a9819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9a98105360 0x7f9a98198780 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 --2- 172.21.15.79:0/4019508200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a98102a30 0x7f9a98198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 >> 172.21.15.79:0/4019508200 conn(0x7f9a980fc440 msgr2=0x7f9a981048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 shutdown_connections 2024-04-22T06:43:51.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.612+0000 7f9a9e115640 1 -- 172.21.15.79:0/4019508200 wait complete. 2024-04-22T06:43:51.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:43:51.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 -- 172.21.15.79:0/1485191334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564100e90 msgr2=0x7f05641012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 --2- 172.21.15.79:0/1485191334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564100e90 0x7f05641012b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f055c009900 tx=0x7f055c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 -- 172.21.15.79:0/1485191334 shutdown_connections 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 --2- 172.21.15.79:0/1485191334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 0x7f05641092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 --2- 172.21.15.79:0/1485191334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0564101fe0 0x7f0564102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 --2- 172.21.15.79:0/1485191334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564100e90 0x7f05641012b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.724+0000 7f0568bbd640 1 -- 172.21.15.79:0/1485191334 >> 172.21.15.79:0/1485191334 conn(0x7f05640fc400 msgr2=0x7f05640fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 -- 172.21.15.79:0/1485191334 shutdown_connections 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 -- 172.21.15.79:0/1485191334 wait complete. 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 Processor -- start 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 -- start start 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0564100e90 0x7f0564069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 0x7f056419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 0x7f056419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f056410ca60 con 0x7f0564101fe0 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f056410c8e0 con 0x7f05641029a0 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0568bbd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f056410cbe0 con 0x7f0564100e90 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 0x7f056419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 0x7f056419cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58418/0 (socket says 172.21.15.79:58418) 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.725+0000 7f0561d9b640 1 -- 172.21.15.79:0/4111806865 learned_addr learned my addr 172.21.15.79:0/4111806865 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0562d9d640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 0x7f056419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0561d9b640 1 -- 172.21.15.79:0/4111806865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0564100e90 msgr2=0x7f0564069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0561d9b640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0564100e90 0x7f0564069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0561d9b640 1 -- 172.21.15.79:0/4111806865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 msgr2=0x7f056419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0561d9b640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 0x7f056419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0561d9b640 1 -- 172.21.15.79:0/4111806865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056419fe70 con 0x7f0564101fe0 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0562d9d640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 0x7f056419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0561d9b640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 0x7f056419cf70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f055800b450 tx=0x7f055800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f05537fe640 1 -- 172.21.15.79:0/4111806865 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558015020 con 0x7f0564101fe0 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05641a0160 con 0x7f0564101fe0 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f05537fe640 1 -- 172.21.15.79:0/4111806865 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0558011d40 con 0x7f0564101fe0 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f05537fe640 1 -- 172.21.15.79:0/4111806865 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055800f340 con 0x7f0564101fe0 2024-04-22T06:43:51.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.726+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05641a0730 con 0x7f0564101fe0 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.728+0000 7f05537fe640 1 -- 172.21.15.79:0/4111806865 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0558018020 con 0x7f0564101fe0 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.728+0000 7f056b7b5740 1 Processor -- start 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 -- start start 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f05537fe640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f054c0a3bb0 0x7f054c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 0x5570293c1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c1830 0x5570293c1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c22d0 0x5570293c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557029325b10 con 0x5570293c0e40 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5570292c3240 con 0x5570293c22d0 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056b7b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557029321ad0 con 0x5570293c1830 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f0562d9d640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f054c0a3bb0 0x7f054c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 0x5570293c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 0x5570293c1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58428/0 (socket says 172.21.15.79:58428) 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 -- 172.21.15.79:0/3861182590 learned_addr learned my addr 172.21.15.79:0/3861182590 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f05537fe640 1 -- 172.21.15.79:0/4111806865 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f05580cff50 con 0x7f0564101fe0 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f0562d9d640 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c22d0 0x5570293c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f0561d9b640 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c1830 0x5570293c1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 -- 172.21.15.79:0/3861182590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c1830 msgr2=0x5570293c1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c1830 0x5570293c1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 -- 172.21.15.79:0/3861182590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c22d0 msgr2=0x5570293c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c22d0 0x5570293c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 -- 172.21.15.79:0/3861182590 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570290382d0 con 0x5570293c0e40 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f0562d9d640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f054c0a3bb0 0x7f054c0a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05540146e0 tx=0x7f0554014ae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.729+0000 7f056259c640 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 0x5570293c1260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f055c005770 tx=0x7f055c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b535e9e77849da7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f0550ff9640 1 -- 172.21.15.79:0/3861182590 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c041020 con 0x5570293c0e40 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f0550ff9640 1 -- 172.21.15.79:0/3861182590 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f055c02cb60 con 0x5570293c0e40 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f0550ff9640 1 -- 172.21.15.79:0/3861182590 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0055d0 con 0x5570293c0e40 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 -- 172.21.15.79:0/3861182590 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 msgr2=0x5570293c1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 0x5570293c1260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f055c005770 tx=0x7f055c005d50 comp rx=0 tx=0).stop 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 -- 172.21.15.79:0/3861182590 shutdown_connections 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c22d0 0x5570293c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c1830 0x5570293c1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3861182590 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c0e40 0x5570293c1260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 -- 172.21.15.79:0/3861182590 >> 172.21.15.79:0/3861182590 conn(0x5570293bcf40 msgr2=0x5570293bb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 -- 172.21.15.79:0/3861182590 shutdown_connections 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.730+0000 7f056b7b5740 1 -- 172.21.15.79:0/3861182590 wait complete. 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 -- start start 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 Processor -- start 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 0x5570293d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5570293c9450 con 0x5570293c22d0 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5570292ae190 con 0x5570293c1830 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5570293c95d0 con 0x5570293c0e40 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056b7b5740 10 client.0 _ll_register_callbacks cb 0x5570292a98b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 0x5570293d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 0x5570293d2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58432/0 (socket says 172.21.15.79:58432) 2024-04-22T06:43:51.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 -- 172.21.15.79:0/1367352339 learned_addr learned my addr 172.21.15.79:0/1367352339 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0561d9b640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056259c640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 -- 172.21.15.79:0/1367352339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c0e40 msgr2=0x5570293cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 -- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c1830 msgr2=0x5570293cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 -- 172.21.15.79:0/1367352339 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570291026b0 con 0x5570293c22d0 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f056259c640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:51.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0562d9d640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 0x5570293d2380 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0554011de0 tx=0x7f0554004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe76b53c4d3c3e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.731+0000 7f0549ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f0549ffb640 1 -- 172.21.15.79:0/1367352339 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554024030 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f0549ffb640 1 -- 172.21.15.79:0/1367352339 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f0554004b20 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f055c02cb60 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f0549ffb640 1 -- 172.21.15.79:0/1367352339 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0554004d20 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 10 client.15078 fetch_fsmap learned FSMap version 21 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5570293d5760 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f0549ffb640 1 -- 172.21.15.79:0/1367352339 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f055c02cb60 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f0549ffb640 1 -- 172.21.15.79:0/1367352339 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f055401f480 con 0x5570293c22d0 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 10 client.15078 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 10 client.15078 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 20 client.15078 populate_metadata read hostname 'smithi079' 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570293d7e60 0x5570293d92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 4 client.15078 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.732+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557029333530 con 0x5570293d7e60 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.733+0000 7f056259c640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570293d7e60 0x5570293d92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.733+0000 7f0549ffb640 10 client.15078 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.733+0000 7f056259c640 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570293d7e60 0x5570293d92d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42279d2165168d9 server_cookie=dae9699cb8874f6c in_seq=0 out_seq=0 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f0549ffb640 1 -- 172.21.15.79:0/1367352339 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557029333530 con 0x5570293d7e60 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f0549ffb640 10 client.15078 handle_command_reply: tid=0 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 client.15078 shutdown 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 msgr2=0x5570293d2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 0x5570293d2380 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0554011de0 tx=0x7f0554004ab0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 shutdown_connections 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570293d7e60 0x5570293d92d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c22d0 0x5570293d2380 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f0549ffb640 0 client.15078 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:43:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1367352339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 >> 172.21.15.79:0/1367352339 conn(0x5570293bcf40 msgr2=0x5570293baa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 shutdown_connections 2024-04-22T06:43:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 1 -- 172.21.15.79:0/1367352339 wait complete. 2024-04-22T06:43:51.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.734+0000 7f056b7b5740 20 client.15078 trim_cache size 0 max 16384 2024-04-22T06:43:51.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 Processor -- start 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 -- start start 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5570293d7610 con 0x5570293c1830 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5570293d7ce0 con 0x5570293c0e40 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f056b7b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5570293d83b0 con 0x5570293c22d0 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58440/0 (socket says 172.21.15.79:58440) 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.752+0000 7f0561d9b640 1 -- 172.21.15.79:0/1681505156 learned_addr learned my addr 172.21.15.79:0/1681505156 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0561d9b640 1 -- 172.21.15.79:0/1681505156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 msgr2=0x5570293d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0562d9d640 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0561d9b640 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0561d9b640 1 -- 172.21.15.79:0/1681505156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 msgr2=0x5570293cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0561d9b640 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0561d9b640 1 -- 172.21.15.79:0/1681505156 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570293bcd40 con 0x5570293c1830 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f0561d9b640 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293cb340 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f05580cda60 tx=0x7f05580108a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33e7dadaa62d14a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f054bfff640 1 -- 172.21.15.79:0/1681505156 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580a4eb0 con 0x5570293c1830 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f054bfff640 1 -- 172.21.15.79:0/1681505156 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f055809a2e0 con 0x5570293c1830 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.753+0000 7f054bfff640 1 -- 172.21.15.79:0/1681505156 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580b5d80 con 0x5570293c1830 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- 172.21.15.79:0/1681505156 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 msgr2=0x5570293cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293cb340 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f05580cda60 tx=0x7f05580108a0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- 172.21.15.79:0/1681505156 shutdown_connections 2024-04-22T06:43:51.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293cb340 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- 172.21.15.79:0/1681505156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- 172.21.15.79:0/1681505156 >> 172.21.15.79:0/1681505156 conn(0x5570293bcf40 msgr2=0x5570293d6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- 172.21.15.79:0/1681505156 shutdown_connections 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- 172.21.15.79:0/1681505156 wait complete. 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- start start 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 Processor -- start 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5570293d9150 con 0x5570293c1830 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5570293d8a80 con 0x5570293c0e40 2024-04-22T06:43:51.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5570293d9820 con 0x5570293c22d0 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f056b7b5740 10 client.0 _ll_register_callbacks cb 0x7f0554011a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f0561d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293d0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58448/0 (socket says 172.21.15.79:58448) 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.754+0000 7f0561d9b640 1 -- 172.21.15.79:0/3946881197 learned_addr learned my addr 172.21.15.79:0/3946881197 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056259c640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0561d9b640 1 -- 172.21.15.79:0/3946881197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 msgr2=0x5570293d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0562d9d640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0561d9b640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0561d9b640 1 -- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 msgr2=0x5570293ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0561d9b640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0561d9b640 1 -- 172.21.15.79:0/3946881197 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570293d1810 con 0x5570293c1830 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0561d9b640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293d0ac0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05580986a0 tx=0x7f05580a06b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc2c3189825021e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0549ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0549ffb640 1 -- 172.21.15.79:0/3946881197 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558010500 con 0x5570293c1830 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0549ffb640 1 -- 172.21.15.79:0/3946881197 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f05580a4a20 con 0x5570293c1830 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f055809a2e0 con 0x5570293c1830 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0549ffb640 1 -- 172.21.15.79:0/3946881197 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f05580106a0 con 0x5570293c1830 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 10 client.15084 fetch_fsmap learned FSMap version 21 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55702943e0a0 con 0x5570293c1830 2024-04-22T06:43:51.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0549ffb640 1 -- 172.21.15.79:0/3946881197 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f055809a2e0 con 0x5570293c1830 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f0549ffb640 1 -- 172.21.15.79:0/3946881197 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f05580b0ce0 con 0x5570293c1830 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 10 client.15084 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 20 client.15084 populate_metadata read hostname 'smithi079' 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570294427c0 0x5570294414d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 4 client.15084 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.755+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "status"}) -- 0x55702932d3a0 con 0x5570294427c0 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056259c640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570294427c0 0x5570294414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056259c640 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570294427c0 0x5570294414d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2083ca68d6aca05b server_cookie=b0a92de1a365b26 in_seq=0 out_seq=0 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f0549ffb640 10 client.15084 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f0549ffb640 1 -- 172.21.15.79:0/3946881197 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55702932d3a0 con 0x5570294427c0 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f0549ffb640 10 client.15084 handle_command_reply: tid=0 2024-04-22T06:43:51.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 client.15084 shutdown 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 msgr2=0x5570293d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293d0ac0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f05580986a0 tx=0x7f05580a06b0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 shutdown_connections 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5570294427c0 0x5570294414d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570293c22d0 0x5570293d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5570293c1830 0x5570293d0ac0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f0549ffb640 0 client.15084 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 --2- 172.21.15.79:0/3946881197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570293c0e40 0x5570293ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.756+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 >> 172.21.15.79:0/3946881197 conn(0x5570293bcf40 msgr2=0x5570293c5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 shutdown_connections 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f056b7b5740 1 -- 172.21.15.79:0/3946881197 wait complete. 2024-04-22T06:43:51.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f056b7b5740 20 client.15084 trim_cache size 0 max 16384 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 0, 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24467, 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 87.085253903999998, 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 21, 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 87, 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 93.381229680000004 2024-04-22T06:43:51.764 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:43:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f054c0a3bb0 msgr2=0x7f054c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f054c0a3bb0 0x7f054c0a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05540146e0 tx=0x7f0554014ae0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 msgr2=0x7f056419cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 0x7f056419cf70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f055800b450 tx=0x7f055800b910 comp rx=0 tx=0).stop 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 shutdown_connections 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f054c0a3bb0 0x7f054c0a6070 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05641029a0 0x7f056419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0564101fe0 0x7f056419cf70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 --2- 172.21.15.79:0/4111806865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0564100e90 0x7f0564069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 >> 172.21.15.79:0/4111806865 conn(0x7f05640fc400 msgr2=0x7f0564105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:51.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 shutdown_connections 2024-04-22T06:43:51.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:51.757+0000 7f0568bbd640 1 -- 172.21.15.79:0/4111806865 wait complete. 2024-04-22T06:43:51.768 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T06:43:51.768 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T06:43:51.783 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:43:51.784 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T06:43:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:51 smithi079 ceph-mon[27946]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:43:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:51 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2548247878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:51.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:51 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3279857537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:51 smithi193 ceph-mon[31815]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:43:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:51 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2548247878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:51 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3279857537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:52.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:51 smithi190 ceph-mon[31421]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:43:52.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:51 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2548247878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:52.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:51 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3279857537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:53.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.688+0000 7f3169df1700 1 -- 172.21.15.79:0/3507803121 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102f80 msgr2=0x7f31641033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.688+0000 7f3169df1700 1 --2- 172.21.15.79:0/3507803121 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102f80 0x7f31641033a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f31540099c0 tx=0x7f3154009cd0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 -- 172.21.15.79:0/3507803121 shutdown_connections 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 --2- 172.21.15.79:0/3507803121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3164102660 0x7f3164109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 --2- 172.21.15.79:0/3507803121 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102f80 0x7f31641033a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 --2- 172.21.15.79:0/3507803121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3164101cc0 0x7f3164102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 -- 172.21.15.79:0/3507803121 >> 172.21.15.79:0/3507803121 conn(0x7f31640fc710 msgr2=0x7f31640feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 -- 172.21.15.79:0/3507803121 shutdown_connections 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 -- 172.21.15.79:0/3507803121 wait complete. 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 Processor -- start 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.689+0000 7f3169df1700 1 -- start start 2024-04-22T06:43:53.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3169df1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3164101cc0 0x7f316419ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3169df1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 0x7f316419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3169df1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3164102f80 0x7f31641a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3169df1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f316410d4e0 con 0x7f3164102660 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3169df1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f316410d360 con 0x7f3164101cc0 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3169df1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f316410d660 con 0x7f3164102f80 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 0x7f316419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 0x7f316419b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43294/0 (socket says 172.21.15.79:43294) 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 -- 172.21.15.79:0/275499779 learned_addr learned my addr 172.21.15.79:0/275499779 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3163fff700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3164102f80 0x7f31641a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f31637fe700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3164101cc0 0x7f316419ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 -- 172.21.15.79:0/275499779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3164102f80 msgr2=0x7f31641a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3164102f80 0x7f31641a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 -- 172.21.15.79:0/275499779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3164101cc0 msgr2=0x7f316419ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3164101cc0 0x7f316419ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 -- 172.21.15.79:0/275499779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31641a2ae0 con 0x7f3164102660 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.690+0000 7f3162ffd700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 0x7f316419b3c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f315800d980 tx=0x7f315800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.691+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158016070 con 0x7f3164102660 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.691+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31641a2dd0 con 0x7f3164102660 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.691+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f315800de70 con 0x7f3164102660 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.691+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31641a32d0 con 0x7f3164102660 2024-04-22T06:43:53.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.691+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315800f3c0 con 0x7f3164102660 2024-04-22T06:43:53.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.692+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3164069730 con 0x7f3164102660 2024-04-22T06:43:53.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.693+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3158019070 con 0x7f3164102660 2024-04-22T06:43:53.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.693+0000 7f3160ff9700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f314c0a3cc0 0x7f314c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.694+0000 7f31637fe700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f314c0a3cc0 0x7f314c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.694+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f31580d02b0 con 0x7f3164102660 2024-04-22T06:43:53.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.694+0000 7f31637fe700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f314c0a3cc0 0x7f314c0a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f315400b580 tx=0x7f3154005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:53.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.697+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f315809cca0 con 0x7f3164102660 2024-04-22T06:43:53.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.920+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f31640fe8d0 con 0x7f314c0a3cc0 2024-04-22T06:43:53.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.921+0000 7f3160ff9700 1 -- 172.21.15.79:0/275499779 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f31640fe8d0 con 0x7f314c0a3cc0 2024-04-22T06:43:53.919 INFO:teuthology.orchestra.run.smithi079.stdout:Backend: cephadm 2024-04-22T06:43:53.919 INFO:teuthology.orchestra.run.smithi079.stdout:Available: Yes 2024-04-22T06:43:53.919 INFO:teuthology.orchestra.run.smithi079.stdout:Paused: No 2024-04-22T06:43:53.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f314c0a3cc0 msgr2=0x7f314c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f314c0a3cc0 0x7f314c0a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f315400b580 tx=0x7f3154005fb0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 msgr2=0x7f316419b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 0x7f316419b3c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f315800d980 tx=0x7f315800dc90 comp rx=0 tx=0).stop 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 shutdown_connections 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3164102f80 0x7f31641a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3164101cc0 0x7f316419ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f314c0a3cc0 0x7f314c0a6180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 --2- 172.21.15.79:0/275499779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3164102660 0x7f316419b3c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 >> 172.21.15.79:0/275499779 conn(0x7f31640fc710 msgr2=0x7f31640fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 shutdown_connections 2024-04-22T06:43:53.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:53.923+0000 7f3169df1700 1 -- 172.21.15.79:0/275499779 wait complete. 2024-04-22T06:43:53.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:53 smithi079 ceph-mon[27946]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:53 smithi193 ceph-mon[31815]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:54.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:53 smithi190 ceph-mon[31421]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:54.743 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T06:43:54.980 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:56.123 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:55 smithi079 ceph-mon[27946]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:43:56.123 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:55 smithi079 ceph-mon[27946]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:43:56.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:55 smithi193 ceph-mon[31815]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:43:56.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:55 smithi193 ceph-mon[31815]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:43:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:55 smithi190 ceph-mon[31421]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:43:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:55 smithi190 ceph-mon[31421]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:43:56.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 -- 172.21.15.79:0/158895755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102660 msgr2=0x7fd890109f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 --2- 172.21.15.79:0/158895755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102660 0x7fd890109f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd88800b1d0 tx=0x7fd88800b4e0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 -- 172.21.15.79:0/158895755 shutdown_connections 2024-04-22T06:43:56.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 --2- 172.21.15.79:0/158895755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102660 0x7fd890109f20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 --2- 172.21.15.79:0/158895755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102f80 0x7fd8901033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 --2- 172.21.15.79:0/158895755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd890101cc0 0x7fd890102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 -- 172.21.15.79:0/158895755 >> 172.21.15.79:0/158895755 conn(0x7fd8900fc710 msgr2=0x7fd8900feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 -- 172.21.15.79:0/158895755 shutdown_connections 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.688+0000 7fd89804b700 1 -- 172.21.15.79:0/158895755 wait complete. 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 Processor -- start 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 -- start start 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd890101cc0 0x7fd890196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102660 0x7fd890196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 0x7fd89019df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd89010c3f0 con 0x7fd890102f80 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd89010c270 con 0x7fd890102660 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd89804b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd89010c570 con 0x7fd890101cc0 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd8965e8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 0x7fd89019df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd8965e8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 0x7fd89019df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43316/0 (socket says 172.21.15.79:43316) 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd8965e8700 1 -- 172.21.15.79:0/3788235651 learned_addr learned my addr 172.21.15.79:0/3788235651 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd895de7700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd890101cc0 0x7fd890196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.689+0000 7fd8955e6700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102660 0x7fd890196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8965e8700 1 -- 172.21.15.79:0/3788235651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd890101cc0 msgr2=0x7fd890196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8965e8700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd890101cc0 0x7fd890196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8965e8700 1 -- 172.21.15.79:0/3788235651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102660 msgr2=0x7fd890196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8965e8700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102660 0x7fd890196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8965e8700 1 -- 172.21.15.79:0/3788235651 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89019e5e0 con 0x7fd890102f80 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8955e6700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102660 0x7fd890196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd8965e8700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 0x7fd89019df40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd888008e70 tx=0x7fd888008ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88800bdc0 con 0x7fd890102f80 2024-04-22T06:43:56.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89019e870 con 0x7fd890102f80 2024-04-22T06:43:56.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd888003c10 con 0x7fd890102f80 2024-04-22T06:43:56.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8880109d0 con 0x7fd890102f80 2024-04-22T06:43:56.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.690+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd89019ec60 con 0x7fd890102f80 2024-04-22T06:43:56.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.691+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd890069730 con 0x7fd890102f80 2024-04-22T06:43:56.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.692+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd888028060 con 0x7fd890102f80 2024-04-22T06:43:56.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.693+0000 7fd886ffd700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd87c0a3c40 0x7fd87c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.693+0000 7fd895de7700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd87c0a3c40 0x7fd87c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.693+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd8880d4d10 con 0x7fd890102f80 2024-04-22T06:43:56.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.693+0000 7fd895de7700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd87c0a3c40 0x7fd87c0a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd89006af10 tx=0x7fd88000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.697+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8880a1790 con 0x7fd890102f80 2024-04-22T06:43:56.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.906+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fd890062e80 con 0x7fd87c0a3c40 2024-04-22T06:43:56.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.920+0000 7fd886ffd700 1 -- 172.21.15.79:0/3788235651 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fd890062e80 con 0x7fd87c0a3c40 2024-04-22T06:43:56.918 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.a smithi079 running (101s) 18s ago 101s 16.3M - 19.0.0-3156-g86f7587a 1b5e03f260bd 4dd988509095 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.b smithi190 running (98s) 32s ago 98s 28.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd bdeb2a276527 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.c smithi193 running (113s) 32s ago 112s 17.7M - 19.0.0-3156-g86f7587a 1b5e03f260bd 98f45dbf9d45 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.d smithi079 running (2m) 18s ago 2m 15.4M - 19.0.0-3156-g86f7587a 1b5e03f260bd 833ef07e27f1 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.e smithi190 running (118s) 32s ago 118s 16.9M - 19.0.0-3156-g86f7587a 1b5e03f260bd 5330aa7da9cc 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.f smithi193 running (116s) 32s ago 115s 18.3M - 19.0.0-3156-g86f7587a 1b5e03f260bd 430ad37b311b 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.g smithi079 running (104s) 18s ago 104s 16.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 23d4fa0bb537 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.h smithi190 running (110s) 32s ago 109s 17.6M - 19.0.0-3156-g86f7587a 1b5e03f260bd 443444e91981 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.i smithi193 running (107s) 32s ago 106s 16.5M - 19.0.0-3156-g86f7587a 1b5e03f260bd 3bd07a4cc9de 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.j smithi079 running (2m) 18s ago 2m 13.8M - 19.0.0-3156-g86f7587a 1b5e03f260bd 291f0baadb8b 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.k smithi190 running (2m) 32s ago 2m 15.4M - 19.0.0-3156-g86f7587a 1b5e03f260bd 77a8771c3b90 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mds.l smithi193 running (2m) 32s ago 2m 14.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 54adb40b1d86 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.x smithi079 *:9283,8765 running (9m) 18s ago 9m 517M - 19.0.0-3156-g86f7587a 1b5e03f260bd 56d7b282d42a 2024-04-22T06:43:56.919 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.y smithi190 *:8443,8765 running (7m) 32s ago 7m 453M - 19.0.0-3156-g86f7587a 1b5e03f260bd e4afe626c571 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.z smithi193 *:8443,8765 running (7m) 32s ago 7m 451M - 19.0.0-3156-g86f7587a 1b5e03f260bd e63eb2aa2768 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:mon.a smithi079 running (9m) 18s ago 9m 53.5M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 5b4c5f2f0f6c 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:mon.b smithi190 running (7m) 32s ago 7m 40.1M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd dedf43993a9b 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:mon.c smithi193 running (7m) 32s ago 7m 39.3M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 892e02a5d47b 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (7m) 18s ago 7m 56.3M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 7149cb2fcc5d 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi190 running (6m) 32s ago 6m 55.6M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 5e3162d4a123 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi193 running (6m) 32s ago 6m 58.4M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd c4eda6159ece 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi079 running (5m) 18s ago 5m 53.1M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd f3a3f6bd700a 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi190 running (5m) 32s ago 5m 52.2M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 705880ad4b02 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi193 running (5m) 32s ago 5m 43.5M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 8029dde9e57b 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.6 smithi079 running (4m) 18s ago 4m 50.9M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd a3c201c66633 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.7 smithi190 running (4m) 32s ago 4m 56.8M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 73760e049025 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.8 smithi193 running (3m) 32s ago 3m 53.3M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd a56c72d62d32 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.9 smithi079 running (3m) 18s ago 3m 53.9M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 52583edaeda5 2024-04-22T06:43:56.920 INFO:teuthology.orchestra.run.smithi079.stdout:osd.10 smithi190 running (2m) 32s ago 2m 42.4M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 6cee66cf8276 2024-04-22T06:43:56.921 INFO:teuthology.orchestra.run.smithi079.stdout:osd.11 smithi193 running (2m) 32s ago 2m 55.3M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd b4320fd88cd4 2024-04-22T06:43:56.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd87c0a3c40 msgr2=0x7fd87c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd87c0a3c40 0x7fd87c0a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd89006af10 tx=0x7fd88000d040 comp rx=0 tx=0).stop 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 msgr2=0x7fd89019df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 0x7fd89019df40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd888008e70 tx=0x7fd888008ea0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 shutdown_connections 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd87c0a3c40 0x7fd87c0a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd890102f80 0x7fd89019df40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd890101cc0 0x7fd890196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 --2- 172.21.15.79:0/3788235651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd890102660 0x7fd890196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 >> 172.21.15.79:0/3788235651 conn(0x7fd8900fc710 msgr2=0x7fd890105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 shutdown_connections 2024-04-22T06:43:56.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:56.922+0000 7fd89804b700 1 -- 172.21.15.79:0/3788235651 wait complete. 2024-04-22T06:43:57.718 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T06:43:57.958 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:57.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:57 smithi079 ceph-mon[27946]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:43:58.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:57 smithi193 ceph-mon[31815]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:43:58.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:57 smithi190 ceph-mon[31421]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:43:59.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:58 smithi193 ceph-mon[31815]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:43:59.103 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:58 smithi079 ceph-mon[27946]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:43:59.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:58 smithi190 ceph-mon[31421]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:43:59.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 -- 172.21.15.79:0/705265259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 msgr2=0x7f96b0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:59.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/705265259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0101dc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f96a00099c0 tx=0x7f96a0009cd0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 -- 172.21.15.79:0/705265259 shutdown_connections 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/705265259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96b0103540 0x7f96b0109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/705265259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0101dc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/705265259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b0102ba0 0x7f96b0103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 -- 172.21.15.79:0/705265259 >> 172.21.15.79:0/705265259 conn(0x7f96b00fcf40 msgr2=0x7f96b00ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 -- 172.21.15.79:0/705265259 shutdown_connections 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.691+0000 7f96b5e7b700 1 -- 172.21.15.79:0/705265259 wait complete. 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 Processor -- start 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 -- start start 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b0102ba0 0x7f96b0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96b0103540 0x7f96b019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:59.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010d430 con 0x7f96b01019a0 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010d2b0 con 0x7f96b0102ba0 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96b5e7b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f96b010d5b0 con 0x7f96b0103540 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43326/0 (socket says 172.21.15.79:43326) 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 -- 172.21.15.79:0/1607279665 learned_addr learned my addr 172.21.15.79:0/1607279665 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96aeffd700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b0102ba0 0x7f96b0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96affff700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96b0103540 0x7f96b019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 -- 172.21.15.79:0/1607279665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96b0103540 msgr2=0x7f96b019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96b0103540 0x7f96b019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 -- 172.21.15.79:0/1607279665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b0102ba0 msgr2=0x7f96b0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b0102ba0 0x7f96b0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.692+0000 7f96af7fe700 1 -- 172.21.15.79:0/1607279665 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b019f780 con 0x7f96b01019a0 2024-04-22T06:43:59.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.693+0000 7f96af7fe700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0197b20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96a000b580 tx=0x7f96a0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:59.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.693+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0029070 con 0x7f96b01019a0 2024-04-22T06:43:59.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.693+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b006a720 con 0x7f96b01019a0 2024-04-22T06:43:59.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.693+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96a0019cc0 con 0x7f96b01019a0 2024-04-22T06:43:59.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.693+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00041e0 con 0x7f96b01019a0 2024-04-22T06:43:59.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.693+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b006ac00 con 0x7f96b01019a0 2024-04-22T06:43:59.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.694+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b010dbf0 con 0x7f96b01019a0 2024-04-22T06:43:59.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.695+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f96a002c070 con 0x7f96b01019a0 2024-04-22T06:43:59.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.696+0000 7f96acff9700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f96980a3c40 0x7f96980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:59.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.696+0000 7f96aeffd700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f96980a3c40 0x7f96980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:59.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.696+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f96a00d34c0 con 0x7f96b01019a0 2024-04-22T06:43:59.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.696+0000 7f96aeffd700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f96980a3c40 0x7f96980a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005950 tx=0x7f96a400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:59.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.699+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96a0013070 con 0x7f96b01019a0 2024-04-22T06:43:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.911+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f96b0108da0 con 0x7f96980a3c40 2024-04-22T06:43:59.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.917+0000 7f96acff9700 1 -- 172.21.15.79:0/1607279665 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f96b0108da0 con 0x7f96980a3c40 2024-04-22T06:43:59.915 INFO:teuthology.orchestra.run.smithi079.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T06:43:59.916 INFO:teuthology.orchestra.run.smithi079.stdout:mds.all 12/12 35s ago 2m smithi079=a;smithi079=d;smithi079=g;smithi079=j;smithi190=b;smithi190=e;smithi190=h;smithi190=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-22T06:43:59.916 INFO:teuthology.orchestra.run.smithi079.stdout:mgr 3/3 35s ago 7m smithi079=x;smithi190=y;smithi193=z;count:3 2024-04-22T06:43:59.916 INFO:teuthology.orchestra.run.smithi079.stdout:mon 3/3 35s ago 8m smithi079:172.21.15.79=a;smithi190:172.21.15.190=b;smithi193:172.21.15.193=c;count:3 2024-04-22T06:43:59.916 INFO:teuthology.orchestra.run.smithi079.stdout:osd 12 35s ago - 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f96980a3c40 msgr2=0x7f96980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f96980a3c40 0x7f96980a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005950 tx=0x7f96a400a300 comp rx=0 tx=0).stop 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 msgr2=0x7f96b0197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0197b20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96a000b580 tx=0x7f96a0009580 comp rx=0 tx=0).stop 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 shutdown_connections 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96b0103540 0x7f96b019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f96980a3c40 0x7f96980a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96b01019a0 0x7f96b0197b20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 --2- 172.21.15.79:0/1607279665 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96b0102ba0 0x7f96b0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 >> 172.21.15.79:0/1607279665 conn(0x7f96b00fcf40 msgr2=0x7f96b0105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 shutdown_connections 2024-04-22T06:43:59.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:43:59.919+0000 7f96b5e7b700 1 -- 172.21.15.79:0/1607279665 wait complete. 2024-04-22T06:44:00.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:43:59 smithi190 ceph-mon[31421]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:44:00.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:43:59 smithi193 ceph-mon[31815]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:44:00.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:43:59 smithi079 ceph-mon[27946]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:44:00.617 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T06:44:00.856 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:01.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:01 smithi079 ceph-mon[27946]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:01.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:01 smithi079 ceph-mon[27946]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:44:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:01 smithi193 ceph-mon[31815]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:02.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:01 smithi193 ceph-mon[31815]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:44:02.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:01 smithi190 ceph-mon[31421]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:02.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:01 smithi190 ceph-mon[31421]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 -- 172.21.15.79:0/3767278983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e4104560 msgr2=0x7f08e410ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3767278983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e4104560 0x7f08e410ae10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00b0d0 tx=0x7f08dc00b3e0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 -- 172.21.15.79:0/3767278983 shutdown_connections 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3767278983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e4104560 0x7f08e410ae10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3767278983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e41029c0 0x7f08e4102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3767278983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e4103bc0 0x7f08e4104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 -- 172.21.15.79:0/3767278983 >> 172.21.15.79:0/3767278983 conn(0x7f08e40fdf80 msgr2=0x7f08e41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:02.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.551+0000 7f08e3fff700 1 -- 172.21.15.79:0/3767278983 shutdown_connections 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.552+0000 7f08e3fff700 1 -- 172.21.15.79:0/3767278983 wait complete. 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.552+0000 7f08e3fff700 1 Processor -- start 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.552+0000 7f08e3fff700 1 -- start start 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.552+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 0x7f08e40767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.552+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e4103bc0 0x7f08e4074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.552+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e4104560 0x7f08e4075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e3fff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f08e410e4b0 con 0x7f08e41029c0 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e3fff700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f08e410e330 con 0x7f08e4103bc0 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e3fff700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08e410e630 con 0x7f08e4104560 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 0x7f08e40767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 0x7f08e40767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53006/0 (socket says 172.21.15.79:53006) 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 -- 172.21.15.79:0/3181445793 learned_addr learned my addr 172.21.15.79:0/3181445793 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e27fc700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e4103bc0 0x7f08e4074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e37fe700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e4104560 0x7f08e4075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 -- 172.21.15.79:0/3181445793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e4104560 msgr2=0x7f08e4075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e4104560 0x7f08e4075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 -- 172.21.15.79:0/3181445793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e4103bc0 msgr2=0x7f08e4074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e4103bc0 0x7f08e4074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 -- 172.21.15.79:0/3181445793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e4075c20 con 0x7f08e41029c0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e2ffd700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 0x7f08e40767e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c970 tx=0x7f08d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004220 con 0x7f08e41029c0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d40043c0 con 0x7f08e41029c0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e41a2c20 con 0x7f08e41029c0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.553+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4003c20 con 0x7f08e41029c0 2024-04-22T06:44:02.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.554+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e41a3110 con 0x7f08e41029c0 2024-04-22T06:44:02.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.555+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e4069730 con 0x7f08e41029c0 2024-04-22T06:44:02.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.555+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f08d40076d0 con 0x7f08e41029c0 2024-04-22T06:44:02.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.556+0000 7f08e89fd700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f08cc0a3cc0 0x7f08cc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.556+0000 7f08e27fc700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f08cc0a3cc0 0x7f08cc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.556+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f08d40cfe30 con 0x7f08e41029c0 2024-04-22T06:44:02.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.557+0000 7f08e27fc700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f08cc0a3cc0 0x7f08cc0a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009a10 tx=0x7f08d8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:02.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.561+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08d4014030 con 0x7f08e41029c0 2024-04-22T06:44:02.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.766+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f08e4062e80 con 0x7f08cc0a3cc0 2024-04-22T06:44:02.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.768+0000 7f08e89fd700 1 -- 172.21.15.79:0/3181445793 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f08e4062e80 con 0x7f08cc0a3cc0 2024-04-22T06:44:02.766 INFO:teuthology.orchestra.run.smithi079.stdout:HOST ADDR LABELS STATUS 2024-04-22T06:44:02.766 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 172.21.15.79 2024-04-22T06:44:02.766 INFO:teuthology.orchestra.run.smithi079.stdout:smithi190 172.21.15.190 2024-04-22T06:44:02.766 INFO:teuthology.orchestra.run.smithi079.stdout:smithi193 172.21.15.193 2024-04-22T06:44:02.767 INFO:teuthology.orchestra.run.smithi079.stdout:3 hosts in cluster 2024-04-22T06:44:02.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f08cc0a3cc0 msgr2=0x7f08cc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f08cc0a3cc0 0x7f08cc0a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009a10 tx=0x7f08d8009190 comp rx=0 tx=0).stop 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 msgr2=0x7f08e40767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 0x7f08e40767e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c970 tx=0x7f08d400cc80 comp rx=0 tx=0).stop 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 shutdown_connections 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e4104560 0x7f08e4075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f08cc0a3cc0 0x7f08cc0a6180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f08e41029c0 0x7f08e40767e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 --2- 172.21.15.79:0/3181445793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08e4103bc0 0x7f08e4074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 >> 172.21.15.79:0/3181445793 conn(0x7f08e40fdf80 msgr2=0x7f08e4106ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 shutdown_connections 2024-04-22T06:44:02.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:02.770+0000 7f08e3fff700 1 -- 172.21.15.79:0/3181445793 wait complete. 2024-04-22T06:44:03.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T06:44:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:04.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:03 smithi079 ceph-mon[27946]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:44:04.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:03 smithi079 ceph-mon[27946]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:04.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:03 smithi193 ceph-mon[31815]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:44:04.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:03 smithi193 ceph-mon[31815]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:04.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:03 smithi190 ceph-mon[31421]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:44:04.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:03 smithi190 ceph-mon[31421]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:05.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.531+0000 7fc85ce7d700 1 -- 172.21.15.79:0/3255632696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 msgr2=0x7fc858103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.531+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/3255632696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103010 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0099c0 tx=0x7fc84c009cd0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 -- 172.21.15.79:0/3255632696 shutdown_connections 2024-04-22T06:44:05.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/3255632696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc858103550 0x7fc858109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/3255632696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 0x7fc858101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/3255632696 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103010 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 -- 172.21.15.79:0/3255632696 >> 172.21.15.79:0/3255632696 conn(0x7fc8580fcf70 msgr2=0x7fc8580ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:05.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 -- 172.21.15.79:0/3255632696 shutdown_connections 2024-04-22T06:44:05.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 -- 172.21.15.79:0/3255632696 wait complete. 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 Processor -- start 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.532+0000 7fc85ce7d700 1 -- start start 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 0x7fc858105940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc858103550 0x7fc8581044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc85810d540 con 0x7fc858102bb0 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc85810d3c0 con 0x7fc8581019b0 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc85810d6c0 con 0x7fc858103550 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53030/0 (socket says 172.21.15.79:53030) 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 -- 172.21.15.79:0/1928668210 learned_addr learned my addr 172.21.15.79:0/1928668210 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:05.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 -- 172.21.15.79:0/1928668210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc858103550 msgr2=0x7fc8581044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85659c700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 0x7fc858105940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc858103550 0x7fc8581044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 -- 172.21.15.79:0/1928668210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 msgr2=0x7fc858105940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 0x7fc858105940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 -- 172.21.15.79:0/1928668210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc858104d80 con 0x7fc858102bb0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85659c700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 0x7fc858105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc855d9b700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103f90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b580 tx=0x7fc84c003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c027070 con 0x7fc858102bb0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8581a2eb0 con 0x7fc858102bb0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc84c022470 con 0x7fc858102bb0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c00f6e0 con 0x7fc858102bb0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.533+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8581a3380 con 0x7fc858102bb0 2024-04-22T06:44:05.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.534+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8581a35e0 con 0x7fc858102bb0 2024-04-22T06:44:05.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.536+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc84c02a070 con 0x7fc858102bb0 2024-04-22T06:44:05.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.536+0000 7fc8477fe700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8400a3c40 0x7fc8400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.537+0000 7fc85659c700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8400a3c40 0x7fc8400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.537+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc84c0d12b0 con 0x7fc858102bb0 2024-04-22T06:44:05.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.537+0000 7fc85659c700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8400a3c40 0x7fc8400a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc858102a10 tx=0x7fc848008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:05.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.540+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc84c013070 con 0x7fc858102bb0 2024-04-22T06:44:05.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.748+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fc85806b910 con 0x7fc8400a3c40 2024-04-22T06:44:05.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.750+0000 7fc8477fe700 1 -- 172.21.15.79:0/1928668210 <== mgr.14152 v2:172.21.15.79:6800/3007546422 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fc85806b910 con 0x7fc8400a3c40 2024-04-22T06:44:05.749 INFO:teuthology.orchestra.run.smithi079.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T06:44:05.749 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013G400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T06:44:05.749 INFO:teuthology.orchestra.run.smithi079.stdout:smithi190 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT63210012400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T06:44:05.749 INFO:teuthology.orchestra.run.smithi079.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8400a3c40 msgr2=0x7fc8400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8400a3c40 0x7fc8400a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc858102a10 tx=0x7fc848008040 comp rx=0 tx=0).stop 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 msgr2=0x7fc858103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103f90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b580 tx=0x7fc84c003960 comp rx=0 tx=0).stop 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 shutdown_connections 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc858103550 0x7fc8581044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8581019b0 0x7fc858105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc8400a3c40 0x7fc8400a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 --2- 172.21.15.79:0/1928668210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc858102bb0 0x7fc858103f90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 >> 172.21.15.79:0/1928668210 conn(0x7fc8580fcf70 msgr2=0x7fc85806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:05.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 shutdown_connections 2024-04-22T06:44:05.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:05.752+0000 7fc85ce7d700 1 -- 172.21.15.79:0/1928668210 wait complete. 2024-04-22T06:44:05.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:05 smithi079 ceph-mon[27946]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:44:06.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:05 smithi193 ceph-mon[31815]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:44:06.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:05 smithi190 ceph-mon[31421]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:44:06.479 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T06:44:06.488 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:44:06.488 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T06:44:06.726 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:07.104 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:06 smithi079 ceph-mon[27946]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:06 smithi193 ceph-mon[31815]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:07.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:06 smithi190 ceph-mon[31421]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:08.026 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:07 smithi079 ceph-mon[27946]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:44:08.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:07 smithi193 ceph-mon[31815]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:44:08.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:07 smithi190 ceph-mon[31421]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 -- 172.21.15.79:0/1337684193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101170 msgr2=0x7f6154101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 --2- 172.21.15.79:0/1337684193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101170 0x7f6154101590 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f61440099c0 tx=0x7f6144009cd0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 -- 172.21.15.79:0/1337684193 shutdown_connections 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 --2- 172.21.15.79:0/1337684193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154102d10 0x7f61541095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 --2- 172.21.15.79:0/1337684193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101170 0x7f6154101590 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 --2- 172.21.15.79:0/1337684193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154102370 0x7f61541027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 -- 172.21.15.79:0/1337684193 >> 172.21.15.79:0/1337684193 conn(0x7f61540fc710 msgr2=0x7f61540feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 -- 172.21.15.79:0/1337684193 shutdown_connections 2024-04-22T06:44:08.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.583+0000 7f615a958700 1 -- 172.21.15.79:0/1337684193 wait complete. 2024-04-22T06:44:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 Processor -- start 2024-04-22T06:44:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 -- start start 2024-04-22T06:44:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154101170 0x7f61541050b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154102370 0x7f6154103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 0x7f6154103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f615410cc70 con 0x7f6154102d10 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f615410caf0 con 0x7f6154101170 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f615a958700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f615410cdf0 con 0x7f6154102370 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f6158ef5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 0x7f6154103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f6158ef5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 0x7f6154103c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53056/0 (socket says 172.21.15.79:53056) 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f6158ef5700 1 -- 172.21.15.79:0/1817888107 learned_addr learned my addr 172.21.15.79:0/1817888107 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.584+0000 7f6153fff700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154101170 0x7f61541050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f61537fe700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154102370 0x7f6154103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f6158ef5700 1 -- 172.21.15.79:0/1817888107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154102370 msgr2=0x7f6154103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f6158ef5700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154102370 0x7f6154103750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f6158ef5700 1 -- 172.21.15.79:0/1817888107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154101170 msgr2=0x7f61541050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f6158ef5700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154101170 0x7f61541050b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f6158ef5700 1 -- 172.21.15.79:0/1817888107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6154104560 con 0x7f6154102d10 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f6158ef5700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 0x7f6154103c90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f614c00ada0 tx=0x7f614c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:08.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c017020 con 0x7f6154102d10 2024-04-22T06:44:08.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61541a2e10 con 0x7f6154102d10 2024-04-22T06:44:08.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614c00ce30 con 0x7f6154102d10 2024-04-22T06:44:08.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c005380 con 0x7f6154102d10 2024-04-22T06:44:08.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.585+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61541a3210 con 0x7f6154102d10 2024-04-22T06:44:08.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.586+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6154069730 con 0x7f6154102d10 2024-04-22T06:44:08.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.588+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f614c00c980 con 0x7f6154102d10 2024-04-22T06:44:08.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.589+0000 7f61517fa700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f613c0a3c40 0x7f613c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.589+0000 7f6153fff700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f613c0a3c40 0x7f613c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:08.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.589+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f614c0cfe40 con 0x7f6154102d10 2024-04-22T06:44:08.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.589+0000 7f6153fff700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f613c0a3c40 0x7f613c0a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f61440038c0 tx=0x7f614400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:08.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.592+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f614c021060 con 0x7f6154102d10 2024-04-22T06:44:08.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.839+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f61541a37a0 con 0x7f6154102d10 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.840+0000 7f61517fa700 1 -- 172.21.15.79:0/1817888107 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3054 (secure 0 0 0) 0x7f614c09c8c0 con 0x7f6154102d10 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:e21 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 21 2024-04-22T06:44:08.839 INFO:teuthology.orchestra.run.smithi079.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-22T06:42:24.011937+0000 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-04-22T06:42:28.714055+0000 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 300 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:max_xattr_size 65536 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 0 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 3 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:in 0,1,2 2024-04-22T06:44:08.840 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=24467,1=24440,2=24436} 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data disabled 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:bal_rank_mask -1 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 0 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:qdb_cluster leader: 24467 members: 24436,24440,24467 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.b{0:24467} state up:active seq 3 addr [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.c{0:24424} state up:standby-replay seq 1 addr [v2:172.21.15.193:6836/678626858,v1:172.21.15.193:6837/678626858] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.h{1:24440} state up:active seq 6 addr [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.e{1:24413} state up:standby-replay seq 1 addr [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.841 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.i{2:24436} state up:active seq 6 addr [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.f{2:24406} state up:standby-replay seq 1 addr [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:Standby daemons: 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.j{-1:14523} state up:standby seq 1 addr [v2:172.21.15.79:6834/2324822974,v1:172.21.15.79:6835/2324822974] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.d{-1:14535} state up:standby seq 1 addr [v2:172.21.15.79:6836/2658402003,v1:172.21.15.79:6837/2658402003] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.g{-1:14604} state up:standby seq 1 addr [v2:172.21.15.79:6838/1490349330,v1:172.21.15.79:6839/1490349330] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.a{-1:14616} state up:standby seq 1 addr [v2:172.21.15.79:6840/3134754161,v1:172.21.15.79:6841/3134754161] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.l{-1:24385} state up:standby seq 1 addr [v2:172.21.15.193:6832/1714090730,v1:172.21.15.193:6833/1714090730] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.842 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.k{-1:24389} state up:standby seq 1 addr [v2:172.21.15.190:6832/1510033395,v1:172.21.15.190:6833/1510033395] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f613c0a3c40 msgr2=0x7f613c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f613c0a3c40 0x7f613c0a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f61440038c0 tx=0x7f614400b540 comp rx=0 tx=0).stop 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 msgr2=0x7f6154103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 0x7f6154103c90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f614c00ada0 tx=0x7f614c00c330 comp rx=0 tx=0).stop 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 shutdown_connections 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f613c0a3c40 0x7f613c0a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154102d10 0x7f6154103c90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154101170 0x7f61541050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 --2- 172.21.15.79:0/1817888107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154102370 0x7f6154103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 >> 172.21.15.79:0/1817888107 conn(0x7f61540fc710 msgr2=0x7f61540fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:08.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 shutdown_connections 2024-04-22T06:44:08.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:08.842+0000 7f615a958700 1 -- 172.21.15.79:0/1817888107 wait complete. 2024-04-22T06:44:08.844 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:44:09.635 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-22T06:44:09.887 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:09.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:09 smithi079 ceph-mon[27946]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:09.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:09 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1817888107' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:44:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:09 smithi193 ceph-mon[31815]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:09 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1817888107' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:44:10.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:09 smithi190 ceph-mon[31421]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:10.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:09 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1817888107' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:44:11.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 -- 172.21.15.79:0/2166354818 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938104880 msgr2=0x7f793810b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 --2- 172.21.15.79:0/2166354818 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938104880 0x7f793810b130 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f793000b0d0 tx=0x7f793000b3e0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 -- 172.21.15.79:0/2166354818 shutdown_connections 2024-04-22T06:44:11.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 --2- 172.21.15.79:0/2166354818 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938104880 0x7f793810b130 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 --2- 172.21.15.79:0/2166354818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938100e60 0x7f7938101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 --2- 172.21.15.79:0/2166354818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7938103ee0 0x7f7938104340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 -- 172.21.15.79:0/2166354818 >> 172.21.15.79:0/2166354818 conn(0x7f79380fc7a0 msgr2=0x7f79380febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 -- 172.21.15.79:0/2166354818 shutdown_connections 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.701+0000 7f793d2ca700 1 -- 172.21.15.79:0/2166354818 wait complete. 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 Processor -- start 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 -- start start 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 0x7f793819aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7938103ee0 0x7f793819b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938104880 0x7f79381a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f793810e710 con 0x7f7938100e60 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f793810e590 con 0x7f7938104880 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f793d2ca700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f793810e890 con 0x7f7938103ee0 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f7936ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 0x7f793819aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f7936ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 0x7f793819aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53078/0 (socket says 172.21.15.79:53078) 2024-04-22T06:44:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f7936ffd700 1 -- 172.21.15.79:0/3672241359 learned_addr learned my addr 172.21.15.79:0/3672241359 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.702+0000 7f79377fe700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938104880 0x7f79381a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f79367fc700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7938103ee0 0x7f793819b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f7936ffd700 1 -- 172.21.15.79:0/3672241359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7938103ee0 msgr2=0x7f793819b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f7936ffd700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7938103ee0 0x7f793819b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f7936ffd700 1 -- 172.21.15.79:0/3672241359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938104880 msgr2=0x7f79381a2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f7936ffd700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938104880 0x7f79381a2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f7936ffd700 1 -- 172.21.15.79:0/3672241359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79381a2b30 con 0x7f7938100e60 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f79377fe700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938104880 0x7f79381a2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f7936ffd700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 0x7f793819aed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f792800b530 tx=0x7f792800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928015020 con 0x7f7938100e60 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7928011ce0 con 0x7f7938100e60 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79381a2e20 con 0x7f7938100e60 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928010d60 con 0x7f7938100e60 2024-04-22T06:44:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.703+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79381a32f0 con 0x7f7938100e60 2024-04-22T06:44:11.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.704+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79381a35e0 con 0x7f7938100e60 2024-04-22T06:44:11.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.705+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7928018020 con 0x7f7938100e60 2024-04-22T06:44:11.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.706+0000 7f791ffff700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79200a3c40 0x7f79200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.706+0000 7f79367fc700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79200a3c40 0x7f79200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.706+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f79280cfbc0 con 0x7f7938100e60 2024-04-22T06:44:11.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.706+0000 7f79367fc700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79200a3c40 0x7f79200a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f792c006fd0 tx=0x7f792c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:11.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.710+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f792801f020 con 0x7f7938100e60 2024-04-22T06:44:11.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.911+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f79381a3e20 con 0x7f7938100e60 2024-04-22T06:44:11.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.912+0000 7f791ffff700 1 -- 172.21.15.79:0/3672241359 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5281 (secure 0 0 0) 0x7f792809c5b0 con 0x7f7938100e60 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 87 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:fsid 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-04-22T06:34:23.219533+0000 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-04-22T06:43:17.795510+0000 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:crush_version 26 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:full_ratio 0.95 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:backfillfull_ratio 0.9 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:nearfull_ratio 0.85 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:require_min_compat_client luminous 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:min_compat_client luminous 2024-04-22T06:44:11.911 INFO:teuthology.orchestra.run.smithi079.stdout:require_osd_release squid 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:stretch_mode_enabled false 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:max_osd 12 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6802/2633795155,v1:172.21.15.79:6803/2633795155] [v2:172.21.15.79:6804/2633795155,v1:172.21.15.79:6805/2633795155] exists,up 1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/237015036,v1:172.21.15.190:6801/237015036] [v2:172.21.15.190:6802/237015036,v1:172.21.15.190:6803/237015036] exists,up 072857b1-5aa7-4510-b9e9-5b6cc192458c 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/4086025232,v1:172.21.15.193:6801/4086025232] [v2:172.21.15.193:6802/4086025232,v1:172.21.15.193:6803/4086025232] exists,up e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6810/433108201,v1:172.21.15.79:6811/433108201] [v2:172.21.15.79:6812/433108201,v1:172.21.15.79:6813/433108201] exists,up 0398c3f2-3fd7-40da-bdd0-20c01352af3c 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/3437671425,v1:172.21.15.190:6809/3437671425] [v2:172.21.15.190:6810/3437671425,v1:172.21.15.190:6811/3437671425] exists,up 4b4fb272-de89-414a-b82c-95b8169e29a8 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/636159812,v1:172.21.15.193:6809/636159812] [v2:172.21.15.193:6810/636159812,v1:172.21.15.193:6811/636159812] exists,up d6cbf2d1-2a90-41c3-a524-243e7ed3e105 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.6 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6818/1077153926,v1:172.21.15.79:6819/1077153926] [v2:172.21.15.79:6820/1077153926,v1:172.21.15.79:6821/1077153926] exists,up 95e2a0de-8900-4bfc-afb5-ed608c8f174d 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.7 up in weight 1 up_from 51 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/2271031515,v1:172.21.15.190:6817/2271031515] [v2:172.21.15.190:6818/2271031515,v1:172.21.15.190:6819/2271031515] exists,up f57546c5-ce88-456a-b5c8-0a8ba8dcafae 2024-04-22T06:44:11.912 INFO:teuthology.orchestra.run.smithi079.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/3840715130,v1:172.21.15.193:6817/3840715130] [v2:172.21.15.193:6818/3840715130,v1:172.21.15.193:6819/3840715130] exists,up 8f6b199d-ad3e-4ac8-815c-4b30b3a7e335 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.79:6826/4293144131,v1:172.21.15.79:6827/4293144131] [v2:172.21.15.79:6828/4293144131,v1:172.21.15.79:6829/4293144131] exists,up 6a68e58f-14cc-4c45-a45b-5cd0505bff3f 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/2195513042,v1:172.21.15.190:6825/2195513042] [v2:172.21.15.190:6826/2195513042,v1:172.21.15.190:6827/2195513042] exists,up 7c7a37b4-3474-4e05-b976-61c5b99dbb98 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/3747742483,v1:172.21.15.193:6825/3747742483] [v2:172.21.15.193:6826/3747742483,v1:172.21.15.193:6827/3747742483] exists,up 343a8050-22fd-4408-a1d5-e058b623348d 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 2.a [2,11] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 2.b [8,11] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 3.0 [4,6] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 3.1d [5,9] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 3.20 [5,9] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 3.21 [4,8] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 3.28 [3,8] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:pg_upmap_items 3.36 [4,9] 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:0/1840338607 expires 2024-04-23T06:35:16.561501+0000 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:6801/3692925838 expires 2024-04-23T06:35:16.561501+0000 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:6800/3692925838 expires 2024-04-23T06:35:16.561501+0000 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:0/1891295292 expires 2024-04-23T06:34:50.796165+0000 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:6801/874344387 expires 2024-04-23T06:34:50.796165+0000 2024-04-22T06:44:11.913 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:0/4061268199 expires 2024-04-23T06:34:50.796165+0000 2024-04-22T06:44:11.914 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:0/3938306769 expires 2024-04-23T06:34:50.796165+0000 2024-04-22T06:44:11.914 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:0/2576810188 expires 2024-04-23T06:35:16.561501+0000 2024-04-22T06:44:11.914 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:6800/874344387 expires 2024-04-23T06:34:50.796165+0000 2024-04-22T06:44:11.914 INFO:teuthology.orchestra.run.smithi079.stdout:blocklist 172.21.15.79:0/66466683 expires 2024-04-23T06:35:16.561501+0000 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79200a3c40 msgr2=0x7f79200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79200a3c40 0x7f79200a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f792c006fd0 tx=0x7f792c008040 comp rx=0 tx=0).stop 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 msgr2=0x7f793819aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 0x7f793819aed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f792800b530 tx=0x7f792800b8f0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 shutdown_connections 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7938104880 0x7f79381a2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79200a3c40 0x7f79200a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7938100e60 0x7f793819aed0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 --2- 172.21.15.79:0/3672241359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7938103ee0 0x7f793819b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.914+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 >> 172.21.15.79:0/3672241359 conn(0x7f79380fc7a0 msgr2=0x7f7938101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.915+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 shutdown_connections 2024-04-22T06:44:11.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:11.915+0000 7f793d2ca700 1 -- 172.21.15.79:0/3672241359 wait complete. 2024-04-22T06:44:12.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:11 smithi193 ceph-mon[31815]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:12.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:11 smithi190 ceph-mon[31421]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:12.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:11 smithi079 ceph-mon[27946]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:12.693 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-22T06:44:12.705 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:12.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 -- 172.21.15.79:0/2757746232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88100e80 msgr2=0x7fee881012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:12.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 --2- 172.21.15.79:0/2757746232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88100e80 0x7fee881012a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fee80009ca0 tx=0x7fee8002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:12.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 -- 172.21.15.79:0/2757746232 shutdown_connections 2024-04-22T06:44:12.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 --2- 172.21.15.79:0/2757746232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88102990 0x7fee881092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:12.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 --2- 172.21.15.79:0/2757746232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee88101fd0 0x7fee88102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 --2- 172.21.15.79:0/2757746232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88100e80 0x7fee881012a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 -- 172.21.15.79:0/2757746232 >> 172.21.15.79:0/2757746232 conn(0x7fee880fc410 msgr2=0x7fee880fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 -- 172.21.15.79:0/2757746232 shutdown_connections 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.814+0000 7fee8f031640 1 -- 172.21.15.79:0/2757746232 wait complete. 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 Processor -- start 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 -- start start 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88100e80 0x7fee8806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee88101fd0 0x7fee8819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 0x7fee8819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fee8810c9d0 con 0x7fee88102990 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee8810c850 con 0x7fee88100e80 2024-04-22T06:44:12.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fee8810cb50 con 0x7fee88101fd0 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 0x7fee8819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 0x7fee8819f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52926/0 (socket says 172.21.15.79:52926) 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 -- 172.21.15.79:0/3926499946 learned_addr learned my addr 172.21.15.79:0/3926499946 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee87fff640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee88101fd0 0x7fee8819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8cdcd640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88100e80 0x7fee8806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 -- 172.21.15.79:0/3926499946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee88101fd0 msgr2=0x7fee8819cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee88101fd0 0x7fee8819cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 -- 172.21.15.79:0/3926499946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88100e80 msgr2=0x7fee8806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88100e80 0x7fee8806b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 -- 172.21.15.79:0/3926499946 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8819fde0 con 0x7fee88102990 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8cdcd640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88100e80 0x7fee8806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8d5ce640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 0x7fee8819f520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fee7000d750 tx=0x7fee7000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:12.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70017070 con 0x7fee88102990 2024-04-22T06:44:12.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee700054d0 con 0x7fee88102990 2024-04-22T06:44:12.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee881a00d0 con 0x7fee88102990 2024-04-22T06:44:12.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.815+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70004c60 con 0x7fee88102990 2024-04-22T06:44:12.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.816+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee881a05d0 con 0x7fee88102990 2024-04-22T06:44:12.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.817+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee881081b0 con 0x7fee88102990 2024-04-22T06:44:12.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.818+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fee70009bd0 con 0x7fee88102990 2024-04-22T06:44:12.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.818+0000 7fee85ffb640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fee740a3bb0 0x7fee740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:12.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.818+0000 7fee8cdcd640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fee740a3bb0 0x7fee740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:12.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.818+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fee700cfd80 con 0x7fee88102990 2024-04-22T06:44:12.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.819+0000 7fee8cdcd640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fee740a3bb0 0x7fee740a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fee80006010 tx=0x7fee80005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:12.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:12.823+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee7009c770 con 0x7fee88102990 2024-04-22T06:44:12.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:12 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3672241359' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:44:13.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.003+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee88062b50 con 0x7fee88102990 2024-04-22T06:44:13.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.005+0000 7fee85ffb640 1 -- 172.21.15.79:0/3926499946 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22650 (secure 0 0 0) 0x7fee7009c070 con 0x7fee88102990 2024-04-22T06:44:13.004 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:13.004 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":87,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:43:17.795510+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:13.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fee740a3bb0 msgr2=0x7fee740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fee740a3bb0 0x7fee740a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fee80006010 tx=0x7fee80005ef0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 msgr2=0x7fee8819f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 0x7fee8819f520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fee7000d750 tx=0x7fee7000dc10 comp rx=0 tx=0).stop 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 shutdown_connections 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fee740a3bb0 0x7fee740a6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee88102990 0x7fee8819f520 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee88101fd0 0x7fee8819cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 --2- 172.21.15.79:0/3926499946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee88100e80 0x7fee8806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 >> 172.21.15.79:0/3926499946 conn(0x7fee880fc410 msgr2=0x7fee88105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 shutdown_connections 2024-04-22T06:44:13.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.007+0000 7fee8f031640 1 -- 172.21.15.79:0/3926499946 wait complete. 2024-04-22T06:44:13.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:13.018 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 -- 172.21.15.79:0/2558158824 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 msgr2=0x7f79bc1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 --2- 172.21.15.79:0/2558158824 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc1024c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009900 tx=0x7f79ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 -- 172.21.15.79:0/2558158824 shutdown_connections 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 --2- 172.21.15.79:0/2558158824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79bc102a00 0x7f79bc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 --2- 172.21.15.79:0/2558158824 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc1024c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 --2- 172.21.15.79:0/2558158824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79bc100e60 0x7f79bc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 -- 172.21.15.79:0/2558158824 >> 172.21.15.79:0/2558158824 conn(0x7f79bc0fc420 msgr2=0x7f79bc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 -- 172.21.15.79:0/2558158824 shutdown_connections 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 -- 172.21.15.79:0/2558158824 wait complete. 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 Processor -- start 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.120+0000 7f79c472e640 1 -- start start 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79bc100e60 0x7f79bc06b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79bc102a00 0x7f79bc19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc10c9f0 con 0x7f79bc102040 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc10c870 con 0x7f79bc100e60 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc10cb70 con 0x7f79bc102a00 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc19cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52950/0 (socket says 172.21.15.79:52950) 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 -- 172.21.15.79:0/2507769855 learned_addr learned my addr 172.21.15.79:0/2507769855 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c24ca640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79bc100e60 0x7f79bc06b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 -- 172.21.15.79:0/2507769855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79bc102a00 msgr2=0x7f79bc19f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79bc102a00 0x7f79bc19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 -- 172.21.15.79:0/2507769855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79bc100e60 msgr2=0x7f79bc06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79bc100e60 0x7f79bc06b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 -- 172.21.15.79:0/2507769855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc19fe70 con 0x7f79bc102040 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c1cc9640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc19cf40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f79ac036040 tx=0x7f79ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac043020 con 0x7f79bc102040 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79bc1a00a0 con 0x7f79bc102040 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ac0054c0 con 0x7f79bc102040 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac003db0 con 0x7f79bc102040 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.121+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79bc1a05f0 con 0x7f79bc102040 2024-04-22T06:44:13.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.122+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79bc1082d0 con 0x7f79bc102040 2024-04-22T06:44:13.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.123+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f79ac046020 con 0x7f79bc102040 2024-04-22T06:44:13.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.124+0000 7f79b37fe640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79a80a3bb0 0x7f79a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.124+0000 7f79c24ca640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79a80a3bb0 0x7f79a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.124+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f79ac0f2990 con 0x7f79bc102040 2024-04-22T06:44:13.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.124+0000 7f79c24ca640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79a80a3bb0 0x7f79a80a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f79b8005db0 tx=0x7f79b8005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:13.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.128+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79ac04d020 con 0x7f79bc102040 2024-04-22T06:44:13.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:12 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3672241359' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:44:13.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:12 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3672241359' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:44:13.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.306+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79bc062bf0 con 0x7f79bc102040 2024-04-22T06:44:13.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.306+0000 7f79b37fe640 1 -- 172.21.15.79:0/2507769855 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f79ac0bf380 con 0x7f79bc102040 2024-04-22T06:44:13.305 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79a80a3bb0 msgr2=0x7f79a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79a80a3bb0 0x7f79a80a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f79b8005db0 tx=0x7f79b8005d40 comp rx=0 tx=0).stop 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 msgr2=0x7f79bc19cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc19cf40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f79ac036040 tx=0x7f79ac0042f0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 shutdown_connections 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f79a80a3bb0 0x7f79a80a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79bc102a00 0x7f79bc19f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79bc102040 0x7f79bc19cf40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 --2- 172.21.15.79:0/2507769855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79bc100e60 0x7f79bc06b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 >> 172.21.15.79:0/2507769855 conn(0x7f79bc0fc420 msgr2=0x7f79bc1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 shutdown_connections 2024-04-22T06:44:13.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.308+0000 7f79c472e640 1 -- 172.21.15.79:0/2507769855 wait complete. 2024-04-22T06:44:13.319 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 -- 172.21.15.79:0/165361667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8101fa0 msgr2=0x7f69d81023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 --2- 172.21.15.79:0/165361667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8101fa0 0x7f69d81023c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009900 tx=0x7f69c802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 -- 172.21.15.79:0/165361667 shutdown_connections 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 --2- 172.21.15.79:0/165361667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103b50 0x7f69d810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 --2- 172.21.15.79:0/165361667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d8103190 0x7f69d8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 --2- 172.21.15.79:0/165361667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8101fa0 0x7f69d81023c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 -- 172.21.15.79:0/165361667 >> 172.21.15.79:0/165361667 conn(0x7f69d80fd530 msgr2=0x7f69d80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 -- 172.21.15.79:0/165361667 shutdown_connections 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.420+0000 7f69df25c640 1 -- 172.21.15.79:0/165361667 wait complete. 2024-04-22T06:44:13.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 Processor -- start 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 -- start start 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d8101fa0 0x7f69d8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103190 0x7f69d81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 0x7f69d819f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f69d810dba0 con 0x7f69d8103b50 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f69d810da20 con 0x7f69d8103190 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69df25c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69d810dd20 con 0x7f69d8101fa0 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 0x7f69d819f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 0x7f69d819f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52968/0 (socket says 172.21.15.79:52968) 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 -- 172.21.15.79:0/2493297568 learned_addr learned my addr 172.21.15.79:0/2493297568 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69d7fff640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103190 0x7f69d81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 -- 172.21.15.79:0/2493297568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d8101fa0 msgr2=0x7f69d8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d8101fa0 0x7f69d8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 -- 172.21.15.79:0/2493297568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103190 msgr2=0x7f69d81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103190 0x7f69d81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69dd7f9640 1 -- 172.21.15.79:0/2493297568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d819ffb0 con 0x7f69d8103b50 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.421+0000 7f69d7fff640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103190 0x7f69d81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.422+0000 7f69dd7f9640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 0x7f69d819f940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f69d000b970 tx=0x7f69d000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.422+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0019070 con 0x7f69d8103b50 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.422+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69d0014440 con 0x7f69d8103b50 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.422+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00075d0 con 0x7f69d8103b50 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.422+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d81a02a0 con 0x7f69d8103b50 2024-04-22T06:44:13.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.422+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d81a0870 con 0x7f69d8103b50 2024-04-22T06:44:13.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.423+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d8109480 con 0x7f69d8103b50 2024-04-22T06:44:13.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.424+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f69d001c070 con 0x7f69d8103b50 2024-04-22T06:44:13.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.424+0000 7f69d5ffb640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69c00a3c30 0x7f69c00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.425+0000 7f69dcff8640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69c00a3c30 0x7f69c00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.425+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f69d00cf9c0 con 0x7f69d8103b50 2024-04-22T06:44:13.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.425+0000 7f69dcff8640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69c00a3c30 0x7f69c00a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036040 tx=0x7f69c8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:13.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.429+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69d00d1e20 con 0x7f69d8103b50 2024-04-22T06:44:13.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.604+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69d8062b50 con 0x7f69d8103b50 2024-04-22T06:44:13.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.604+0000 7f69d5ffb640 1 -- 172.21.15.79:0/2493297568 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f69d009c3b0 con 0x7f69d8103b50 2024-04-22T06:44:13.603 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69c00a3c30 msgr2=0x7f69c00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69c00a3c30 0x7f69c00a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036040 tx=0x7f69c8000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 msgr2=0x7f69d819f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 0x7f69d819f940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f69d000b970 tx=0x7f69d000be30 comp rx=0 tx=0).stop 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 shutdown_connections 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f69c00a3c30 0x7f69c00a60f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69d8103190 0x7f69d81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69d8103b50 0x7f69d819f940 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 --2- 172.21.15.79:0/2493297568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69d8101fa0 0x7f69d8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 >> 172.21.15.79:0/2493297568 conn(0x7f69d80fd530 msgr2=0x7f69d8106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 shutdown_connections 2024-04-22T06:44:13.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.606+0000 7f69df25c640 1 -- 172.21.15.79:0/2493297568 wait complete. 2024-04-22T06:44:13.616 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 -- 172.21.15.79:0/2345658375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0100e90 msgr2=0x7efee01012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 --2- 172.21.15.79:0/2345658375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0100e90 0x7efee01012b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7efed0009900 tx=0x7efed002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 -- 172.21.15.79:0/2345658375 shutdown_connections 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 --2- 172.21.15.79:0/2345658375 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee0101fe0 0x7efee0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 --2- 172.21.15.79:0/2345658375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee01029a0 0x7efee01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 --2- 172.21.15.79:0/2345658375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0100e90 0x7efee01012b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 -- 172.21.15.79:0/2345658375 >> 172.21.15.79:0/2345658375 conn(0x7efee00fc400 msgr2=0x7efee00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 -- 172.21.15.79:0/2345658375 shutdown_connections 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.718+0000 7efee5231640 1 -- 172.21.15.79:0/2345658375 wait complete. 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 Processor -- start 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 -- start start 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100e90 0x7efee0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 0x7efee01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee01029a0 0x7efee019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efee010ca40 con 0x7efee0101fe0 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efee010c8c0 con 0x7efee01029a0 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efee5231640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efee010cbc0 con 0x7efee0100e90 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 0x7efee01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 0x7efee01987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52982/0 (socket says 172.21.15.79:52982) 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 -- 172.21.15.79:0/4058088102 learned_addr learned my addr 172.21.15.79:0/4058088102 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:13.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 -- 172.21.15.79:0/4058088102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100e90 msgr2=0x7efee0198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efeded9d640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100e90 0x7efee0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100e90 0x7efee0198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efedf59e640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee01029a0 0x7efee019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 -- 172.21.15.79:0/4058088102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee01029a0 msgr2=0x7efee019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee01029a0 0x7efee019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 -- 172.21.15.79:0/4058088102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee019feb0 con 0x7efee0101fe0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efeded9d640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100e90 0x7efee0198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efedf59e640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee01029a0 0x7efee019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.719+0000 7efede59c640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 0x7efee01987d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efed400d770 tx=0x7efed400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.720+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4017070 con 0x7efee0101fe0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.720+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efed4009c30 con 0x7efee0101fe0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.720+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4003cc0 con 0x7efee0101fe0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.720+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee01a01a0 con 0x7efee0101fe0 2024-04-22T06:44:13.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.720+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee01a0770 con 0x7efee0101fe0 2024-04-22T06:44:13.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.721+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee01a09d0 con 0x7efee0101fe0 2024-04-22T06:44:13.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.722+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efed4003e60 con 0x7efee0101fe0 2024-04-22T06:44:13.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.723+0000 7efec7fff640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efec80a3bb0 0x7efec80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:13.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.723+0000 7efeded9d640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efec80a3bb0 0x7efec80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:13.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.723+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7efed40cfe30 con 0x7efee0101fe0 2024-04-22T06:44:13.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.723+0000 7efeded9d640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efec80a3bb0 0x7efec80a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efed00096c0 tx=0x7efed0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:13.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.727+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efed409c820 con 0x7efee0101fe0 2024-04-22T06:44:13.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.907+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efee01a0ba0 con 0x7efee0101fe0 2024-04-22T06:44:13.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.907+0000 7efec7fff640 1 -- 172.21.15.79:0/4058088102 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7efed4009de0 con 0x7efee0101fe0 2024-04-22T06:44:13.906 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:13.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efec80a3bb0 msgr2=0x7efec80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efec80a3bb0 0x7efec80a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efed00096c0 tx=0x7efed0000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 msgr2=0x7efee01987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 0x7efee01987d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efed400d770 tx=0x7efed400dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 shutdown_connections 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7efec80a3bb0 0x7efec80a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efee01029a0 0x7efee019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efee0101fe0 0x7efee01987d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 --2- 172.21.15.79:0/4058088102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100e90 0x7efee0198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 >> 172.21.15.79:0/4058088102 conn(0x7efee00fc400 msgr2=0x7efee00690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:13.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 shutdown_connections 2024-04-22T06:44:13.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:13.909+0000 7efee5231640 1 -- 172.21.15.79:0/4058088102 wait complete. 2024-04-22T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:13 smithi079 ceph-mon[27946]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:13.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:13 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3926499946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:13.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:13 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2507769855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:13.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:13 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2493297568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:13.919 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:14.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 -- 172.21.15.79:0/3638453205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103180 msgr2=0x7f5318103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 --2- 172.21.15.79:0/3638453205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103180 0x7f5318103600 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5308009900 tx=0x7f530802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:14.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 -- 172.21.15.79:0/3638453205 shutdown_connections 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 --2- 172.21.15.79:0/3638453205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103b40 0x7f531810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 --2- 172.21.15.79:0/3638453205 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103180 0x7f5318103600 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 --2- 172.21.15.79:0/3638453205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 0x7f53181023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 -- 172.21.15.79:0/3638453205 >> 172.21.15.79:0/3638453205 conn(0x7f53180fd500 msgr2=0x7f53180ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 -- 172.21.15.79:0/3638453205 shutdown_connections 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.018+0000 7f53204f1640 1 -- 172.21.15.79:0/3638453205 wait complete. 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 Processor -- start 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 -- start start 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 0x7f53181981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103180 0x7f53181986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 0x7f531819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f531810dad0 con 0x7f5318103b40 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f531810d950 con 0x7f5318101f90 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f53204f1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f531810dc50 con 0x7f5318103180 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 0x7f531819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 0x7f531819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52992/0 (socket says 172.21.15.79:52992) 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 -- 172.21.15.79:0/4174531553 learned_addr learned my addr 172.21.15.79:0/4174531553 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531e28d640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 0x7f53181981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531da8c640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103180 0x7f53181986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 -- 172.21.15.79:0/4174531553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103180 msgr2=0x7f53181986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103180 0x7f53181986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 -- 172.21.15.79:0/4174531553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 msgr2=0x7f53181981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 0x7f53181981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 -- 172.21.15.79:0/4174531553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531819fdd0 con 0x7f5318103b40 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531da8c640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103180 0x7f53181986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531e28d640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 0x7f53181981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.019+0000 7f531ea8e640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 0x7f531819f760 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5310009fd0 tx=0x7f531000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.020+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310017070 con 0x7f5318103b40 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.020+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53181a00c0 con 0x7f5318103b40 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.020+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5310003e20 con 0x7f5318103b40 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.020+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310010460 con 0x7f5318103b40 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.020+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53181a0610 con 0x7f5318103b40 2024-04-22T06:44:14.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.021+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5318109300 con 0x7f5318103b40 2024-04-22T06:44:14.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.022+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5310003970 con 0x7f5318103b40 2024-04-22T06:44:14.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.022+0000 7f530f7fe640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f53040a3bb0 0x7f53040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.022+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f53100d01f0 con 0x7f5318103b40 2024-04-22T06:44:14.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.022+0000 7f531e28d640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f53040a3bb0 0x7f53040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.023+0000 7f531e28d640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f53040a3bb0 0x7f53040a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53140058c0 tx=0x7f5314005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.026+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f531009cbe0 con 0x7f5318103b40 2024-04-22T06:44:14.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:13 smithi193 ceph-mon[31815]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:14.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:13 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3926499946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:14.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:13 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2507769855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:14.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:13 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2493297568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:14.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:13 smithi190 ceph-mon[31421]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:14.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:13 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3926499946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:13 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2507769855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:14.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:13 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2493297568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:14.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.248+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f531802d5c0 con 0x7f5318103b40 2024-04-22T06:44:14.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.248+0000 7f530f7fe640 1 -- 172.21.15.79:0/4174531553 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7f531009c4e0 con 0x7f5318103b40 2024-04-22T06:44:14.248 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:14.248 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:42:28.714055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24436,24440,24467]},"id":1}]} 2024-04-22T06:44:14.250 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:44:14.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f53040a3bb0 msgr2=0x7f53040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f53040a3bb0 0x7f53040a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53140058c0 tx=0x7f5314005ec0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 msgr2=0x7f531819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 0x7f531819f760 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5310009fd0 tx=0x7f531000b260 comp rx=0 tx=0).stop 2024-04-22T06:44:14.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 shutdown_connections 2024-04-22T06:44:14.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f53040a3bb0 0x7f53040a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5318103b40 0x7f531819f760 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5318103180 0x7f53181986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 --2- 172.21.15.79:0/4174531553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5318101f90 0x7f53181981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.250+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 >> 172.21.15.79:0/4174531553 conn(0x7f53180fd500 msgr2=0x7f53181064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.251+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 shutdown_connections 2024-04-22T06:44:14.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.251+0000 7f53204f1640 1 -- 172.21.15.79:0/4174531553 wait complete. 2024-04-22T06:44:14.261 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:14.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 -- 172.21.15.79:0/1389067170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c102420 msgr2=0x7f242c102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 --2- 172.21.15.79:0/1389067170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c102420 0x7f242c102840 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2424009ca0 tx=0x7f242402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 -- 172.21.15.79:0/1389067170 shutdown_connections 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 --2- 172.21.15.79:0/1389067170 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f242c101b10 0x7f242c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 --2- 172.21.15.79:0/1389067170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f242c101150 0x7f242c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 --2- 172.21.15.79:0/1389067170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c102420 0x7f242c102840 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 -- 172.21.15.79:0/1389067170 >> 172.21.15.79:0/1389067170 conn(0x7f242c0fbb80 msgr2=0x7f242c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 -- 172.21.15.79:0/1389067170 shutdown_connections 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 -- 172.21.15.79:0/1389067170 wait complete. 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 Processor -- start 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.360+0000 7f2433a25640 1 -- start start 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f242c101150 0x7f242c1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 0x7f242c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f242c102420 0x7f242c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f242c10ca20 con 0x7f242c101b10 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f242c10c8a0 con 0x7f242c101150 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f242c10cba0 con 0x7f242c102420 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 0x7f242c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 0x7f242c197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53020/0 (socket says 172.21.15.79:53020) 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 -- 172.21.15.79:0/438759172 learned_addr learned my addr 172.21.15.79:0/438759172 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f24317c1640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f242c101150 0x7f242c1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2431fc2640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f242c102420 0x7f242c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 -- 172.21.15.79:0/438759172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f242c102420 msgr2=0x7f242c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f242c102420 0x7f242c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 -- 172.21.15.79:0/438759172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f242c101150 msgr2=0x7f242c1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f242c101150 0x7f242c1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 -- 172.21.15.79:0/438759172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242c19ed10 con 0x7f242c101b10 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2430fc0640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 0x7f242c197630 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f241c00b4a0 tx=0x7f241c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c014070 con 0x7f242c101b10 2024-04-22T06:44:14.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f241c010ac0 con 0x7f242c101b10 2024-04-22T06:44:14.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242c19efa0 con 0x7f242c101b10 2024-04-22T06:44:14.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c017c30 con 0x7f242c101b10 2024-04-22T06:44:14.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.361+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f242c19f4c0 con 0x7f242c101b10 2024-04-22T06:44:14.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.362+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f242c1082b0 con 0x7f242c101b10 2024-04-22T06:44:14.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.364+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f241c017070 con 0x7f242c101b10 2024-04-22T06:44:14.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.364+0000 7f24227fc640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f24180a3c30 0x7f24180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.364+0000 7f24317c1640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f24180a3c30 0x7f24180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.365+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f241c0cf700 con 0x7f242c101b10 2024-04-22T06:44:14.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.365+0000 7f24317c1640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f24180a3c30 0x7f24180a60f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2424032380 tx=0x7f2424000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.369+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f241c09c0f0 con 0x7f242c101b10 2024-04-22T06:44:14.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.547+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f242c062b50 con 0x7f242c101b10 2024-04-22T06:44:14.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.549+0000 7f24227fc640 1 -- 172.21.15.79:0/438759172 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22650 (secure 0 0 0) 0x7f241c09b9f0 con 0x7f242c101b10 2024-04-22T06:44:14.547 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:14.547 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":87,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:43:17.795510+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:14.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f24180a3c30 msgr2=0x7f24180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f24180a3c30 0x7f24180a60f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2424032380 tx=0x7f2424000f40 comp rx=0 tx=0).stop 2024-04-22T06:44:14.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 msgr2=0x7f242c197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 0x7f242c197630 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f241c00b4a0 tx=0x7f241c00b960 comp rx=0 tx=0).stop 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 shutdown_connections 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f24180a3c30 0x7f24180a60f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f242c102420 0x7f242c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f242c101b10 0x7f242c197630 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 --2- 172.21.15.79:0/438759172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f242c101150 0x7f242c1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 >> 172.21.15.79:0/438759172 conn(0x7f242c0fbb80 msgr2=0x7f242c105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 shutdown_connections 2024-04-22T06:44:14.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.551+0000 7f2433a25640 1 -- 172.21.15.79:0/438759172 wait complete. 2024-04-22T06:44:14.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:14.561 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:14.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 -- 172.21.15.79:0/1848694241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c8100e90 msgr2=0x7f40c81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 --2- 172.21.15.79:0/1848694241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c8100e90 0x7f40c81012b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009ca0 tx=0x7f40c002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 -- 172.21.15.79:0/1848694241 shutdown_connections 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 --2- 172.21.15.79:0/1848694241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c8101fe0 0x7f40c8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 --2- 172.21.15.79:0/1848694241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c81029a0 0x7f40c81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 --2- 172.21.15.79:0/1848694241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c8100e90 0x7f40c81012b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 -- 172.21.15.79:0/1848694241 >> 172.21.15.79:0/1848694241 conn(0x7f40c80fc400 msgr2=0x7f40c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 -- 172.21.15.79:0/1848694241 shutdown_connections 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 -- 172.21.15.79:0/1848694241 wait complete. 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 Processor -- start 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.663+0000 7f40d0069640 1 -- start start 2024-04-22T06:44:14.664 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c8100e90 0x7f40c81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c8101fe0 0x7f40c81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 0x7f40c819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f40c810c9e0 con 0x7f40c81029a0 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40c810c860 con 0x7f40c8101fe0 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40c810cb60 con 0x7f40c8100e90 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 0x7f40c819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 0x7f40c819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53028/0 (socket says 172.21.15.79:53028) 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 -- 172.21.15.79:0/594482700 learned_addr learned my addr 172.21.15.79:0/594482700 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40cde05640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c8100e90 0x7f40c81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 -- 172.21.15.79:0/594482700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c8100e90 msgr2=0x7f40c81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c8100e90 0x7f40c81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 -- 172.21.15.79:0/594482700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c8101fe0 msgr2=0x7f40c81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c8101fe0 0x7f40c81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 -- 172.21.15.79:0/594482700 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c819fdd0 con 0x7f40c81029a0 2024-04-22T06:44:14.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40cde05640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c8100e90 0x7f40c81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40ce606640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 0x7f40c819f760 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f40b800d750 tx=0x7f40b800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8017070 con 0x7f40c81029a0 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40b80054d0 con 0x7f40c81029a0 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c81a00c0 con 0x7f40c81029a0 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8004c60 con 0x7f40c81029a0 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.664+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c81a0610 con 0x7f40c81029a0 2024-04-22T06:44:14.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.665+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c81a0870 con 0x7f40c81029a0 2024-04-22T06:44:14.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.666+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f40b8009bd0 con 0x7f40c81029a0 2024-04-22T06:44:14.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.667+0000 7f40c6ffd640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f40b00a3bb0 0x7f40b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.667+0000 7f40cde05640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f40b00a3bb0 0x7f40b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.667+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f40b80cfe40 con 0x7f40c81029a0 2024-04-22T06:44:14.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.667+0000 7f40cde05640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f40b00a3bb0 0x7f40b00a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40c0000f80 tx=0x7f40c00324c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.671+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40b809c830 con 0x7f40c81029a0 2024-04-22T06:44:14.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.850+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f40c81a0a40 con 0x7f40c81029a0 2024-04-22T06:44:14.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.851+0000 7f40c6ffd640 1 -- 172.21.15.79:0/594482700 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f40b809c130 con 0x7f40c81029a0 2024-04-22T06:44:14.849 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:14.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f40b00a3bb0 msgr2=0x7f40b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f40b00a3bb0 0x7f40b00a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40c0000f80 tx=0x7f40c00324c0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 msgr2=0x7f40c819f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 0x7f40c819f760 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f40b800d750 tx=0x7f40b800dc10 comp rx=0 tx=0).stop 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 shutdown_connections 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f40b00a3bb0 0x7f40b00a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40c8101fe0 0x7f40c81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40c81029a0 0x7f40c819f760 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 --2- 172.21.15.79:0/594482700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40c8100e90 0x7f40c81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 >> 172.21.15.79:0/594482700 conn(0x7f40c80fc400 msgr2=0x7f40c80fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 shutdown_connections 2024-04-22T06:44:14.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.853+0000 7f40d0069640 1 -- 172.21.15.79:0/594482700 wait complete. 2024-04-22T06:44:14.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:14 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4058088102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:14.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:14 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4174531553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:14.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:14 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/438759172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:14.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 -- 172.21.15.79:0/859431255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408102a30 msgr2=0x7fd408104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 --2- 172.21.15.79:0/859431255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408102a30 0x7fd408104e20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd404009900 tx=0x7fd40402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 -- 172.21.15.79:0/859431255 shutdown_connections 2024-04-22T06:44:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 --2- 172.21.15.79:0/859431255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd408105360 0x7fd408107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 --2- 172.21.15.79:0/859431255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd408107c90 0x7fd40810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 --2- 172.21.15.79:0/859431255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408102a30 0x7fd408104e20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 -- 172.21.15.79:0/859431255 >> 172.21.15.79:0/859431255 conn(0x7fd4080fc440 msgr2=0x7fd4080fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 -- 172.21.15.79:0/859431255 shutdown_connections 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 -- 172.21.15.79:0/859431255 wait complete. 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.965+0000 7fd410274640 1 Processor -- start 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 -- start start 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd408102a30 0x7fd408198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 0x7fd408198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd408107c90 0x7fd40819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd40810dbd0 con 0x7fd408105360 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd40810da50 con 0x7fd408102a30 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd40810dd50 con 0x7fd408107c90 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 0x7fd408198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 0x7fd408198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53050/0 (socket says 172.21.15.79:53050) 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 -- 172.21.15.79:0/154491686 learned_addr learned my addr 172.21.15.79:0/154491686 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40e811640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd408107c90 0x7fd40819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 -- 172.21.15.79:0/154491686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd408107c90 msgr2=0x7fd40819f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd408107c90 0x7fd40819f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40e010640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd408102a30 0x7fd408198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 -- 172.21.15.79:0/154491686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd408102a30 msgr2=0x7fd408198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd408102a30 0x7fd408198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 -- 172.21.15.79:0/154491686 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40819ff70 con 0x7fd408105360 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd40d80f640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 0x7fd408198890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800b450 tx=0x7fd3f800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8015020 con 0x7fd408105360 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4081a0260 con 0x7fd408105360 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3f8011d40 con 0x7fd408105360 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f800f340 con 0x7fd408105360 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.966+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4081a0830 con 0x7fd408105360 2024-04-22T06:44:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.967+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4080fdf30 con 0x7fd408105360 2024-04-22T06:44:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.968+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd3f8018020 con 0x7fd408105360 2024-04-22T06:44:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.969+0000 7fd3ff7fe640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd3f40a3bb0 0x7fd3f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.969+0000 7fd40e010640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd3f40a3bb0 0x7fd3f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.969+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd3f80d0510 con 0x7fd408105360 2024-04-22T06:44:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.970+0000 7fd40e010640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd3f40a3bb0 0x7fd3f40a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4040096c0 tx=0x7fd404000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:14.973+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3f801f020 con 0x7fd408105360 2024-04-22T06:44:15.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.151+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd40802d810 con 0x7fd408105360 2024-04-22T06:44:15.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.152+0000 7fd3ff7fe640 1 -- 172.21.15.79:0/154491686 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fd3f800d040 con 0x7fd408105360 2024-04-22T06:44:15.150 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.153+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd3f40a3bb0 msgr2=0x7fd3f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.153+0000 7fd410274640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd3f40a3bb0 0x7fd3f40a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4040096c0 tx=0x7fd404000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.153+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 msgr2=0x7fd408198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.153+0000 7fd410274640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 0x7fd408198890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800b450 tx=0x7fd3f800b910 comp rx=0 tx=0).stop 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 shutdown_connections 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd3f40a3bb0 0x7fd3f40a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd408107c90 0x7fd40819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd408105360 0x7fd408198890 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 --2- 172.21.15.79:0/154491686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd408102a30 0x7fd408198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 >> 172.21.15.79:0/154491686 conn(0x7fd4080fc440 msgr2=0x7fd4081034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:15.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 shutdown_connections 2024-04-22T06:44:15.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.154+0000 7fd410274640 1 -- 172.21.15.79:0/154491686 wait complete. 2024-04-22T06:44:15.163 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:15.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:14 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4058088102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:15.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:14 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4174531553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:15.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:14 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/438759172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:15.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:14 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4058088102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:15.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:14 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4174531553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:14 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/438759172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:15.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.265+0000 7f585c8f2640 1 -- 172.21.15.79:0/887261255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 msgr2=0x7f5858107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.265+0000 7f585c8f2640 1 --2- 172.21.15.79:0/887261255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f5858107730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f584c009900 tx=0x7f584c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- 172.21.15.79:0/887261255 shutdown_connections 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 --2- 172.21.15.79:0/887261255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5858107c70 0x7f585810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 --2- 172.21.15.79:0/887261255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f5858107730 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 --2- 172.21.15.79:0/887261255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858102a10 0x7f5858104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- 172.21.15.79:0/887261255 >> 172.21.15.79:0/887261255 conn(0x7f58580fc400 msgr2=0x7f58580fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- 172.21.15.79:0/887261255 shutdown_connections 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- 172.21.15.79:0/887261255 wait complete. 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 Processor -- start 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- start start 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5858102a10 0x7f5858102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f58581009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858107c70 0x7f5858100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f585810dac0 con 0x7f5858105340 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f585810d940 con 0x7f5858102a10 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.266+0000 7f585c8f2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f585810dc40 con 0x7f5858107c70 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f58581009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f58581009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53068/0 (socket says 172.21.15.79:53068) 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 -- 172.21.15.79:0/2552560426 learned_addr learned my addr 172.21.15.79:0/2552560426 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f585659c640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5858102a10 0x7f5858102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5856d9d640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858107c70 0x7f5858100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 -- 172.21.15.79:0/2552560426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858107c70 msgr2=0x7f5858100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858107c70 0x7f5858100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 -- 172.21.15.79:0/2552560426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5858102a10 msgr2=0x7f5858102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5858102a10 0x7f5858102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 -- 172.21.15.79:0/2552560426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5858101790 con 0x7f5858105340 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5856d9d640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858107c70 0x7f5858100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f5855d9b640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f58581009c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f584c000c00 tx=0x7f584c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c043020 con 0x7f5858105340 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58581a2320 con 0x7f5858105340 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f584c004440 con 0x7f5858105340 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c003c90 con 0x7f5858105340 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.267+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58581a28c0 con 0x7f5858105340 2024-04-22T06:44:15.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.268+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58580fdef0 con 0x7f5858105340 2024-04-22T06:44:15.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.269+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f584c046020 con 0x7f5858105340 2024-04-22T06:44:15.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.270+0000 7f58477fe640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58400a3bb0 0x7f58400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.270+0000 7f585659c640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58400a3bb0 0x7f58400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.270+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f584c0f2640 con 0x7f5858105340 2024-04-22T06:44:15.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.270+0000 7f585659c640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58400a3bb0 0x7f58400a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5848005fd0 tx=0x7f5848009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:15.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.275+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f584c04d020 con 0x7f5858105340 2024-04-22T06:44:15.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.452+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5858062b50 con 0x7f5858105340 2024-04-22T06:44:15.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.452+0000 7f58477fe640 1 -- 172.21.15.79:0/2552560426 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f584c0bf030 con 0x7f5858105340 2024-04-22T06:44:15.451 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58400a3bb0 msgr2=0x7f58400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58400a3bb0 0x7f58400a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5848005fd0 tx=0x7f5848009680 comp rx=0 tx=0).stop 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 msgr2=0x7f58581009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f58581009c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f584c000c00 tx=0x7f584c004040 comp rx=0 tx=0).stop 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 shutdown_connections 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f58400a3bb0 0x7f58400a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5858107c70 0x7f5858100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5858105340 0x7f58581009c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 --2- 172.21.15.79:0/2552560426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5858102a10 0x7f5858102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 >> 172.21.15.79:0/2552560426 conn(0x7f58580fc400 msgr2=0x7f58581048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 shutdown_connections 2024-04-22T06:44:15.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.454+0000 7f585c8f2640 1 -- 172.21.15.79:0/2552560426 wait complete. 2024-04-22T06:44:15.464 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:15.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 -- 172.21.15.79:0/2663309836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4107d00 msgr2=0x7fb7a410a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/2663309836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4107d00 0x7fb7a410a0f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009ca0 tx=0x7fb79c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 -- 172.21.15.79:0/2663309836 shutdown_connections 2024-04-22T06:44:15.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/2663309836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 0x7fb7a41077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/2663309836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4107d00 0x7fb7a410a0f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/2663309836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 -- 172.21.15.79:0/2663309836 >> 172.21.15.79:0/2663309836 conn(0x7fb7a40fc410 msgr2=0x7fb7a40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 -- 172.21.15.79:0/2663309836 shutdown_connections 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 -- 172.21.15.79:0/2663309836 wait complete. 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 Processor -- start 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.563+0000 7fb7a89e6640 1 -- start start 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 0x7fb7a4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7a4107d00 0x7fb7a419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a410dae0 con 0x7fb7a4102aa0 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a410d960 con 0x7fb7a41053d0 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a410dc60 con 0x7fb7a4107d00 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53076/0 (socket says 172.21.15.79:53076) 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 -- 172.21.15.79:0/3106858767 learned_addr learned my addr 172.21.15.79:0/3106858767 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a2d9d640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7a4107d00 0x7fb7a419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a1d9b640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 0x7fb7a4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 -- 172.21.15.79:0/3106858767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7a4107d00 msgr2=0x7fb7a419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7a4107d00 0x7fb7a419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 -- 172.21.15.79:0/3106858767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 msgr2=0x7fb7a4198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 0x7fb7a4198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 -- 172.21.15.79:0/3106858767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a419fd60 con 0x7fb7a4102aa0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a1d9b640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 0x7fb7a4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a259c640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4198140 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb79400d950 tx=0x7fb79400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794017070 con 0x7fb7a4102aa0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a419fff0 con 0x7fb7a4102aa0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb794009c30 con 0x7fb7a4102aa0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79400f620 con 0x7fb7a4102aa0 2024-04-22T06:44:15.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.564+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a41a0500 con 0x7fb7a4102aa0 2024-04-22T06:44:15.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.565+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a40fdf00 con 0x7fb7a4102aa0 2024-04-22T06:44:15.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.567+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb794005030 con 0x7fb7a4102aa0 2024-04-22T06:44:15.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.567+0000 7fb79b7fe640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7900a3bb0 0x7fb7900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.568+0000 7fb7a1d9b640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7900a3bb0 0x7fb7900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.568+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb7940d0000 con 0x7fb7a4102aa0 2024-04-22T06:44:15.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.568+0000 7fb7a1d9b640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7900a3bb0 0x7fb7900a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0058c0 tx=0x7fb78c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.571+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb79409c9f0 con 0x7fb7a4102aa0 2024-04-22T06:44:15.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.789+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7a402d5c0 con 0x7fb7a4102aa0 2024-04-22T06:44:15.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.790+0000 7fb79b7fe640 1 -- 172.21.15.79:0/3106858767 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11878 (secure 0 0 0) 0x7fb794009de0 con 0x7fb7a4102aa0 2024-04-22T06:44:15.789 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:15.789 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:42:28.714055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24436,24440,24467]},"id":1}]} 2024-04-22T06:44:15.789 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 21 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7900a3bb0 msgr2=0x7fb7900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7900a3bb0 0x7fb7900a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb78c0058c0 tx=0x7fb78c005850 comp rx=0 tx=0).stop 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 msgr2=0x7fb7a4198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4198140 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb79400d950 tx=0x7fb79400de10 comp rx=0 tx=0).stop 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 shutdown_connections 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb7900a3bb0 0x7fb7900a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7a41053d0 0x7fb7a4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7a4107d00 0x7fb7a419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 --2- 172.21.15.79:0/3106858767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7a4102aa0 0x7fb7a4198140 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 >> 172.21.15.79:0/3106858767 conn(0x7fb7a40fc410 msgr2=0x7fb7a4103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:15.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.792+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 shutdown_connections 2024-04-22T06:44:15.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.793+0000 7fb7a89e6640 1 -- 172.21.15.79:0/3106858767 wait complete. 2024-04-22T06:44:15.802 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 -- 172.21.15.79:0/3941226058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c107cd0 msgr2=0x7f684c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 --2- 172.21.15.79:0/3941226058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c107cd0 0x7f684c10a0c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6844009ca0 tx=0x7f684402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 -- 172.21.15.79:0/3941226058 shutdown_connections 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 --2- 172.21.15.79:0/3941226058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c107cd0 0x7f684c10a0c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 --2- 172.21.15.79:0/3941226058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f684c1053a0 0x7f684c107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 --2- 172.21.15.79:0/3941226058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f684c102a70 0x7f684c104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 -- 172.21.15.79:0/3941226058 >> 172.21.15.79:0/3941226058 conn(0x7f684c0fc400 msgr2=0x7f684c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 -- 172.21.15.79:0/3941226058 shutdown_connections 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.902+0000 7f6851596640 1 -- 172.21.15.79:0/3941226058 wait complete. 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 Processor -- start 2024-04-22T06:44:15.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 -- start start 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f684c102a70 0x7f684c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 0x7f684c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f684c107cd0 0x7f684c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f684c10db50 con 0x7f684c1053a0 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f684c10d9d0 con 0x7f684c107cd0 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f6851596640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f684c10dcd0 con 0x7f684c102a70 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f68427fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 0x7f684c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f68427fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 0x7f684c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53100/0 (socket says 172.21.15.79:53100) 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.903+0000 7f68427fc640 1 -- 172.21.15.79:0/711433088 learned_addr learned my addr 172.21.15.79:0/711433088 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f684affd640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f684c102a70 0x7f684c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f684b7fe640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f684c107cd0 0x7f684c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f68427fc640 1 -- 172.21.15.79:0/711433088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f684c102a70 msgr2=0x7f684c198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f68427fc640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f684c102a70 0x7f684c198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f68427fc640 1 -- 172.21.15.79:0/711433088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f684c107cd0 msgr2=0x7f684c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f68427fc640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f684c107cd0 0x7f684c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f68427fc640 1 -- 172.21.15.79:0/711433088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684c19fe40 con 0x7f684c1053a0 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f68427fc640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 0x7f684c198760 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f683400c970 tx=0x7f683400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834007bb0 con 0x7f684c1053a0 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6834007d50 con 0x7f684c1053a0 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684c1a0130 con 0x7f684c1053a0 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834018730 con 0x7f684c1053a0 2024-04-22T06:44:15.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.904+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684c1a0630 con 0x7f684c1053a0 2024-04-22T06:44:15.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.905+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f684c0fdef0 con 0x7f684c1053a0 2024-04-22T06:44:15.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.906+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f683400f990 con 0x7f684c1053a0 2024-04-22T06:44:15.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.906+0000 7f6848ff9640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f68380a3bb0 0x7f68380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:15.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.907+0000 7f684affd640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f68380a3bb0 0x7f68380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:15.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.907+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f68340d04e0 con 0x7f684c1053a0 2024-04-22T06:44:15.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.907+0000 7f684affd640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f68380a3bb0 0x7f68380a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f683c009890 tx=0x7f683c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:15.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:15.911+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6834014030 con 0x7f684c1053a0 2024-04-22T06:44:16.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.092+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f684c062b50 con 0x7f684c1053a0 2024-04-22T06:44:16.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.093+0000 7f6848ff9640 1 -- 172.21.15.79:0/711433088 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22650 (secure 0 0 0) 0x7f683409c880 con 0x7f684c1053a0 2024-04-22T06:44:16.092 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:16.092 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":87,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:43:17.795510+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f68380a3bb0 msgr2=0x7f68380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f68380a3bb0 0x7f68380a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f683c009890 tx=0x7f683c008040 comp rx=0 tx=0).stop 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 msgr2=0x7f684c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 0x7f684c198760 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f683400c970 tx=0x7f683400ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 shutdown_connections 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f68380a3bb0 0x7f68380a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f684c107cd0 0x7f684c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f684c1053a0 0x7f684c198760 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 --2- 172.21.15.79:0/711433088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f684c102a70 0x7f684c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 >> 172.21.15.79:0/711433088 conn(0x7f684c0fc400 msgr2=0x7f684c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 shutdown_connections 2024-04-22T06:44:16.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.095+0000 7f6851596640 1 -- 172.21.15.79:0/711433088 wait complete. 2024-04-22T06:44:16.104 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:15 smithi079 ceph-mon[27946]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:44:16.104 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:15 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/594482700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.104 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:15 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/154491686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.104 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:15 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2552560426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:42:24.011937+0000', 'modified': '2024-04-22T06:42:28.714055+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24467, 'mds_1': 24440, 'mds_2': 24436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24406': {'gid': 24406, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3775212644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3775212644}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3775212644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6835/2543246858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 2543246858}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 2543246858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/678626858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 678626858}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 678626858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.193:6839/746968688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 746968688}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 746968688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.190:6837/2879949884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 2879949884}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 2879949884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.190:6839/4154999918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 4154999918}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 4154999918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24467, 'qdb_cluster': [24436, 24440, 24467]} 2024-04-22T06:44:16.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T06:44:16.106 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:44:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:15 smithi193 ceph-mon[31815]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:44:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:15 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/594482700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:15 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/154491686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:15 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2552560426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:15 smithi190 ceph-mon[31421]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:44:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:15 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/594482700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:15 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/154491686' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:15 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2552560426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:16.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 -- 172.21.15.79:0/4089085122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d8103b60 msgr2=0x7fb0d810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/4089085122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d8103b60 0x7fb0d810a3f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009ca0 tx=0x7fb0d402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 -- 172.21.15.79:0/4089085122 shutdown_connections 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/4089085122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d8103b60 0x7fb0d810a3f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/4089085122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d81031a0 0x7fb0d8103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/4089085122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d8101fb0 0x7fb0d81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 -- 172.21.15.79:0/4089085122 >> 172.21.15.79:0/4089085122 conn(0x7fb0d80fd520 msgr2=0x7fb0d80ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 -- 172.21.15.79:0/4089085122 shutdown_connections 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.209+0000 7fb0dfa11640 1 -- 172.21.15.79:0/4089085122 wait complete. 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 Processor -- start 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 -- start start 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d8101fb0 0x7fb0d8075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 0x7fb0d8074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d8103b60 0x7fb0d8074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d810daf0 con 0x7fb0d81031a0 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d810d970 con 0x7fb0d8101fb0 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dfa11640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d810dc70 con 0x7fb0d8103b60 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 0x7fb0d8074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 0x7fb0d8074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53120/0 (socket says 172.21.15.79:53120) 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 -- 172.21.15.79:0/2146698491 learned_addr learned my addr 172.21.15.79:0/2146698491 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d8101fb0 0x7fb0d8075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d8103b60 0x7fb0d8074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 -- 172.21.15.79:0/2146698491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d8103b60 msgr2=0x7fb0d8074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d8103b60 0x7fb0d8074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 -- 172.21.15.79:0/2146698491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d8101fb0 msgr2=0x7fb0d8075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d8101fb0 0x7fb0d8075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.210+0000 7fb0dcfac640 1 -- 172.21.15.79:0/2146698491 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d8075110 con 0x7fb0d81031a0 2024-04-22T06:44:16.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.211+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 0x7fb0d8074340 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d770 tx=0x7fb0c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.211+0000 7fb0ce7fc640 1 -- 172.21.15.79:0/2146698491 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8017070 con 0x7fb0d81031a0 2024-04-22T06:44:16.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.211+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d81a22e0 con 0x7fb0d81031a0 2024-04-22T06:44:16.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.211+0000 7fb0ce7fc640 1 -- 172.21.15.79:0/2146698491 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c80054d0 con 0x7fb0d81031a0 2024-04-22T06:44:16.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.211+0000 7fb0ce7fc640 1 -- 172.21.15.79:0/2146698491 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8004c30 con 0x7fb0d81031a0 2024-04-22T06:44:16.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.211+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d81a27e0 con 0x7fb0d81031a0 2024-04-22T06:44:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0ce7fc640 1 -- 172.21.15.79:0/2146698491 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb0c8004e50 con 0x7fb0d81031a0 2024-04-22T06:44:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 Processor -- start 2024-04-22T06:44:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 -- start start 2024-04-22T06:44:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df262260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df262cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df269eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b2df1c6b10 con 0x55b2df2632d0 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b2df164240 con 0x55b2df262830 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b2df1c2ad0 con 0x55b2df261e40 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df269eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df269eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53136/0 (socket says 172.21.15.79:53136) 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 -- 172.21.15.79:0/264953948 learned_addr learned my addr 172.21.15.79:0/264953948 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df262cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0ce7fc640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0c40a3bb0 0x7fb0c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df262260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0c40a3bb0 0x7fb0c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 -- 172.21.15.79:0/264953948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 msgr2=0x55b2df262260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df262260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 -- 172.21.15.79:0/264953948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 msgr2=0x55b2df262cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df262cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.214+0000 7fb0ce7fc640 1 -- 172.21.15.79:0/2146698491 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb0c80cfe20 con 0x7fb0d81031a0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0dcfac640 1 -- 172.21.15.79:0/264953948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2deed92d0 con 0x55b2df2632d0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df262cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df269eb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8098620 tx=0x7fb0c8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a509232161a0c103 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0c40a3bb0 0x7fb0c40a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4032380 tx=0x7fb0d40324c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0c37fe640 1 -- 172.21.15.79:0/264953948 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80a4d80 con 0x55b2df2632d0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0c37fe640 1 -- 172.21.15.79:0/264953948 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c809c1c0 con 0x55b2df2632d0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.215+0000 7fb0c37fe640 1 -- 172.21.15.79:0/264953948 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80988b0 con 0x55b2df2632d0 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- 172.21.15.79:0/264953948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 msgr2=0x55b2df269eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df269eb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8098620 tx=0x7fb0c8009f90 comp rx=0 tx=0).stop 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- 172.21.15.79:0/264953948 shutdown_connections 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df262cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df269eb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/264953948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df262260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- 172.21.15.79:0/264953948 >> 172.21.15.79:0/264953948 conn(0x55b2df25df40 msgr2=0x55b2df25c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- 172.21.15.79:0/264953948 shutdown_connections 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- 172.21.15.79:0/264953948 wait complete. 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- start start 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 Processor -- start 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df26bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df26c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b2df26a450 con 0x55b2df2632d0 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b2df14f190 con 0x55b2df261e40 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b2df26a5d0 con 0x55b2df262830 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0e25ff740 10 client.0 _ll_register_callbacks cb 0x55b2df14a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.216+0000 7fb0ddfae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53140/0 (socket says 172.21.15.79:53140) 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 -- 172.21.15.79:0/2269821340 learned_addr learned my addr 172.21.15.79:0/2269821340 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df26c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df26bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 -- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 msgr2=0x55b2df26c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df26c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 -- 172.21.15.79:0/2269821340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 msgr2=0x55b2df26bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df26bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 -- 172.21.15.79:0/2269821340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2defa36b0 con 0x55b2df2632d0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df26bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273330 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4036000 tx=0x7fb0d40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be414bd068ae6c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/2269821340 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4052060 con 0x55b2df2632d0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/2269821340 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb0d4004040 con 0x55b2df2632d0 2024-04-22T06:44:16.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c809c1c0 con 0x55b2df2632d0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/2269821340 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb0d4039040 con 0x55b2df2632d0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0e25ff740 10 client.15204 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b2df276730 con 0x55b2df2632d0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.217+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/2269821340 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c809c1c0 con 0x55b2df2632d0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/2269821340 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb0d4034970 con 0x55b2df2632d0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0e25ff740 10 client.15204 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0e25ff740 10 client.15204 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0e25ff740 20 client.15204 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df278d80 0x55b2df27a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0e25ff740 4 client.15204 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2df1d4530 con 0x55b2df278d80 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df278d80 0x55b2df27a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df278d80 0x55b2df27a1f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dfa02d989ba5ef1a server_cookie=222a42890043fcf2 in_seq=0 out_seq=0 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.218+0000 7fb0c0ff9640 10 client.15204 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:16.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/2269821340 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2df1d4530 con 0x55b2df278d80 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0c0ff9640 10 client.15204 handle_command_reply: tid=0 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 client.15204 shutdown 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 msgr2=0x55b2df273330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273330 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4036000 tx=0x7fb0d40095c0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 shutdown_connections 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273330 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df26c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df278d80 0x55b2df27a1f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/2269821340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df26bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0c0ff9640 0 client.15204 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 >> 172.21.15.79:0/2269821340 conn(0x55b2df25df40 msgr2=0x55b2df25ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 shutdown_connections 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 1 -- 172.21.15.79:0/2269821340 wait complete. 2024-04-22T06:44:16.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.219+0000 7fb0e25ff740 20 client.15204 trim_cache size 0 max 16384 2024-04-22T06:44:16.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 Processor -- start 2024-04-22T06:44:16.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 -- start start 2024-04-22T06:44:16.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df266790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df266cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b2df278530 con 0x55b2df2632d0 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b2df278c00 con 0x55b2df261e40 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.237+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b2df2792d0 con 0x55b2df262830 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53152/0 (socket says 172.21.15.79:53152) 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1549756048 learned_addr learned my addr 172.21.15.79:0/1549756048 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0dcfac640 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df266cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1549756048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 msgr2=0x55b2df266cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df266cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1549756048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 msgr2=0x55b2df266790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df266790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1549756048 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2df25d630 con 0x55b2df2632d0 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4034d90 tx=0x7fb0d4005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac39abd5cb9caa10 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0c2ffd640 1 -- 172.21.15.79:0/1549756048 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d405c020 con 0x55b2df2632d0 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0c2ffd640 1 -- 172.21.15.79:0/1549756048 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0d4046030 con 0x55b2df2632d0 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.238+0000 7fb0c2ffd640 1 -- 172.21.15.79:0/1549756048 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4052070 con 0x55b2df2632d0 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1549756048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 msgr2=0x55b2df273250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4034d90 tx=0x7fb0d4005f30 comp rx=0 tx=0).stop 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1549756048 shutdown_connections 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df273250 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df262830 0x55b2df266cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1549756048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df261e40 0x55b2df266790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1549756048 >> 172.21.15.79:0/1549756048 conn(0x55b2df25df40 msgr2=0x55b2df269e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1549756048 shutdown_connections 2024-04-22T06:44:16.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1549756048 wait complete. 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- start start 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 Processor -- start 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df26eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df26f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df272550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b2df27a070 con 0x55b2df2632d0 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b2df2799a0 con 0x55b2df262830 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b2df27a740 con 0x55b2df261e40 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.239+0000 7fb0e25ff740 10 client.0 _ll_register_callbacks cb 0x7fb0d4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df272550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df272550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53154/0 (socket says 172.21.15.79:53154) 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1467813930 learned_addr learned my addr 172.21.15.79:0/1467813930 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df26eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 msgr2=0x55b2df26eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df26eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1467813930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 msgr2=0x55b2df26f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df26f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 -- 172.21.15.79:0/1467813930 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2df272b40 con 0x55b2df2632d0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df26eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0ddfae640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df272550 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4034940 tx=0x7fb0d4034970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bd9361ff386c64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/1467813930 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4066060 con 0x55b2df2632d0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/1467813930 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fb0d40042c0 con 0x55b2df2632d0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d4046030 con 0x55b2df2632d0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/1467813930 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb0d405c030 con 0x55b2df2632d0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0e25ff740 10 client.15210 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b2df2df350 con 0x55b2df2632d0 2024-04-22T06:44:16.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/1467813930 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0d4046030 con 0x55b2df2632d0 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.240+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/1467813930 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb0d404e900 con 0x55b2df2632d0 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 10 client.15210 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 10 client.15210 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 20 client.15210 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df2e3b30 0x55b2df2e2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 4 client.15210 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "status"}) -- 0x55b2df1ce3a0 con 0x55b2df2e3b30 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df2e3b30 0x55b2df2e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0c0ff9640 10 client.15210 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0dd7ad640 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df2e3b30 0x55b2df2e2840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f257da1a0204243c server_cookie=4fdd76f569e59db in_seq=0 out_seq=0 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0c0ff9640 1 -- 172.21.15.79:0/1467813930 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b2df1ce3a0 con 0x55b2df2e3b30 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0c0ff9640 10 client.15210 handle_command_reply: tid=0 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 client.15210 shutdown 2024-04-22T06:44:16.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 msgr2=0x55b2df272550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df272550 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4034940 tx=0x7fb0d4034970 comp rx=0 tx=0).stop 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 shutdown_connections 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.241+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x55b2df2e3b30 0x55b2df2e2840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b2df262830 0x55b2df26f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b2df2632d0 0x55b2df272550 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 1 --2- 172.21.15.79:0/1467813930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b2df261e40 0x55b2df26eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0c0ff9640 0 client.15210 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 >> 172.21.15.79:0/1467813930 conn(0x55b2df25df40 msgr2=0x55b2df269c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 shutdown_connections 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 1 -- 172.21.15.79:0/1467813930 wait complete. 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0e25ff740 20 client.15210 trim_cache size 0 max 16384 2024-04-22T06:44:16.247 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 2, 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24436, 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 108.53896635700001, 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 21, 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 87, 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 126.37992046700001 2024-04-22T06:44:16.248 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:16.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0c40a3bb0 msgr2=0x7fb0c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0c40a3bb0 0x7fb0c40a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4032380 tx=0x7fb0d40324c0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 msgr2=0x7fb0d8074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 0x7fb0d8074340 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d770 tx=0x7fb0c800dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:16.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 shutdown_connections 2024-04-22T06:44:16.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fb0c40a3bb0 0x7fb0c40a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d8103b60 0x7fb0d8074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0d81031a0 0x7fb0d8074340 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 --2- 172.21.15.79:0/2146698491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d8101fb0 0x7fb0d8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 >> 172.21.15.79:0/2146698491 conn(0x7fb0d80fd520 msgr2=0x7fb0d8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 shutdown_connections 2024-04-22T06:44:16.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.242+0000 7fb0dfa11640 1 -- 172.21.15.79:0/2146698491 wait complete. 2024-04-22T06:44:16.251 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 -- 172.21.15.79:0/680663021 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c1053c0 msgr2=0x7f131c1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 --2- 172.21.15.79:0/680663021 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c1053c0 0x7f131c1077b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f130c009900 tx=0x7f130c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 -- 172.21.15.79:0/680663021 shutdown_connections 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 --2- 172.21.15.79:0/680663021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c107cf0 0x7f131c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 --2- 172.21.15.79:0/680663021 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c1053c0 0x7f131c1077b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 --2- 172.21.15.79:0/680663021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f131c102a90 0x7f131c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 -- 172.21.15.79:0/680663021 >> 172.21.15.79:0/680663021 conn(0x7f131c0fc420 msgr2=0x7f131c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 -- 172.21.15.79:0/680663021 shutdown_connections 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.352+0000 7f1323a11640 1 -- 172.21.15.79:0/680663021 wait complete. 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 Processor -- start 2024-04-22T06:44:16.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 -- start start 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 0x7f131c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c1053c0 0x7f131c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f131c107cf0 0x7f131c19f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f131c10dbb0 con 0x7f131c102a90 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f131c10da30 con 0x7f131c1053c0 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f131c10dd30 con 0x7f131c107cf0 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 0x7f131c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 0x7f131c1983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53168/0 (socket says 172.21.15.79:53168) 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 -- 172.21.15.79:0/2098716717 learned_addr learned my addr 172.21.15.79:0/2098716717 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1320fac640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c1053c0 0x7f131c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1321fae640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f131c107cf0 0x7f131c19f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 -- 172.21.15.79:0/2098716717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f131c107cf0 msgr2=0x7f131c19f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f131c107cf0 0x7f131c19f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 -- 172.21.15.79:0/2098716717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c1053c0 msgr2=0x7f131c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c1053c0 0x7f131c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 -- 172.21.15.79:0/2098716717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c19ffe0 con 0x7f131c102a90 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1320fac640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c1053c0 0x7f131c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13217ad640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 0x7f131c1983c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f131800b4a0 tx=0x7f131800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13127fc640 1 -- 172.21.15.79:0/2098716717 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318014070 con 0x7f131c102a90 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131c1a02d0 con 0x7f131c102a90 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13127fc640 1 -- 172.21.15.79:0/2098716717 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1318010ac0 con 0x7f131c102a90 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.353+0000 7f13127fc640 1 -- 172.21.15.79:0/2098716717 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318017c30 con 0x7f131c102a90 2024-04-22T06:44:16.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.354+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131c1a0820 con 0x7f131c102a90 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.356+0000 7f13127fc640 1 -- 172.21.15.79:0/2098716717 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1318017070 con 0x7f131c102a90 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.356+0000 7f1326600740 1 Processor -- start 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.356+0000 7f1326600740 1 -- start start 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1326600740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64974e40 0x55bb64975260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1326600740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64975cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1326600740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb6497ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1326600740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bb648d9b10 con 0x55bb649762d0 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1326600740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb64877240 con 0x55bb64974e40 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1326600740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb648d5ad0 con 0x55bb64975830 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb6497ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb6497ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53182/0 (socket says 172.21.15.79:53182) 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 -- 172.21.15.79:0/2774025903 learned_addr learned my addr 172.21.15.79:0/2774025903 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f13217ad640 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64975cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f13127fc640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f13080a3bb0 0x7f13080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1321fae640 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64974e40 0x55bb64975260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 -- 172.21.15.79:0/2774025903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 msgr2=0x55bb64975cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64975cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 -- 172.21.15.79:0/2774025903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64974e40 msgr2=0x55bb64975260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64974e40 0x55bb64975260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 -- 172.21.15.79:0/2774025903 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb645ec2d0 con 0x55bb649762d0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1321fae640 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64974e40 0x55bb64975260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1321fae640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f13080a3bb0 0x7f13080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f13127fc640 1 -- 172.21.15.79:0/2098716717 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f13180cf840 con 0x7f131c102a90 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f1320fac640 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb6497ceb0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f131c06b050 tx=0x7f130c02c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a633678858d93cfb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f13077fe640 1 -- 172.21.15.79:0/2774025903 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c044020 con 0x55bb649762d0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f13077fe640 1 -- 172.21.15.79:0/2774025903 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f130c005040 con 0x55bb649762d0 2024-04-22T06:44:16.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.357+0000 7f13077fe640 1 -- 172.21.15.79:0/2774025903 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c004a20 con 0x55bb649762d0 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 -- 172.21.15.79:0/2774025903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 msgr2=0x55bb6497ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1321fae640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f13080a3bb0 0x7f13080a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55bb64970200 tx=0x7f131400e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb6497ceb0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f131c06b050 tx=0x7f130c02c960 comp rx=0 tx=0).stop 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 -- 172.21.15.79:0/2774025903 shutdown_connections 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb6497ceb0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64975cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 --2- 172.21.15.79:0/2774025903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64974e40 0x55bb64975260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 -- 172.21.15.79:0/2774025903 >> 172.21.15.79:0/2774025903 conn(0x55bb64970f40 msgr2=0x55bb6496f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 -- 172.21.15.79:0/2774025903 shutdown_connections 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 -- 172.21.15.79:0/2774025903 wait complete. 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.358+0000 7f1326600740 1 -- start start 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 Processor -- start 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64974e40 0x55bb6497ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 0x55bb6497f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb649762d0 0x55bb64986380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bb6497d450 con 0x55bb64975830 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb64862190 con 0x55bb649762d0 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb6497d5d0 con 0x55bb64974e40 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1326600740 10 client.0 _ll_register_callbacks cb 0x55bb6485d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 0x55bb6497f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 0x55bb6497f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53184/0 (socket says 172.21.15.79:53184) 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 -- 172.21.15.79:0/4180865112 learned_addr learned my addr 172.21.15.79:0/4180865112 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1321fae640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb649762d0 0x55bb64986380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f13217ad640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64974e40 0x55bb6497ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 -- 172.21.15.79:0/4180865112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64974e40 msgr2=0x55bb6497ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64974e40 0x55bb6497ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 -- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb649762d0 msgr2=0x55bb64986380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb649762d0 0x55bb64986380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 -- 172.21.15.79:0/4180865112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb646b66b0 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1321fae640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb649762d0 0x55bb64986380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f13217ad640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64974e40 0x55bb6497ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1320fac640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 0x55bb6497f340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f130c036040 tx=0x7f130c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb4976929afd6ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.359+0000 7f1304ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1304ff9640 1 -- 172.21.15.79:0/4180865112 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c04d030 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1304ff9640 1 -- 172.21.15.79:0/4180865112 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f130c004bf0 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f130c005040 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1304ff9640 1 -- 172.21.15.79:0/4180865112 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f130c044040 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 10 client.15222 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb64989660 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1304ff9640 1 -- 172.21.15.79:0/4180865112 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f130c005040 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1304ff9640 1 -- 172.21.15.79:0/4180865112 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f130c0544c0 con 0x55bb64975830 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 10 client.15222 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:16.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 10 client.15222 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 20 client.15222 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb6498b820 0x55bb6498cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 4 client.15222 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb648e7530 con 0x55bb6498b820 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.360+0000 7f13217ad640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb6498b820 0x55bb6498cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.361+0000 7f1304ff9640 10 client.15222 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.361+0000 7f13217ad640 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb6498b820 0x55bb6498cc90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5fd435854aedb03 server_cookie=808094ca7150afbe in_seq=0 out_seq=0 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1304ff9640 1 -- 172.21.15.79:0/4180865112 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb648e7530 con 0x55bb6498b820 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1304ff9640 10 client.15222 handle_command_reply: tid=0 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 client.15222 shutdown 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 msgr2=0x55bb6497f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 0x55bb6497f340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f130c036040 tx=0x7f130c005bd0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 shutdown_connections 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb6498b820 0x55bb6498cc90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb649762d0 0x55bb64986380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64975830 0x55bb6497f340 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1304ff9640 0 client.15222 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 --2- 172.21.15.79:0/4180865112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64974e40 0x55bb6497ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 >> 172.21.15.79:0/4180865112 conn(0x55bb64970f40 msgr2=0x55bb6496ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 shutdown_connections 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 1 -- 172.21.15.79:0/4180865112 wait complete. 2024-04-22T06:44:16.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.362+0000 7f1326600740 20 client.15222 trim_cache size 0 max 16384 2024-04-22T06:44:16.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 Processor -- start 2024-04-22T06:44:16.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 -- start start 2024-04-22T06:44:16.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 0x55bb6498a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64975830 0x55bb64986340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb649762d0 0x55bb64978c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bb6498afd0 con 0x55bb64974e40 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb6498b6a0 con 0x55bb64975830 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1326600740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb6498bd70 con 0x55bb649762d0 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 0x55bb6498a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 0x55bb6498a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53186/0 (socket says 172.21.15.79:53186) 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 -- 172.21.15.79:0/405624584 learned_addr learned my addr 172.21.15.79:0/405624584 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 -- 172.21.15.79:0/405624584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb649762d0 msgr2=0x55bb64978c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb649762d0 0x55bb64978c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 -- 172.21.15.79:0/405624584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64975830 msgr2=0x55bb64986340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64975830 0x55bb64986340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 -- 172.21.15.79:0/405624584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6497ff80 con 0x55bb64974e40 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f13217ad640 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 0x55bb6498a9b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f131809b830 tx=0x7f1318010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f408b158fd5deaa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1306ffd640 1 -- 172.21.15.79:0/405624584 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131809ce60 con 0x55bb64974e40 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1306ffd640 1 -- 172.21.15.79:0/405624584 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13180a47a0 con 0x55bb64974e40 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.380+0000 7f1306ffd640 1 -- 172.21.15.79:0/405624584 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131801e040 con 0x55bb64974e40 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 -- 172.21.15.79:0/405624584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 msgr2=0x55bb6498a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 0x55bb6498a9b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f131809b830 tx=0x7f1318010f90 comp rx=0 tx=0).stop 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 -- 172.21.15.79:0/405624584 shutdown_connections 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64975830 0x55bb64986340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb649762d0 0x55bb64978c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 --2- 172.21.15.79:0/405624584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb64974e40 0x55bb6498a9b0 secure :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f131809b830 tx=0x7f1318010f90 comp rx=0 tx=0).stop 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 -- 172.21.15.79:0/405624584 >> 172.21.15.79:0/405624584 conn(0x55bb64970f40 msgr2=0x55bb6496a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 -- 172.21.15.79:0/405624584 shutdown_connections 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 -- 172.21.15.79:0/405624584 wait complete. 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 -- start start 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.381+0000 7f1326600740 1 Processor -- start 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64980580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb64980ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64985470 0x55bb649858f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bb6498cb10 con 0x55bb649762d0 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb6498c440 con 0x55bb64985470 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb6498d1e0 con 0x55bb64975830 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1326600740 10 client.0 _ll_register_callbacks cb 0x7f130c005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb64980ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb64980ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53198/0 (socket says 172.21.15.79:53198) 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 -- 172.21.15.79:0/3566276994 learned_addr learned my addr 172.21.15.79:0/3566276994 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f13217ad640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64980580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1321fae640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64985470 0x55bb649858f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 -- 172.21.15.79:0/3566276994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 msgr2=0x55bb64980580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64980580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 -- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64985470 msgr2=0x55bb649858f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64985470 0x55bb649858f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 -- 172.21.15.79:0/3566276994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb64985e60 con 0x55bb649762d0 2024-04-22T06:44:16.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1321fae640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64985470 0x55bb649858f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1320fac640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb64980ac0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f130c054900 tx=0x7f130c02cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c521b03f3b75be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1304ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1304ff9640 1 -- 172.21.15.79:0/3566276994 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c04d020 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.382+0000 7f1304ff9640 1 -- 172.21.15.79:0/3566276994 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f130c004bf0 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13180a47a0 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1304ff9640 1 -- 172.21.15.79:0/3566276994 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f130c004df0 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 10 client.15228 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb649f20b0 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1304ff9640 1 -- 172.21.15.79:0/3566276994 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13180a47a0 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1304ff9640 1 -- 172.21.15.79:0/3566276994 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f130c054250 con 0x55bb649762d0 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 10 client.15228 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 10 client.15228 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 20 client.15228 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb649f6be0 0x55bb649f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 4 client.15228 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:16.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "status"}) -- 0x55bb648e13a0 con 0x55bb649f6be0 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.383+0000 7f13217ad640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb649f6be0 0x55bb649f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1304ff9640 10 client.15228 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f13217ad640 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb649f6be0 0x55bb649f36f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=914b1e8cf05b5c56 server_cookie=e0f78920b2bcfdca in_seq=0 out_seq=0 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1304ff9640 1 -- 172.21.15.79:0/3566276994 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55bb648e13a0 con 0x55bb649f6be0 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1304ff9640 10 client.15228 handle_command_reply: tid=0 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 client.15228 shutdown 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 msgr2=0x55bb64980ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb64980ac0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f130c054900 tx=0x7f130c02cd60 comp rx=0 tx=0).stop 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 shutdown_connections 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55bb649f6be0 0x55bb649f36f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb64985470 0x55bb649858f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb649762d0 0x55bb64980ac0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1304ff9640 0 client.15228 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 --2- 172.21.15.79:0/3566276994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb64975830 0x55bb64980580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 >> 172.21.15.79:0/3566276994 conn(0x55bb64970f40 msgr2=0x55bb6497ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 shutdown_connections 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 1 -- 172.21.15.79:0/3566276994 wait complete. 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.384+0000 7f1326600740 20 client.15228 trim_cache size 0 max 16384 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 1, 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24440, 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 109.688931787, 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 21, 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 87, 2024-04-22T06:44:16.389 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:44:16.390 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 129.36602442 2024-04-22T06:44:16.390 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:16.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f13080a3bb0 msgr2=0x7f13080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f13080a3bb0 0x7f13080a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55bb64970200 tx=0x7f131400e040 comp rx=0 tx=0).stop 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 msgr2=0x7f131c1983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 0x7f131c1983c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f131800b4a0 tx=0x7f131800b960 comp rx=0 tx=0).stop 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 shutdown_connections 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f13080a3bb0 0x7f13080a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f131c1053c0 0x7f131c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f131c107cf0 0x7f131c19f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 --2- 172.21.15.79:0/2098716717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f131c102a90 0x7f131c1983c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 >> 172.21.15.79:0/2098716717 conn(0x7f131c0fc420 msgr2=0x7f131c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 shutdown_connections 2024-04-22T06:44:16.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.385+0000 7f1323a11640 1 -- 172.21.15.79:0/2098716717 wait complete. 2024-04-22T06:44:16.394 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- 172.21.15.79:0/1751692555 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 msgr2=0x7fd7241024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/1751692555 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd7241024c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd720009900 tx=0x7fd72002bc50 comp rx=0 tx=0).stop 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- 172.21.15.79:0/1751692555 shutdown_connections 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/1751692555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724102a00 0x7fd724109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/1751692555 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd7241024c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/1751692555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd724100e60 0x7fd724101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- 172.21.15.79:0/1751692555 >> 172.21.15.79:0/1751692555 conn(0x7fd7240fc420 msgr2=0x7fd7240fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- 172.21.15.79:0/1751692555 shutdown_connections 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- 172.21.15.79:0/1751692555 wait complete. 2024-04-22T06:44:16.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 Processor -- start 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- start start 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724100e60 0x7fd72406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd72419d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd724102a00 0x7fd72419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd72410cb10 con 0x7fd724102040 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd72410c990 con 0x7fd724102a00 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd72b6d2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd72410cc90 con 0x7fd724100e60 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd728c6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd72419d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd728c6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd72419d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53216/0 (socket says 172.21.15.79:53216) 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.494+0000 7fd728c6d640 1 -- 172.21.15.79:0/2304971800 learned_addr learned my addr 172.21.15.79:0/2304971800 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd728c6d640 1 -- 172.21.15.79:0/2304971800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724100e60 msgr2=0x7fd72406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd728c6d640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724100e60 0x7fd72406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd728c6d640 1 -- 172.21.15.79:0/2304971800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd724102a00 msgr2=0x7fd72419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd728c6d640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd724102a00 0x7fd72419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd728c6d640 1 -- 172.21.15.79:0/2304971800 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7241a0050 con 0x7fd724102040 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd728c6d640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd72419d120 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd720036040 tx=0x7fd7200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd71a7fc640 1 -- 172.21.15.79:0/2304971800 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720043020 con 0x7fd724102040 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd71a7fc640 1 -- 172.21.15.79:0/2304971800 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7200054c0 con 0x7fd724102040 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7241a0280 con 0x7fd724102040 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd71a7fc640 1 -- 172.21.15.79:0/2304971800 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720003db0 con 0x7fd724102040 2024-04-22T06:44:16.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.495+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7241a0850 con 0x7fd724102040 2024-04-22T06:44:16.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.497+0000 7fd71a7fc640 1 -- 172.21.15.79:0/2304971800 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd720046020 con 0x7fd724102040 2024-04-22T06:44:16.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 Processor -- start 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 -- start start 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f7e40 0x5560183f8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183f8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f92d0 0x5560183ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55601835cb10 con 0x5560183f8830 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5560182fa240 con 0x5560183f7e40 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556018358ad0 con 0x5560183f92d0 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd71a7fc640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd7100a3bb0 0x7fd7100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183f8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183f8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53230/0 (socket says 172.21.15.79:53230) 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 -- 172.21.15.79:0/2388385049 learned_addr learned my addr 172.21.15.79:0/2388385049 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd728c6d640 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f92d0 0x5560183ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd729c6f640 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f7e40 0x5560183f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd729c6f640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd7100a3bb0 0x7fd7100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd71a7fc640 1 -- 172.21.15.79:0/2304971800 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd7200f2900 con 0x7fd724102040 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 -- 172.21.15.79:0/2388385049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f92d0 msgr2=0x5560183ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f92d0 0x5560183ffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 -- 172.21.15.79:0/2388385049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f7e40 msgr2=0x5560183f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f7e40 0x5560183f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.498+0000 7fd72946e640 1 -- 172.21.15.79:0/2388385049 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55601806f2d0 con 0x5560183f8830 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd72946e640 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183f8cb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd7140097f0 tx=0x7fd714009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3afef5cf8968f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd729c6f640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd7100a3bb0 0x7fd7100a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00d8a0 tx=0x7fd71c00b680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd70f7fe640 1 -- 172.21.15.79:0/2388385049 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714015020 con 0x5560183f8830 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd70f7fe640 1 -- 172.21.15.79:0/2388385049 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd714011420 con 0x5560183f8830 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd70f7fe640 1 -- 172.21.15.79:0/2388385049 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7140105a0 con 0x5560183f8830 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd72e3a0740 1 -- 172.21.15.79:0/2388385049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 msgr2=0x5560183f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.499+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183f8cb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd7140097f0 tx=0x7fd714009cb0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- 172.21.15.79:0/2388385049 shutdown_connections 2024-04-22T06:44:16.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f92d0 0x5560183ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183f8cb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/2388385049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f7e40 0x5560183f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- 172.21.15.79:0/2388385049 >> 172.21.15.79:0/2388385049 conn(0x5560183f3f40 msgr2=0x5560183f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- 172.21.15.79:0/2388385049 shutdown_connections 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- 172.21.15.79:0/2388385049 wait complete. 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- start start 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 Processor -- start 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x556018401e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018402340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018409380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5560184004b0 con 0x5560183f92d0 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5560182e5190 con 0x5560183f8830 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556018400630 con 0x5560183f7e40 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd72e3a0740 10 client.0 _ll_register_callbacks cb 0x5560182e08b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd729c6f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018409380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd729c6f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018409380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53242/0 (socket says 172.21.15.79:53242) 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd729c6f640 1 -- 172.21.15.79:0/4170012290 learned_addr learned my addr 172.21.15.79:0/4170012290 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.500+0000 7fd728c6d640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018402340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72946e640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x556018401e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd729c6f640 1 -- 172.21.15.79:0/4170012290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 msgr2=0x556018401e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd729c6f640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x556018401e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd729c6f640 1 -- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 msgr2=0x556018402340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd729c6f640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018402340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd729c6f640 1 -- 172.21.15.79:0/4170012290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560181396b0 con 0x5560183f92d0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd728c6d640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018402340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd729c6f640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018409380 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd71c014a40 tx=0x7fd71c008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120257c9e79a1284 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd70cff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd70cff9640 1 -- 172.21.15.79:0/4170012290 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c014e40 con 0x5560183f92d0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd70cff9640 1 -- 172.21.15.79:0/4170012290 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd71c0067e0 con 0x5560183f92d0 2024-04-22T06:44:16.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd714011420 con 0x5560183f92d0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd70cff9640 1 -- 172.21.15.79:0/4170012290 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd71c00de20 con 0x5560183f92d0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 10 client.15240 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55601840c7b0 con 0x5560183f92d0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd70cff9640 1 -- 172.21.15.79:0/4170012290 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd714011420 con 0x5560183f92d0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd70cff9640 1 -- 172.21.15.79:0/4170012290 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd71c0243f0 con 0x5560183f92d0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 10 client.15240 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 10 client.15240 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 20 client.15240 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x5560184102f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 4 client.15240 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.501+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55601836a530 con 0x55601840ee80 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.502+0000 7fd72946e640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x5560184102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.502+0000 7fd70cff9640 10 client.15240 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.502+0000 7fd72946e640 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x5560184102f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abf05e023398f022 server_cookie=66d40a60ba9c131d in_seq=0 out_seq=0 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd70cff9640 1 -- 172.21.15.79:0/4170012290 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55601836a530 con 0x55601840ee80 2024-04-22T06:44:16.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd70cff9640 10 client.15240 handle_command_reply: tid=0 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 client.15240 shutdown 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 msgr2=0x556018409380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018409380 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd71c014a40 tx=0x7fd71c008f90 comp rx=0 tx=0).stop 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 shutdown_connections 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x5560184102f0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018402340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018409380 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd70cff9640 0 client.15240 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/4170012290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x556018401e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 >> 172.21.15.79:0/4170012290 conn(0x5560183f3f40 msgr2=0x5560183f1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 shutdown_connections 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 1 -- 172.21.15.79:0/4170012290 wait complete. 2024-04-22T06:44:16.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.503+0000 7fd72e3a0740 20 client.15240 trim_cache size 0 max 16384 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.521+0000 7fd72e3a0740 1 Processor -- start 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.521+0000 7fd72e3a0740 1 -- start start 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560183f2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183fc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f92d0 0x5560183fcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55601840e630 con 0x5560183f8830 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55601840ed00 con 0x5560183f92d0 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55601840f3d0 con 0x5560183f7e40 2024-04-22T06:44:16.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183fc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183fc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53254/0 (socket says 172.21.15.79:53254) 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 -- 172.21.15.79:0/1403296988 learned_addr learned my addr 172.21.15.79:0/1403296988 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 -- 172.21.15.79:0/1403296988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 msgr2=0x5560183f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560183f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 -- 172.21.15.79:0/1403296988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f92d0 msgr2=0x5560183fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f92d0 0x5560183fcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 -- 172.21.15.79:0/1403296988 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560183ffa20 con 0x5560183f8830 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd728c6d640 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183fc730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd720006010 tx=0x7fd7200916d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5f1b129c5aa974 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd70effd640 1 -- 172.21.15.79:0/1403296988 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7200be4f0 con 0x5560183f8830 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd70effd640 1 -- 172.21.15.79:0/1403296988 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd72003ce30 con 0x5560183f8830 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.522+0000 7fd70effd640 1 -- 172.21.15.79:0/1403296988 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7200c7800 con 0x5560183f8830 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1403296988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 msgr2=0x5560183fc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183fc730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd720006010 tx=0x7fd7200916d0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1403296988 shutdown_connections 2024-04-22T06:44:16.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f92d0 0x5560183fcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f8830 0x5560183fc730 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1403296988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560183f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1403296988 >> 172.21.15.79:0/1403296988 conn(0x5560183f3f40 msgr2=0x5560183ed800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1403296988 shutdown_connections 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1403296988 wait complete. 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- start start 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 Processor -- start 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560184089c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018408f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018477270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x556018410170 con 0x5560183f92d0 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55601840faa0 con 0x5560183f8830 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556018410840 con 0x5560183f7e40 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.523+0000 7fd72e3a0740 10 client.0 _ll_register_callbacks cb 0x7fd71c004e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:16.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018477270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018477270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53270/0 (socket says 172.21.15.79:53270) 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 -- 172.21.15.79:0/1795741552 learned_addr learned my addr 172.21.15.79:0/1795741552 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 -- 172.21.15.79:0/1795741552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 msgr2=0x5560184089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd72946e640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560184089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd728c6d640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018408f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560184089c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 -- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 msgr2=0x556018408f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018408f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 -- 172.21.15.79:0/1795741552 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556018477880 con 0x5560183f92d0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd72946e640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560184089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd729c6f640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018477270 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009120 tx=0x7fd71c009150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6422d31f441b92eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd70cff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd70cff9640 1 -- 172.21.15.79:0/1795741552 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c034030 con 0x5560183f92d0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd70cff9640 1 -- 172.21.15.79:0/1795741552 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fd71c016040 con 0x5560183f92d0 2024-04-22T06:44:16.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd72003ce30 con 0x5560183f92d0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd70cff9640 1 -- 172.21.15.79:0/1795741552 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd71c024ce0 con 0x5560183f92d0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd72e3a0740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556018478be0 con 0x5560183f92d0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd70cff9640 1 -- 172.21.15.79:0/1795741552 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd72003ce30 con 0x5560183f92d0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.524+0000 7fd70cff9640 1 -- 172.21.15.79:0/1795741552 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd71c03b710 con 0x5560183f92d0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72e3a0740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72e3a0740 10 client.15246 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72e3a0740 20 client.15246 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x55601847c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72e3a0740 4 client.15246 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "status"}) -- 0x5560183643a0 con 0x55601840ee80 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72946e640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x55601847c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd70cff9640 10 client.15246 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd72946e640 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x55601847c510 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b620bb6ae560b5f0 server_cookie=3b2ee7b61c221e43 in_seq=0 out_seq=0 2024-04-22T06:44:16.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd70cff9640 1 -- 172.21.15.79:0/1795741552 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5560183643a0 con 0x55601840ee80 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.525+0000 7fd70cff9640 10 client.15246 handle_command_reply: tid=0 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 client.15246 shutdown 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 msgr2=0x556018477270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018477270 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009120 tx=0x7fd71c009150 comp rx=0 tx=0).stop 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 shutdown_connections 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55601840ee80 0x55601847c510 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5560183f8830 0x556018408f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5560183f92d0 0x556018477270 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 --2- 172.21.15.79:0/1795741552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560183f7e40 0x5560184089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd70cff9640 0 client.15246 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 >> 172.21.15.79:0/1795741552 conn(0x5560183f3f40 msgr2=0x5560183fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 shutdown_connections 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 1 -- 172.21.15.79:0/1795741552 wait complete. 2024-04-22T06:44:16.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72e3a0740 20 client.15246 trim_cache size 0 max 16384 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 0, 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24467, 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 111.854427497, 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 21, 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 87, 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:44:16.531 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 118.150401632 2024-04-22T06:44:16.532 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:16.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd7100a3bb0 msgr2=0x7fd7100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd7100a3bb0 0x7fd7100a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00d8a0 tx=0x7fd71c00b680 comp rx=0 tx=0).stop 2024-04-22T06:44:16.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 msgr2=0x7fd72419d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd72419d120 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd720036040 tx=0x7fd7200042f0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 shutdown_connections 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fd7100a3bb0 0x7fd7100a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd724102a00 0x7fd72419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd724102040 0x7fd72419d120 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 --2- 172.21.15.79:0/2304971800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd724100e60 0x7fd72406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 >> 172.21.15.79:0/2304971800 conn(0x7fd7240fc420 msgr2=0x7fd724105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 shutdown_connections 2024-04-22T06:44:16.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.526+0000 7fd72b6d2640 1 -- 172.21.15.79:0/2304971800 wait complete. 2024-04-22T06:44:16.536 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T06:44:16.544 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T06:44:16.544 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T06:44:16.545 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T06:44:16.545 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T06:44:16.546 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:44:16.546 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:44:16.546 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-22T06:44:16.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- 172.21.15.79:0/1939248105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102040 msgr2=0x7f63481024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- 172.21.15.79:0/1939248105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102040 0x7f63481024c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f633c009900 tx=0x7f633c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:16.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- 172.21.15.79:0/1939248105 shutdown_connections 2024-04-22T06:44:16.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- 172.21.15.79:0/1939248105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6348102a00 0x7f6348109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- 172.21.15.79:0/1939248105 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102040 0x7f63481024c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- 172.21.15.79:0/1939248105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6348100e60 0x7f6348101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- 172.21.15.79:0/1939248105 >> 172.21.15.79:0/1939248105 conn(0x7f63480fc420 msgr2=0x7f63480fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- 172.21.15.79:0/1939248105 shutdown_connections 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- 172.21.15.79:0/1939248105 wait complete. 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 Processor -- start 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- start start 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6348100e60 0x7f63481982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6348102040 0x7f6348198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 0x7f634819f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f634810dc10 con 0x7f6348102a00 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f634810da90 con 0x7f6348100e60 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f6350043640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f634810dd90 con 0x7f6348102040 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.646+0000 7f634e5e0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 0x7f634819f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 0x7f634819f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53296/0 (socket says 172.21.15.79:53296) 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 -- 172.21.15.79:0/3615121184 learned_addr learned my addr 172.21.15.79:0/3615121184 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634dddf640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6348100e60 0x7f63481982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634d5de640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6348102040 0x7f6348198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 -- 172.21.15.79:0/3615121184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6348102040 msgr2=0x7f6348198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6348102040 0x7f6348198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 -- 172.21.15.79:0/3615121184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6348100e60 msgr2=0x7f63481982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6348100e60 0x7f63481982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 -- 172.21.15.79:0/3615121184 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634819ff10 con 0x7f6348102a00 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f634e5e0640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 0x7f634819f8a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f633800b970 tx=0x7f633800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338019070 con 0x7f6348102a00 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6338014440 con 0x7f6348102a00 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63481a0200 con 0x7f6348102a00 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63380075d0 con 0x7f6348102a00 2024-04-22T06:44:16.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.647+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63481a0780 con 0x7f6348102a00 2024-04-22T06:44:16.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.648+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6348108250 con 0x7f6348102a00 2024-04-22T06:44:16.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.649+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f633801c070 con 0x7f6348102a00 2024-04-22T06:44:16.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.650+0000 7f6346ffd640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63300a3c50 0x7f63300a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:16.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.650+0000 7f634dddf640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63300a3c50 0x7f63300a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:16.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.650+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f63380cf950 con 0x7f6348102a00 2024-04-22T06:44:16.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.650+0000 7f634dddf640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63300a3c50 0x7f63300a6110 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6348101bb0 tx=0x7f6340009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:16.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.655+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f633809c340 con 0x7f6348102a00 2024-04-22T06:44:16.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:16.899+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f6348062b50 con 0x7f6348102a00 2024-04-22T06:44:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:16 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3106858767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:16 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/711433088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:17.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:16 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3106858767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:17.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:16 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/711433088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:16 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3106858767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:16 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/711433088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:17.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.789+0000 7f6346ffd640 1 -- 172.21.15.79:0/3615121184 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f633809bc40 con 0x7f6348102a00 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63300a3c50 msgr2=0x7f63300a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63300a3c50 0x7f63300a6110 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6348101bb0 tx=0x7f6340009110 comp rx=0 tx=0).stop 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 msgr2=0x7f634819f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 0x7f634819f8a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f633800b970 tx=0x7f633800bd70 comp rx=0 tx=0).stop 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 shutdown_connections 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f63300a3c50 0x7f63300a6110 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6348102a00 0x7f634819f8a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6348102040 0x7f6348198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 --2- 172.21.15.79:0/3615121184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6348100e60 0x7f63481982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 >> 172.21.15.79:0/3615121184 conn(0x7f63480fc420 msgr2=0x7f6348105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:17.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 shutdown_connections 2024-04-22T06:44:17.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.792+0000 7f6350043640 1 -- 172.21.15.79:0/3615121184 wait complete. 2024-04-22T06:44:17.808 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-22T06:44:17.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 -- 172.21.15.79:0/2296077877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102040 msgr2=0x7f2bbc1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 --2- 172.21.15.79:0/2296077877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102040 0x7f2bbc1024c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0009900 tx=0x7f2bb002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 -- 172.21.15.79:0/2296077877 shutdown_connections 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 --2- 172.21.15.79:0/2296077877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bbc102a00 0x7f2bbc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 --2- 172.21.15.79:0/2296077877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102040 0x7f2bbc1024c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 --2- 172.21.15.79:0/2296077877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bbc100e60 0x7f2bbc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 -- 172.21.15.79:0/2296077877 >> 172.21.15.79:0/2296077877 conn(0x7f2bbc0fc420 msgr2=0x7f2bbc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 -- 172.21.15.79:0/2296077877 shutdown_connections 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 -- 172.21.15.79:0/2296077877 wait complete. 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.916+0000 7f2bc1184640 1 Processor -- start 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 -- start start 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bbc100e60 0x7f2bbc06a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bbc102040 0x7f2bbc19f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 0x7f2bbc1a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2bbc10ca50 con 0x7f2bbc102a00 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2bbc10c8d0 con 0x7f2bbc100e60 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2bbc10cbd0 con 0x7f2bbc102040 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 0x7f2bbc1a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bbc100e60 0x7f2bbc06a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 0x7f2bbc1a17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53330/0 (socket says 172.21.15.79:53330) 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bba59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bbc102040 0x7f2bbc19f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 -- 172.21.15.79:0/322875419 learned_addr learned my addr 172.21.15.79:0/322875419 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 -- 172.21.15.79:0/322875419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bbc102040 msgr2=0x7f2bbc19f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bbc102040 0x7f2bbc19f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 -- 172.21.15.79:0/322875419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bbc100e60 msgr2=0x7f2bbc06a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bbc100e60 0x7f2bbc06a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 -- 172.21.15.79:0/322875419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bbc1a1fd0 con 0x7f2bbc102a00 2024-04-22T06:44:17.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bbb59e640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 0x7f2bbc1a17e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400eaa0 tx=0x7f2bb400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:17.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb400cdb0 con 0x7f2bbc102a00 2024-04-22T06:44:17.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bb4005020 con 0x7f2bbc102a00 2024-04-22T06:44:17.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bbc1a2260 con 0x7f2bbc102a00 2024-04-22T06:44:17.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.917+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40116d0 con 0x7f2bbc102a00 2024-04-22T06:44:17.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.918+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bbc1a27d0 con 0x7f2bbc102a00 2024-04-22T06:44:17.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.919+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bbc108200 con 0x7f2bbc102a00 2024-04-22T06:44:17.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.920+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f2bb40051d0 con 0x7f2bbc102a00 2024-04-22T06:44:17.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.920+0000 7f2ba3fff640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ba40a3d50 0x7f2ba40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.920+0000 7f2bbad9d640 1 -- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ba40a3d50 msgr2=0x7f2ba40a6210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:17.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.920+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f2bb40d0400 con 0x7f2bbc102a00 2024-04-22T06:44:17.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.920+0000 7f2bbad9d640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ba40a3d50 0x7f2ba40a6210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:17.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:17.925+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bb4016030 con 0x7f2bbc102a00 2024-04-22T06:44:18.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.110+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f2bbc062b80 con 0x7f2bbc102a00 2024-04-22T06:44:18.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.114+0000 7f2ba3fff640 1 -- 172.21.15.79:0/322875419 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v33)=0 v33) ==== 138+0+0 (secure 0 0 0) 0x7f2bb409c7a0 con 0x7f2bbc102a00 2024-04-22T06:44:18.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ba40a3d50 msgr2=0x7f2ba40a6210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:18.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ba40a3d50 0x7f2ba40a6210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 msgr2=0x7f2bbc1a17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 0x7f2bbc1a17e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400eaa0 tx=0x7f2bb400ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:18.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 shutdown_connections 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f2ba40a3d50 0x7f2ba40a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bbc102a00 0x7f2bbc1a17e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bbc102040 0x7f2bbc19f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 --2- 172.21.15.79:0/322875419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bbc100e60 0x7f2bbc06a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 >> 172.21.15.79:0/322875419 conn(0x7f2bbc0fc420 msgr2=0x7f2bbc1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 shutdown_connections 2024-04-22T06:44:18.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.117+0000 7f2bc1184640 1 -- 172.21.15.79:0/322875419 wait complete. 2024-04-22T06:44:18.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:17 smithi079 ceph-mon[27946]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:44:18.126 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:17 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3615121184' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:44:18.126 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: ignoring --setuser ceph since I am not root 2024-04-22T06:44:18.126 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: ignoring --setgroup ceph since I am not root 2024-04-22T06:44:18.126 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:17.978+0000 7f9cec857200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:44:18.127 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:17 smithi190 ceph-mon[31421]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:17 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3615121184' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: ignoring --setuser ceph since I am not root 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: ignoring --setgroup ceph since I am not root 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:17 smithi190 ceph-mgr[32805]: -- 172.21.15.190:0/3192015633 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563231a725a0 con 0x563231a46800 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:17.973+0000 7fe0e05fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:44:18.157 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:18.158+0000 7fe0e05fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:17 smithi193 ceph-mon[31815]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:17 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3615121184' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: ignoring --setuser ceph since I am not root 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: ignoring --setgroup ceph since I am not root 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:17 smithi193 ceph-mgr[33686]: -- 172.21.15.193:0/675428924 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c89c9a25a0 con 0x55c89c977000 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:17.976+0000 7f450bdf1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:44:18.160 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:18.161+0000 7f450bdf1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 -- 172.21.15.79:0/967517675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 msgr2=0x7fc060107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 --2- 172.21.15.79:0/967517675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc060107810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc050009900 tx=0x7fc05002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 -- 172.21.15.79:0/967517675 shutdown_connections 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 --2- 172.21.15.79:0/967517675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc060107d50 0x7fc06010a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 --2- 172.21.15.79:0/967517675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc060107810 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 --2- 172.21.15.79:0/967517675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc060102af0 0x7fc060104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.229+0000 7fc06810e640 1 -- 172.21.15.79:0/967517675 >> 172.21.15.79:0/967517675 conn(0x7fc0600fc420 msgr2=0x7fc0600fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 -- 172.21.15.79:0/967517675 shutdown_connections 2024-04-22T06:44:18.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 -- 172.21.15.79:0/967517675 wait complete. 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 Processor -- start 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 -- start start 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc060102af0 0x7fc060198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc0601986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc060107d50 0x7fc06019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc06010db00 con 0x7fc060105420 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc06010d980 con 0x7fc060102af0 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc06810e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc06010dc80 con 0x7fc060107d50 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc0656a9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc0601986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc0656a9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc0601986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53342/0 (socket says 172.21.15.79:53342) 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc0656a9640 1 -- 172.21.15.79:0/4277167284 learned_addr learned my addr 172.21.15.79:0/4277167284 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc065eaa640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc060102af0 0x7fc060198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.230+0000 7fc0666ab640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc060107d50 0x7fc06019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc0656a9640 1 -- 172.21.15.79:0/4277167284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc060107d50 msgr2=0x7fc06019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc0656a9640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc060107d50 0x7fc06019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc0656a9640 1 -- 172.21.15.79:0/4277167284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc060102af0 msgr2=0x7fc060198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc0656a9640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc060102af0 0x7fc060198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc0656a9640 1 -- 172.21.15.79:0/4277167284 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06019fda0 con 0x7fc060105420 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc0656a9640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc0601986c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc050000c00 tx=0x7fc050005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050043020 con 0x7fc060105420 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc050004350 con 0x7fc060105420 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0601a0030 con 0x7fc060105420 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050003b80 con 0x7fc060105420 2024-04-22T06:44:18.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.231+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0601a0580 con 0x7fc060105420 2024-04-22T06:44:18.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.232+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0600fdf10 con 0x7fc060105420 2024-04-22T06:44:18.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.233+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fc050046020 con 0x7fc060105420 2024-04-22T06:44:18.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.234+0000 7fc056ffd640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc04c0a3ca0 0x7fc04c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.234+0000 7fc065eaa640 1 -- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc04c0a3ca0 msgr2=0x7fc04c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:18.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.234+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7fc0500f29b0 con 0x7fc060105420 2024-04-22T06:44:18.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.234+0000 7fc065eaa640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc04c0a3ca0 0x7fc04c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:18.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.242+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc05004d020 con 0x7fc060105420 2024-04-22T06:44:18.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.425+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fc060062bc0 con 0x7fc060105420 2024-04-22T06:44:18.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.430+0000 7fc056ffd640 1 -- 172.21.15.79:0/4277167284 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) ==== 133+0+0 (secure 0 0 0) 0x7fc0500bf3a0 con 0x7fc060105420 2024-04-22T06:44:18.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc04c0a3ca0 msgr2=0x7fc04c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:18.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc04c0a3ca0 0x7fc04c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 msgr2=0x7fc0601986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc0601986c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc050000c00 tx=0x7fc050005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 shutdown_connections 2024-04-22T06:44:18.431 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7fc04c0a3ca0 0x7fc04c0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc060107d50 0x7fc06019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc060105420 0x7fc0601986c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 --2- 172.21.15.79:0/4277167284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc060102af0 0x7fc060198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 >> 172.21.15.79:0/4277167284 conn(0x7fc0600fc420 msgr2=0x7fc060103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:18.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 shutdown_connections 2024-04-22T06:44:18.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:18.433+0000 7fc06810e640 1 -- 172.21.15.79:0/4277167284 wait complete. 2024-04-22T06:44:18.439 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:18.263+0000 7f450bdf1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:44:18.441 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:18.171+0000 7f9cec857200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:18.441 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:18.283+0000 7f9cec857200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:44:18.442 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T06:44:18.465 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T06:44:18.465 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-22T06:44:18.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:18.466 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-04-22T06:44:18.466 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:18.262+0000 7fe0e05fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:44:18.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:18.481 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link delete ceph-brx 2024-04-22T06:44:18.546 INFO:teuthology.orchestra.run.smithi079.stderr:Cannot find device "ceph-brx" 2024-04-22T06:44:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:44:18.548 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-22T06:44:18.548 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T06:44:18.548 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:18.549 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T06:44:18.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:18.602+0000 7f9cec857200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:44:18.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:18.692+0000 7f9cec857200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:44:18.831 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:18.939 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:18.569+0000 7f450bdf1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:44:18.939 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:18.653+0000 7f450bdf1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:44:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3615121184' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:44:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-mon[31815]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T06:44:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/322875419' entity='client.admin' 2024-04-22T06:44:18.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:18 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4277167284' entity='client.admin' 2024-04-22T06:44:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3615121184' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:44:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-mon[31421]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T06:44:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/322875419' entity='client.admin' 2024-04-22T06:44:18.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4277167284' entity='client.admin' 2024-04-22T06:44:18.953 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:18.573+0000 7fe0e05fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:44:18.953 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:18.659+0000 7fe0e05fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:44:19.072 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3615121184' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:44:19.072 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-mon[27946]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T06:44:19.072 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/322875419' entity='client.admin' 2024-04-22T06:44:19.072 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:18 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4277167284' entity='client.admin' 2024-04-22T06:44:20.168 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:19.830+0000 7f450bdf1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:44:20.168 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:19.999+0000 7f450bdf1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:20.168 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:20.169+0000 7f450bdf1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:44:20.188 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:19.846+0000 7fe0e05fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:44:20.188 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:20.017+0000 7fe0e05fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:20.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:19.907+0000 7f9cec857200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:44:20.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:20.082+0000 7f9cec857200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:20.452 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:20.189+0000 7fe0e05fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:44:20.500 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:20.257+0000 7f9cec857200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:44:20.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/4039302715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0102d60 msgr2=0x7f4fd01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/4039302715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0102d60 0x7f4fd01031e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009990 tx=0x7f4fc0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/4039302715 shutdown_connections 2024-04-22T06:44:20.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/4039302715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0103720 0x7f4fd010f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/4039302715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fd0108bc0 0x7f4fd0108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/4039302715 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0102d60 0x7f4fd01031e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/4039302715 >> 172.21.15.79:0/4039302715 conn(0x7f4fd007a9d0 msgr2=0x7f4fd007ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/4039302715 shutdown_connections 2024-04-22T06:44:20.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/4039302715 wait complete. 2024-04-22T06:44:20.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 Processor -- start 2024-04-22T06:44:20.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.605+0000 7f4fd7d2c700 1 -- start start 2024-04-22T06:44:20.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102d60 0x7f4fd0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 0x7f4fd0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fd0108bc0 0x7f4fd00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd0112cf0 con 0x7f4fd0103720 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd0112b70 con 0x7f4fd0108bc0 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd0112e70 con 0x7f4fd0102d60 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 0x7f4fd0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 0x7f4fd0075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53366/0 (socket says 172.21.15.79:53366) 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd5ac8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102d60 0x7f4fd0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 -- 172.21.15.79:0/261155969 learned_addr learned my addr 172.21.15.79:0/261155969 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd62c9700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fd0108bc0 0x7f4fd00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 -- 172.21.15.79:0/261155969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102d60 msgr2=0x7f4fd0074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102d60 0x7f4fd0074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 -- 172.21.15.79:0/261155969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fd0108bc0 msgr2=0x7f4fd00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fd0108bc0 0x7f4fd00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 -- 172.21.15.79:0/261155969 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd01a7650 con 0x7f4fd0103720 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd52c7700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 0x7f4fd0075280 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0000c00 tx=0x7f4fc00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0027070 con 0x7f4fd0103720 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fc0003f20 con 0x7f4fd0103720 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.606+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd01a78e0 con 0x7f4fd0103720 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.607+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0022ca0 con 0x7f4fd0103720 2024-04-22T06:44:20.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.607+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd01a7d30 con 0x7f4fd0103720 2024-04-22T06:44:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.608+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd0069730 con 0x7f4fd0103720 2024-04-22T06:44:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.609+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f4fc002a070 con 0x7f4fd0103720 2024-04-22T06:44:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.609+0000 7f4fc6ffd700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 0x7f4fbc0a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.609+0000 7f4fd5ac8700 1 -- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 msgr2=0x7f4fbc0a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.609+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f4fc00d12b0 con 0x7f4fd0103720 2024-04-22T06:44:20.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.609+0000 7f4fd5ac8700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 0x7f4fbc0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:20.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.614+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fc005eff0 con 0x7f4fd0103720 2024-04-22T06:44:20.751 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:20.458+0000 7f450bdf1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:44:20.751 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:20.550+0000 7f450bdf1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:44:20.752 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:20.752+0000 7f450bdf1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:44:20.778 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:20.481+0000 7fe0e05fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:44:20.778 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:20.574+0000 7fe0e05fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:44:20.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.810+0000 7f4fd5ac8700 1 -- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 msgr2=0x7f4fbc0a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:20.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.810+0000 7f4fd5ac8700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 0x7f4fbc0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:20.808 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:20.558+0000 7f9cec857200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:44:20.808 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:20.655+0000 7f9cec857200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.863+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4fd002df80 con 0x7f4fd0103720 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.863+0000 7f4fc6ffd700 1 -- 172.21.15.79:0/261155969 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7f4fc0013070 con 0x7f4fd0103720 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi079.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 msgr2=0x7f4fbc0a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 0x7f4fbc0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 msgr2=0x7f4fd0075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 0x7f4fd0075280 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0000c00 tx=0x7f4fc00047b0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 shutdown_connections 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fd0108bc0 0x7f4fd00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102d60 0x7f4fd0074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4fbc0a3c90 0x7f4fbc0a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 --2- 172.21.15.79:0/261155969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fd0103720 0x7f4fd0075280 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 >> 172.21.15.79:0/261155969 conn(0x7f4fd007a9d0 msgr2=0x7f4fd0101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 shutdown_connections 2024-04-22T06:44:20.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:20.866+0000 7f4fd7d2c700 1 -- 172.21.15.79:0/261155969 wait complete. 2024-04-22T06:44:21.084 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:20 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/261155969' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:44:21.085 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:20.860+0000 7f9cec857200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:44:21.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/261155969' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:44:21.189 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:20.947+0000 7f450bdf1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:21.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/261155969' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:44:21.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:20.779+0000 7fe0e05fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:44:21.203 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:20.976+0000 7fe0e05fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:21.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:21.086+0000 7f9cec857200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:21.734 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T06:44:21.734 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T06:44:21.734 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi079.front.sepia.ceph.com 2024-04-22T06:44:21.734 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T06:44:21.734 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:21.735 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T06:44:21.735 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T06:44:21.735 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T06:44:21.735 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T06:44:21.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:21.735 DEBUG:teuthology.orchestra.run.smithi079:> ip addr 2024-04-22T06:44:21.752 INFO:teuthology.orchestra.run.smithi079.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T06:44:21.753 INFO:teuthology.orchestra.run.smithi079.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T06:44:21.753 INFO:teuthology.orchestra.run.smithi079.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T06:44:21.753 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:44:21.753 INFO:teuthology.orchestra.run.smithi079.stdout: inet6 ::1/128 scope host 2024-04-22T06:44:21.753 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:44:21.754 INFO:teuthology.orchestra.run.smithi079.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:44:21.754 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:d8:a1:6a brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:21.754 INFO:teuthology.orchestra.run.smithi079.stdout: altname enp7s0f0 2024-04-22T06:44:21.754 INFO:teuthology.orchestra.run.smithi079.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:44:21.754 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:d8:a1:6b brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:21.754 INFO:teuthology.orchestra.run.smithi079.stdout: altname enp7s0f1 2024-04-22T06:44:21.755 INFO:teuthology.orchestra.run.smithi079.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:44:21.755 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:8f:ce:86 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:21.755 INFO:teuthology.orchestra.run.smithi079.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T06:44:21.755 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:8f:ce:87 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:21.755 INFO:teuthology.orchestra.run.smithi079.stdout: inet 172.21.15.79/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T06:44:21.755 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft 42043sec preferred_lft 42043sec 2024-04-22T06:44:21.756 INFO:teuthology.orchestra.run.smithi079.stdout: inet6 fe80::29a7:525d:ae18:4d4e/64 scope link noprefixroute 2024-04-22T06:44:21.756 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:44:21.756 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link add name ceph-brx type bridge 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip addr flush dev ceph-brx 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set ceph-brx up 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T06:44:21.757 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-04-22T06:44:21.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:21.935 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:21.683+0000 7f9cec857200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:44:21.939 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:21.536+0000 7f450bdf1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:44:21.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:21.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:21.947 DEBUG:teuthology.orchestra.run.smithi079:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T06:44:21.952 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:21.577+0000 7fe0e05fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:44:21.972 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-04-22T06:44:21.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:21.973 DEBUG:teuthology.orchestra.run.smithi079:> route 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi079.stdout:Kernel IP routing table 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi079.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T06:44:22.042 INFO:teuthology.orchestra.run.smithi079.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T06:44:22.042 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T06:44:22.042 INFO:teuthology.orchestra.run.smithi079.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T06:44:22.042 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:22.042 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-04-22T06:44:22.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T06:44:22.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T06:44:22.043 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T06:44:22.043 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-04-22T06:44:22.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:22.193 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:22 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:22.162+0000 7f9cec857200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:44:22.247 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:21.955+0000 7f450bdf1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:44:22.248 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:22.051+0000 7f450bdf1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:44:22.300 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:22.002+0000 7fe0e05fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:44:22.301 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:22.098+0000 7fe0e05fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:44:22.301 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:22.301+0000 7fe0e05fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:44:22.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:22.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:22.315 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-04-22T06:44:22.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:22.329 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list-id 2024-04-22T06:44:22.384 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:22.384 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-04-22T06:44:22.384 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T06:44:22.384 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T06:44:22.384 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:22.480 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:22 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:22.273+0000 7f9cec857200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:44:22.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:22.505 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T06:44:22.506 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-04-22T06:44:22.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:22.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:22.630 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:22.631 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-04-22T06:44:22.631 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set brx.0 up 2024-04-22T06:44:22.631 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T06:44:22.631 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-04-22T06:44:22.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:22.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:22.711 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T06:44:22.711 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:22.725 INFO:teuthology.orchestra.run.smithi079.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T06:44:22.725 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T06:44:22.725 DEBUG:teuthology.orchestra.run.smithi079:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:22.740 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:22 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:22.506+0000 7f9cec857200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:44:22.753 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-22T06:44:22.753 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-22T06:44:22.753 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/c65b5f2f-2526-4052-a234-373065fd4b48 2024-04-22T06:44:22.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:22.753 DEBUG:teuthology.orchestra.run.smithi079:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/c65b5f2f-2526-4052-a234-373065fd4b48 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T06:44:22.849 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T06:44:22.850 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T06:44:22.850 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-04-22T06:44:22.960 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T06:44:22.960 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T06:44:22.960 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-04-22T06:44:22.960 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.190:3300,172.21.15.193:3300" 2024-04-22T06:44:22.960 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.79:3300,172.21.15.190:3300,172.21.15.193:3300:/volumes/qa/sv_1/c65b5f2f-2526-4052-a234-373065fd4b48 2024-04-22T06:44:22.960 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,key=0,fsid=146994a8-0072-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T06:44:22.961 INFO:teuthology.orchestra.run.smithi079.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T06:44:22.961 INFO:teuthology.orchestra.run.smithi079.stdout: You just mounted a file system that supports labels which does not 2024-04-22T06:44:22.961 INFO:teuthology.orchestra.run.smithi079.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T06:44:22.961 INFO:teuthology.orchestra.run.smithi079.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T06:44:22.961 INFO:teuthology.orchestra.run.smithi079.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T06:44:22.963 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T06:44:22.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:22.963 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:22.988 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:44:23.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:44:23.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:23.085 INFO:teuthology.orchestra.run.smithi079.stdout:client24851 2024-04-22T06:44:23.086 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:44:23.086 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:44:23.097 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:22.249+0000 7f450bdf1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:44:23.097 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:22.766+0000 7f450bdf1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:44:23.097 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:22.859+0000 7f450bdf1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:44:23.101 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:44:23.163 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:22.827+0000 7fe0e05fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:44:23.163 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:22.921+0000 7fe0e05fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:44:23.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:44:23.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:23.198 INFO:teuthology.orchestra.run.smithi079.stdout:client24851 2024-04-22T06:44:23.199 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:44:23.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:44:23.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:23.255 INFO:teuthology.orchestra.run.smithi079.stdout:client24851 2024-04-22T06:44:23.256 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/146994a8-0072-11ef-bc93-c7b262605968.client24851/status') 2024-04-22T06:44:23.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:23.313 INFO:teuthology.orchestra.run.smithi079.stdout:instance: client.24851 (3)172.21.15.79:0/2094839437 2024-04-22T06:44:23.313 INFO:teuthology.orchestra.run.smithi079.stdout:blocklisted: false 2024-04-22T06:44:23.313 INFO:teuthology.orchestra.run.smithi079.stderr:0+1 records in 2024-04-22T06:44:23.313 INFO:teuthology.orchestra.run.smithi079.stderr:0+1 records out 2024-04-22T06:44:23.314 INFO:teuthology.orchestra.run.smithi079.stderr:71 bytes copied, 2.227e-05 s, 3.2 MB/s 2024-04-22T06:44:23.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:23.321 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T06:44:23.331 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:44:23.331 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:44:23.331 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-22T06:44:23.351 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:23.072+0000 7f9cec857200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:44:23.351 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:23.170+0000 7f9cec857200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- 172.21.15.79:0/3936192925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 msgr2=0x7f4c981012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/3936192925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c981012c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- 172.21.15.79:0/3936192925 shutdown_connections 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/3936192925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98102a40 0x7f4c98109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/3936192925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98102080 0x7f4c98102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/3936192925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c981012c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- 172.21.15.79:0/3936192925 >> 172.21.15.79:0/3936192925 conn(0x7f4c980fc460 msgr2=0x7f4c980fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- 172.21.15.79:0/3936192925 shutdown_connections 2024-04-22T06:44:23.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- 172.21.15.79:0/3936192925 wait complete. 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 Processor -- start 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- start start 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c9806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98102080 0x7f4c9819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98102a40 0x7f4c9819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9810ca40 con 0x7f4c98100ea0 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9810c8c0 con 0x7f4c98102080 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.434+0000 7f4c9efbd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9810cbc0 con 0x7f4c98102a40 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c9806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c9806ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32858/0 (socket says 172.21.15.79:32858) 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 -- 172.21.15.79:0/668807793 learned_addr learned my addr 172.21.15.79:0/668807793 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c97fff640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98102080 0x7f4c9819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9d55a640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98102a40 0x7f4c9819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 -- 172.21.15.79:0/668807793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98102a40 msgr2=0x7f4c9819f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98102a40 0x7f4c9819f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 -- 172.21.15.79:0/668807793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98102080 msgr2=0x7f4c9819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98102080 0x7f4c9819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 -- 172.21.15.79:0/668807793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9819fdc0 con 0x7f4c98100ea0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9cd59640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c9806ad20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c000c00 tx=0x7f4c8c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c043020 con 0x7f4c98100ea0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c981a0050 con 0x7f4c98100ea0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c004350 con 0x7f4c98100ea0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.435+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c003b80 con 0x7f4c98100ea0 2024-04-22T06:44:23.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.436+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c981a05a0 con 0x7f4c98100ea0 2024-04-22T06:44:23.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.437+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c98108240 con 0x7f4c98100ea0 2024-04-22T06:44:23.438 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:23.098+0000 7f450bdf1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:44:23.438 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:23.229+0000 7f450bdf1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:44:23.438 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:23.324+0000 7f450bdf1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:44:23.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.438+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f4c8c046020 con 0x7f4c98100ea0 2024-04-22T06:44:23.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.439+0000 7f4c95ffb640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 0x7f4c880a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.439+0000 7f4c97fff640 1 -- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 msgr2=0x7f4c880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:23.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.439+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f4c8c0f29d0 con 0x7f4c98100ea0 2024-04-22T06:44:23.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.439+0000 7f4c97fff640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 0x7f4c880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:23.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.442+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c8c04d020 con 0x7f4c98100ea0 2024-04-22T06:44:23.452 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:23.164+0000 7fe0e05fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:44:23.452 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:23.298+0000 7fe0e05fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:44:23.452 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:23.394+0000 7fe0e05fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:44:23.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.640+0000 7f4c97fff640 1 -- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 msgr2=0x7f4c880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:23.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.640+0000 7f4c97fff640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 0x7f4c880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:23.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.665+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f4c9802d5c0 con 0x7f4c98100ea0 2024-04-22T06:44:23.673 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:23.435+0000 7f9cec857200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:44:23.673 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:23.574+0000 7f9cec857200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:44:23.673 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:23.674+0000 7f9cec857200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:44:23.689 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:23.438+0000 7f450bdf1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:23.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.717+0000 7f4c95ffb640 1 -- 172.21.15.79:0/668807793 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7f4c8c0bf3c0 con 0x7f4c98100ea0 2024-04-22T06:44:23.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 msgr2=0x7f4c880a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 0x7f4c880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 msgr2=0x7f4c9806ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c9806ad20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c000c00 tx=0x7f4c8c005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 shutdown_connections 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f4c880a3ca0 0x7f4c880a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98102080 0x7f4c9819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98102a40 0x7f4c9819f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 --2- 172.21.15.79:0/668807793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98100ea0 0x7f4c9806ad20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 >> 172.21.15.79:0/668807793 conn(0x7f4c980fc460 msgr2=0x7f4c98105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 shutdown_connections 2024-04-22T06:44:23.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.719+0000 7f4c9efbd640 1 -- 172.21.15.79:0/668807793 wait complete. 2024-04-22T06:44:23.729 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-22T06:44:23.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:23 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/668807793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:23.819 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:23.511+0000 7fe0e05fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/1433073027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 msgr2=0x7f5f94101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 --2- 172.21.15.79:0/1433073027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f94101280 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009900 tx=0x7f5f8402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/1433073027 shutdown_connections 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 --2- 172.21.15.79:0/1433073027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f94102040 0x7f5f941024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 --2- 172.21.15.79:0/1433073027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f94102a00 0x7f5f94109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 --2- 172.21.15.79:0/1433073027 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f94101280 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/1433073027 >> 172.21.15.79:0/1433073027 conn(0x7f5f940fc420 msgr2=0x7f5f940fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/1433073027 shutdown_connections 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.873+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/1433073027 wait complete. 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 Processor -- start 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 -- start start 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f941981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f94102040 0x7f5f94198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f94102a00 0x7f5f9419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9410c9f0 con 0x7f5f94100e60 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9410c870 con 0x7f5f94102040 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f9a3e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9410cb70 con 0x7f5f94102a00 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f93fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f941981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f93fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f941981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32886/0 (socket says 172.21.15.79:32886) 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f93fff640 1 -- 172.21.15.79:0/876348494 learned_addr learned my addr 172.21.15.79:0/876348494 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:23.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f937fe640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f94102040 0x7f5f94198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.874+0000 7f5f98983640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f94102a00 0x7f5f9419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f93fff640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f94102a00 msgr2=0x7f5f9419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f93fff640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f94102a00 0x7f5f9419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f93fff640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f94102040 msgr2=0x7f5f94198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f93fff640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f94102040 0x7f5f94198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f93fff640 1 -- 172.21.15.79:0/876348494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9419fdc0 con 0x7f5f94100e60 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f93fff640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f941981f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5f84000c00 tx=0x7f5f84005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f84043020 con 0x7f5f94100e60 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f941a0050 con 0x7f5f94100e60 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f84004350 con 0x7f5f94100e60 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f84003b80 con 0x7f5f94100e60 2024-04-22T06:44:23.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.875+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f941a05f0 con 0x7f5f94100e60 2024-04-22T06:44:23.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.876+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f941a07f0 con 0x7f5f94100e60 2024-04-22T06:44:23.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.877+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f5f84046020 con 0x7f5f94100e60 2024-04-22T06:44:23.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.877+0000 7f5f917fa640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 0x7f5f7c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.878+0000 7f5f937fe640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 msgr2=0x7f5f7c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:23.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.878+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8211+0+0 (secure 0 0 0) 0x7f5f840f2550 con 0x7f5f94100e60 2024-04-22T06:44:23.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.878+0000 7f5f937fe640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 0x7f5f7c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:23.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:23.882+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f8404d020 con 0x7f5f94100e60 2024-04-22T06:44:23.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/668807793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:23.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:23.798+0000 7f9cec857200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:24.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.079+0000 7f5f937fe640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 msgr2=0x7f5f7c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:24.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.079+0000 7f5f937fe640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 0x7f5f7c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:24.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.089+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 --> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f5f940fe6f0 con 0x7f5f7c0a3ca0 2024-04-22T06:44:24.189 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:23.762+0000 7f450bdf1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:44:24.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:23 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/668807793' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:24.202 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:23.820+0000 7fe0e05fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:44:24.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.480+0000 7f5f937fe640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 msgr2=0x7f5f7c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3007546422 2024-04-22T06:44:24.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.480+0000 7f5f937fe640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 0x7f5f7c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:44:24.489 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:24.107+0000 7f9cec857200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:44:24.689 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:24.369+0000 7f450bdf1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:44:24.689 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z[33682]: 2024-04-22T06:44:24.587+0000 7f450bdf1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:44:24.702 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:24.425+0000 7fe0e05fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:44:24.703 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y[32801]: 2024-04-22T06:44:24.640+0000 7fe0e05fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:44:24.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.738+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f5f84005460 con 0x7f5f94100e60 2024-04-22T06:44:24.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.953+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f5f840baeb0 con 0x7f5f94100e60 2024-04-22T06:44:24.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.953+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 msgr2=0x7f5f7c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:24.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:24.953+0000 7f5f917fa640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/3007546422,v1:172.21.15.79:6801/3007546422] conn(0x7f5f7c0a3ca0 0x7f5f7c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:24.716+0000 7f9cec857200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:24.941+0000 7f9cec857200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/668807793' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: Standby manager daemon z restarted 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: Standby manager daemon z started 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:44:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:24.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: Standby manager daemon y restarted 2024-04-22T06:44:24.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: Standby manager daemon y started 2024-04-22T06:44:24.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:44:24.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:24.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:44:24.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:24 smithi079 ceph-mon[27946]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:25.095 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/668807793' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: Standby manager daemon z restarted 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: Standby manager daemon z started 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: Standby manager daemon y restarted 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: Standby manager daemon y started 2024-04-22T06:44:25.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:24 smithi190 ceph-mon[31421]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/668807793' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: Standby manager daemon z restarted 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: Standby manager daemon z started 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:44:25.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.193:0/4162718192' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: Standby manager daemon y restarted 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: Standby manager daemon y started 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:44:25.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:24 smithi193 ceph-mon[31815]: from='mgr.? 172.21.15.190:0/576588018' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:25.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4e990700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4e990700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4e990700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4e990700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4e990700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.490 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4a988700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.491 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4a988700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.491 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4a988700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.491 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4a988700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.491 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.187+0000 7f9c4a988700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.959+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mon.0 v2:172.21.15.79:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f5f8409a740 con 0x7f5f94100e60 2024-04-22T06:44:25.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.960+0000 7f5f917fa640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5f7c0352e0 0x7f5f7c0356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:25.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.960+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f5f84043e80 con 0x7f5f7c0352e0 2024-04-22T06:44:25.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.964+0000 7f5f937fe640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5f7c0352e0 0x7f5f7c0356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:25.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.968+0000 7f5f937fe640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5f7c0352e0 0x7f5f7c0356c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5f941996b0 tx=0x7f5f88005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:25.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.989+0000 7f5f917fa640 1 -- 172.21.15.79:0/876348494 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f5f84043e80 con 0x7f5f7c0352e0 2024-04-22T06:44:25.987 INFO:teuthology.orchestra.run.smithi079.stdout:{} 2024-04-22T06:44:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.991+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5f7c0352e0 msgr2=0x7f5f7c0356c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.991+0000 7f5f9a3e6640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5f7c0352e0 0x7f5f7c0356c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5f941996b0 tx=0x7f5f88005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.991+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 msgr2=0x7f5f941981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.991+0000 7f5f9a3e6640 1 --2- 172.21.15.79:0/876348494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5f94100e60 0x7f5f941981f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5f84000c00 tx=0x7f5f84005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.991+0000 7f5f98983640 1 -- 172.21.15.79:0/876348494 reap_dead start 2024-04-22T06:44:25.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.975+0000 7f9c49185700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.975+0000 7f9c49185700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.975+0000 7f9c49185700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.975+0000 7f9c49185700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:44:25.975+0000 7f9c49185700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:25.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.992+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 shutdown_connections 2024-04-22T06:44:25.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.992+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 >> 172.21.15.79:0/876348494 conn(0x7f5f940fc420 msgr2=0x7f5f940fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:25.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.992+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 shutdown_connections 2024-04-22T06:44:25.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:25.992+0000 7f5f9a3e6640 1 -- 172.21.15.79:0/876348494 wait complete. 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: Active manager daemon x restarted 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: Activating manager daemon x 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: mgrmap e21: x(active, starting, since 0.0112417s), standbys: z, y 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:44:25.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:44:25.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:44:25.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: Manager daemon x is now available 2024-04-22T06:44:25.995 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:25.995 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:44:25.995 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:25 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:44:26.002 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-22T06:44:26.013 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-22T06:44:26.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- 172.21.15.79:0/871255426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c1029f0 msgr2=0x7fb53c104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- 172.21.15.79:0/871255426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c1029f0 0x7fb53c104de0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb538009900 tx=0x7fb53802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- 172.21.15.79:0/871255426 shutdown_connections 2024-04-22T06:44:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- 172.21.15.79:0/871255426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c107c50 0x7fb53c10a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- 172.21.15.79:0/871255426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb53c105320 0x7fb53c107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- 172.21.15.79:0/871255426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c1029f0 0x7fb53c104de0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- 172.21.15.79:0/871255426 >> 172.21.15.79:0/871255426 conn(0x7fb53c0fc3e0 msgr2=0x7fb53c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- 172.21.15.79:0/871255426 shutdown_connections 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- 172.21.15.79:0/871255426 wait complete. 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 Processor -- start 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- start start 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c1029f0 0x7fb53c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 0x7fb53c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb53c107c50 0x7fb53c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c10dab0 con 0x7fb53c105320 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c10d930 con 0x7fb53c1029f0 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb543c31640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c10dc30 con 0x7fb53c107c50 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb5411cc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 0x7fb53c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.162+0000 7fb5411cc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 0x7fb53c198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33048/0 (socket says 172.21.15.79:33048) 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5421ce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb53c107c50 0x7fb53c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 -- 172.21.15.79:0/737164557 learned_addr learned my addr 172.21.15.79:0/737164557 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5419cd640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c1029f0 0x7fb53c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 -- 172.21.15.79:0/737164557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb53c107c50 msgr2=0x7fb53c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb53c107c50 0x7fb53c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 -- 172.21.15.79:0/737164557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c1029f0 msgr2=0x7fb53c1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c1029f0 0x7fb53c1981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 -- 172.21.15.79:0/737164557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c19fde0 con 0x7fb53c105320 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb5411cc640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 0x7fb53c198700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00e9f0 tx=0x7fb52c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00ccc0 con 0x7fb53c105320 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb52c0054d0 con 0x7fb53c105320 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c1a00d0 con 0x7fb53c105320 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c010430 con 0x7fb53c105320 2024-04-22T06:44:26.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.163+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c1a05d0 con 0x7fb53c105320 2024-04-22T06:44:26.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.164+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53c0fded0 con 0x7fb53c105320 2024-04-22T06:44:26.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.165+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fb52c0046c0 con 0x7fb53c105320 2024-04-22T06:44:26.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.166+0000 7fb532ffd640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb5280a39b0 0x7fb5280a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.166+0000 7fb5419cd640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb5280a39b0 0x7fb5280a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.166+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fb52c0d0430 con 0x7fb53c105320 2024-04-22T06:44:26.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.166+0000 7fb5419cd640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb5280a39b0 0x7fb5280a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb538009230 tx=0x7fb538005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.167 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-22T06:44:26.167 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: Active manager daemon x restarted 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: Activating manager daemon x 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: mgrmap e21: x(active, starting, since 0.0112417s), standbys: z, y 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:44:26.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: Manager daemon x is now available 2024-04-22T06:44:26.170 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:26.171 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:44:26.171 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:25 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:44:26.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.170+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb52c09d010 con 0x7fb53c105320 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: Active manager daemon x restarted 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: Activating manager daemon x 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: mgrmap e21: x(active, starting, since 0.0112417s), standbys: z, y 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:44:26.174 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:44:26.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:44:26.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:44:26.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:44:26.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:44:26.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: Manager daemon x is now available 2024-04-22T06:44:26.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:26.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:44:26.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:25 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:44:26.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.369+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb53c062b50 con 0x7fb53c105320 2024-04-22T06:44:26.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.371+0000 7fb532ffd640 1 -- 172.21.15.79:0/737164557 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7fb52c09c910 con 0x7fb53c105320 2024-04-22T06:44:26.369 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:26.369 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb5280a39b0 msgr2=0x7fb5280a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb5280a39b0 0x7fb5280a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb538009230 tx=0x7fb538005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 msgr2=0x7fb53c198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 0x7fb53c198700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00e9f0 tx=0x7fb52c00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 shutdown_connections 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb53c107c50 0x7fb53c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb53c105320 0x7fb53c198700 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb5280a39b0 0x7fb5280a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 --2- 172.21.15.79:0/737164557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb53c1029f0 0x7fb53c1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 >> 172.21.15.79:0/737164557 conn(0x7fb53c0fc3e0 msgr2=0x7fb53c1034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 shutdown_connections 2024-04-22T06:44:26.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.373+0000 7fb543c31640 1 -- 172.21.15.79:0/737164557 wait complete. 2024-04-22T06:44:26.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:26.383 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 -- 172.21.15.79:0/3640256990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 msgr2=0x7f0c70107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3640256990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70107810 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009900 tx=0x7f0c5c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 -- 172.21.15.79:0/3640256990 shutdown_connections 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3640256990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c70107d50 0x7f0c7010a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3640256990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70107810 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3640256990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c70102af0 0x7f0c70104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 -- 172.21.15.79:0/3640256990 >> 172.21.15.79:0/3640256990 conn(0x7f0c700fc420 msgr2=0x7f0c700fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 -- 172.21.15.79:0/3640256990 shutdown_connections 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.487+0000 7f0c7717e640 1 -- 172.21.15.79:0/3640256990 wait complete. 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 Processor -- start 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 -- start start 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c70102af0 0x7f0c70198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c70107d50 0x7f0c7019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7010db00 con 0x7f0c70105420 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7010d980 con 0x7f0c70107d50 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7717e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7010dc80 con 0x7f0c70102af0 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33072/0 (socket says 172.21.15.79:33072) 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 -- 172.21.15.79:0/3090568670 learned_addr learned my addr 172.21.15.79:0/3090568670 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:26.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c7571b640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c70107d50 0x7f0c7019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c74f1a640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c70102af0 0x7f0c70198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 -- 172.21.15.79:0/3090568670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c70102af0 msgr2=0x7f0c70198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c70102af0 0x7f0c70198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 -- 172.21.15.79:0/3090568670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c70107d50 msgr2=0x7f0c7019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c70107d50 0x7f0c7019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.488+0000 7f0c6ffff640 1 -- 172.21.15.79:0/3090568670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7019fe60 con 0x7f0c70105420 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.489+0000 7f0c6ffff640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70198780 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c000c00 tx=0x7f0c5c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.489+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c043020 con 0x7f0c70105420 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.489+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c5c004aa0 con 0x7f0c70105420 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.489+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c003d00 con 0x7f0c70105420 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.489+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c701a00f0 con 0x7f0c70105420 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.489+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c701a05f0 con 0x7f0c70105420 2024-04-22T06:44:26.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.490+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c700fdf10 con 0x7f0c70105420 2024-04-22T06:44:26.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.491+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f0c5c046020 con 0x7f0c70105420 2024-04-22T06:44:26.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.491+0000 7f0c6dffb640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c600a39b0 0x7f0c600a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.492+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f0c5c0f2690 con 0x7f0c70105420 2024-04-22T06:44:26.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.492+0000 7f0c74f1a640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c600a39b0 0x7f0c600a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.492+0000 7f0c74f1a640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c600a39b0 0x7f0c600a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0c64005fd0 tx=0x7f0c64005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.496+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c5c04d020 con 0x7f0c70105420 2024-04-22T06:44:26.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.680+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0c70062b80 con 0x7f0c70105420 2024-04-22T06:44:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.680+0000 7f0c6dffb640 1 -- 172.21.15.79:0/3090568670 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f0c5c0bf270 con 0x7f0c70105420 2024-04-22T06:44:26.679 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c600a39b0 msgr2=0x7f0c600a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c600a39b0 0x7f0c600a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0c64005fd0 tx=0x7f0c64005850 comp rx=0 tx=0).stop 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 msgr2=0x7f0c70198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70198780 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c000c00 tx=0x7f0c5c004270 comp rx=0 tx=0).stop 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 shutdown_connections 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c600a39b0 0x7f0c600a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c70107d50 0x7f0c7019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c70105420 0x7f0c70198780 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 --2- 172.21.15.79:0/3090568670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c70102af0 0x7f0c70198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 >> 172.21.15.79:0/3090568670 conn(0x7f0c700fc420 msgr2=0x7f0c70103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 shutdown_connections 2024-04-22T06:44:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.683+0000 7f0c7717e640 1 -- 172.21.15.79:0/3090568670 wait complete. 2024-04-22T06:44:26.695 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 -- 172.21.15.79:0/672914303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 msgr2=0x7f77f81093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 --2- 172.21.15.79:0/672914303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f81093a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009cd0 tx=0x7f77f002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 -- 172.21.15.79:0/672914303 shutdown_connections 2024-04-22T06:44:26.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 --2- 172.21.15.79:0/672914303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77f8101140 0x7f77f81015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 --2- 172.21.15.79:0/672914303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f81093a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 --2- 172.21.15.79:0/672914303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f81024a0 0x7f77f81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 -- 172.21.15.79:0/672914303 >> 172.21.15.79:0/672914303 conn(0x7f77f80fbb90 msgr2=0x7f77f80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 -- 172.21.15.79:0/672914303 shutdown_connections 2024-04-22T06:44:26.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 -- 172.21.15.79:0/672914303 wait complete. 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.813+0000 7f77fced2640 1 Processor -- start 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 -- start start 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f8101140 0x7f77f8104ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f8103530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77f81024a0 0x7f77f8103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f77f810c9f0 con 0x7f77f8101b00 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f77f810c870 con 0x7f77f81024a0 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77fced2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f77f810cb70 con 0x7f77f8101140 2024-04-22T06:44:26.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f8103530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f8103530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33084/0 (socket says 172.21.15.79:33084) 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 -- 172.21.15.79:0/2516653263 learned_addr learned my addr 172.21.15.79:0/2516653263 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77f659c640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f8101140 0x7f77f8104ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77f6d9d640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77f81024a0 0x7f77f8103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 -- 172.21.15.79:0/2516653263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f8101140 msgr2=0x7f77f8104ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f8101140 0x7f77f8104ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 -- 172.21.15.79:0/2516653263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77f81024a0 msgr2=0x7f77f8103a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77f81024a0 0x7f77f8103a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 -- 172.21.15.79:0/2516653263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f8104300 con 0x7f77f8101b00 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77effff640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f8103530 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f77e000e9f0 tx=0x7f77e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.814+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e000cd80 con 0x7f77f8101b00 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.815+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f81a22d0 con 0x7f77f8101b00 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.815+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77e00108e0 con 0x7f77f8101b00 2024-04-22T06:44:26.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.815+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0018980 con 0x7f77f8101b00 2024-04-22T06:44:26.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.815+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f81a2870 con 0x7f77f8101b00 2024-04-22T06:44:26.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.816+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77f81a2b30 con 0x7f77f8101b00 2024-04-22T06:44:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.817+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f77e0010430 con 0x7f77f8101b00 2024-04-22T06:44:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.817+0000 7f77ef7fe640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e40a39b0 0x7f77e40a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.817+0000 7f77f659c640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e40a39b0 0x7f77e40a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.817+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f77e00d03e0 con 0x7f77f8101b00 2024-04-22T06:44:26.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.818+0000 7f77f659c640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e40a39b0 0x7f77e40a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77f806a310 tx=0x7f77e8005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.822+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77e009c970 con 0x7f77f8101b00 2024-04-22T06:44:26.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:26.975+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f77e0098fe0 con 0x7f77f8101b00 2024-04-22T06:44:27.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.049+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77f81a2d00 con 0x7f77f8101b00 2024-04-22T06:44:27.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.050+0000 7f77ef7fe640 1 -- 172.21.15.79:0/2516653263 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f77e0092910 con 0x7f77f8101b00 2024-04-22T06:44:27.048 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e40a39b0 msgr2=0x7f77e40a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e40a39b0 0x7f77e40a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77f806a310 tx=0x7f77e8005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 msgr2=0x7f77f8103530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f8103530 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f77e000e9f0 tx=0x7f77e000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 shutdown_connections 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e40a39b0 0x7f77e40a5e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77f81024a0 0x7f77f8103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77f8101b00 0x7f77f8103530 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 --2- 172.21.15.79:0/2516653263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77f8101140 0x7f77f8104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 >> 172.21.15.79:0/2516653263 conn(0x7f77f80fbb90 msgr2=0x7f77f80fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:27.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 shutdown_connections 2024-04-22T06:44:27.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.052+0000 7f77fced2640 1 -- 172.21.15.79:0/2516653263 wait complete. 2024-04-22T06:44:27.064 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.176+0000 7f12102eb640 1 -- 172.21.15.79:0/1105604112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1208100e90 msgr2=0x7f12081012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.176+0000 7f12102eb640 1 --2- 172.21.15.79:0/1105604112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1208100e90 0x7f12081012b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1204009900 tx=0x7f120402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 -- 172.21.15.79:0/1105604112 shutdown_connections 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 --2- 172.21.15.79:0/1105604112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1208101fe0 0x7f1208102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 --2- 172.21.15.79:0/1105604112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12081029a0 0x7f12081092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 --2- 172.21.15.79:0/1105604112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1208100e90 0x7f12081012b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 -- 172.21.15.79:0/1105604112 >> 172.21.15.79:0/1105604112 conn(0x7f12080fc400 msgr2=0x7f12080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 -- 172.21.15.79:0/1105604112 shutdown_connections 2024-04-22T06:44:27.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 -- 172.21.15.79:0/1105604112 wait complete. 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 Processor -- start 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 -- start start 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1208100e90 0x7f12081981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1208101fe0 0x7f1208198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.177+0000 7f12102eb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 0x7f120819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f12102eb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f120810ca10 con 0x7f12081029a0 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f12102eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f120810c890 con 0x7f1208101fe0 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f12102eb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f120810cb90 con 0x7f1208100e90 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 0x7f120819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 0x7f120819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33140/0 (socket says 172.21.15.79:33140) 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 -- 172.21.15.79:0/1625007237 learned_addr learned my addr 172.21.15.79:0/1625007237 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e087640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1208100e90 0x7f12081981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 -- 172.21.15.79:0/1625007237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1208100e90 msgr2=0x7f12081981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1208100e90 0x7f12081981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 -- 172.21.15.79:0/1625007237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1208101fe0 msgr2=0x7f1208198730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1208101fe0 0x7f1208198730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 -- 172.21.15.79:0/1625007237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120819fe10 con 0x7f12081029a0 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f120e888640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 0x7f120819f7a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1200009fd0 tx=0x7f120000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200017070 con 0x7f12081029a0 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12081a0100 con 0x7f12081029a0 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1200003e20 con 0x7f12081029a0 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200010460 con 0x7f12081029a0 2024-04-22T06:44:27.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.178+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12081a0650 con 0x7f12081029a0 2024-04-22T06:44:27.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.180+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12081a08b0 con 0x7f12081029a0 2024-04-22T06:44:27.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.181+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f1200003970 con 0x7f12081029a0 2024-04-22T06:44:27.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.182+0000 7f11ff7fe640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f11f40a3db0 0x7f11f40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.182+0000 7f120e087640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f11f40a3db0 0x7f11f40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.182+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f12000d0640 con 0x7f12081029a0 2024-04-22T06:44:27.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.183+0000 7f120e087640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f11f40a3db0 0x7f11f40a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1204000c00 tx=0x7f120402c7e0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:27.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.186+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f120009cb80 con 0x7f12081029a0 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: mgrmap e22: x(active, since 1.02423s), standbys: z, y 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: [22/Apr/2024:06:44:26] ENGINE Bus STARTING 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: [22/Apr/2024:06:44:26] ENGINE Serving on https://172.21.15.79:7150 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: [22/Apr/2024:06:44:26] ENGINE Serving on http://172.21.15.79:8765 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: [22/Apr/2024:06:44:26] ENGINE Bus STARTED 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/737164557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:26 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3090568670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:27.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.375+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f12081a0a80 con 0x7f12081029a0 2024-04-22T06:44:27.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.376+0000 7f11ff7fe640 1 -- 172.21.15.79:0/1625007237 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f120009c480 con 0x7f12081029a0 2024-04-22T06:44:27.374 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f11f40a3db0 msgr2=0x7f11f40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f11f40a3db0 0x7f11f40a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1204000c00 tx=0x7f120402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 msgr2=0x7f120819f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 0x7f120819f7a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1200009fd0 tx=0x7f120000b260 comp rx=0 tx=0).stop 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 shutdown_connections 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f11f40a3db0 0x7f11f40a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1208101fe0 0x7f1208198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f12081029a0 0x7f120819f7a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 --2- 172.21.15.79:0/1625007237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1208100e90 0x7f12081981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 >> 172.21.15.79:0/1625007237 conn(0x7f12080fc400 msgr2=0x7f12080fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 shutdown_connections 2024-04-22T06:44:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.378+0000 7f12102eb640 1 -- 172.21.15.79:0/1625007237 wait complete. 2024-04-22T06:44:27.388 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:27.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: mgrmap e22: x(active, since 1.02423s), standbys: z, y 2024-04-22T06:44:27.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: [22/Apr/2024:06:44:26] ENGINE Bus STARTING 2024-04-22T06:44:27.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: [22/Apr/2024:06:44:26] ENGINE Serving on https://172.21.15.79:7150 2024-04-22T06:44:27.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: [22/Apr/2024:06:44:26] ENGINE Serving on http://172.21.15.79:8765 2024-04-22T06:44:27.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: [22/Apr/2024:06:44:26] ENGINE Bus STARTED 2024-04-22T06:44:27.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/737164557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:27.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:26 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3090568670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: mgrmap e22: x(active, since 1.02423s), standbys: z, y 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: [22/Apr/2024:06:44:26] ENGINE Bus STARTING 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: [22/Apr/2024:06:44:26] ENGINE Serving on https://172.21.15.79:7150 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: [22/Apr/2024:06:44:26] ENGINE Serving on http://172.21.15.79:8765 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: [22/Apr/2024:06:44:26] ENGINE Bus STARTED 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/737164557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:27.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:26 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3090568670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 -- 172.21.15.79:0/2456517924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 msgr2=0x7fbf2c102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 --2- 172.21.15.79:0/2456517924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c102440 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009900 tx=0x7fbf2002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 -- 172.21.15.79:0/2456517924 shutdown_connections 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 --2- 172.21.15.79:0/2456517924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf2c102980 0x7fbf2c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 --2- 172.21.15.79:0/2456517924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c102440 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 --2- 172.21.15.79:0/2456517924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf2c100e70 0x7fbf2c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 -- 172.21.15.79:0/2456517924 >> 172.21.15.79:0/2456517924 conn(0x7fbf2c0fc3e0 msgr2=0x7fbf2c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 -- 172.21.15.79:0/2456517924 shutdown_connections 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.490+0000 7fbf329b1640 1 -- 172.21.15.79:0/2456517924 wait complete. 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 Processor -- start 2024-04-22T06:44:27.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 -- start start 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf2c100e70 0x7fbf2c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf2c102980 0x7fbf2c1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c10c9d0 con 0x7fbf2c101fc0 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c10c850 con 0x7fbf2c100e70 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf329b1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c10cb50 con 0x7fbf2c102980 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf2b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf2b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c19a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33152/0 (socket says 172.21.15.79:33152) 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf2b7fe640 1 -- 172.21.15.79:0/279678836 learned_addr learned my addr 172.21.15.79:0/279678836 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf2bfff640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf2c100e70 0x7fbf2c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf30f4e640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf2c102980 0x7fbf2c1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf2b7fe640 1 -- 172.21.15.79:0/279678836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf2c102980 msgr2=0x7fbf2c1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.491+0000 7fbf2b7fe640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf2c102980 0x7fbf2c1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf2b7fe640 1 -- 172.21.15.79:0/279678836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf2c100e70 msgr2=0x7fbf2c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf2b7fe640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf2c100e70 0x7fbf2c19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf2b7fe640 1 -- 172.21.15.79:0/279678836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2c1a2020 con 0x7fbf2c101fc0 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf2b7fe640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c19a940 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbf20000c00 tx=0x7fbf20005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20043020 con 0x7fbf2c101fc0 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf20004350 con 0x7fbf2c101fc0 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2c1a22b0 con 0x7fbf2c101fc0 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20003b80 con 0x7fbf2c101fc0 2024-04-22T06:44:27.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.492+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2c1a2800 con 0x7fbf2c101fc0 2024-04-22T06:44:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.493+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf2c1a2a00 con 0x7fbf2c101fc0 2024-04-22T06:44:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.494+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fbf20046020 con 0x7fbf2c101fc0 2024-04-22T06:44:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.495+0000 7fbf297fa640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf140a3db0 0x7fbf140a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.495+0000 7fbf2bfff640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf140a3db0 0x7fbf140a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.495+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fbf200f2970 con 0x7fbf2c101fc0 2024-04-22T06:44:27.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.495+0000 7fbf2bfff640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf140a3db0 0x7fbf140a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009bd0 tx=0x7fbf1c005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:27.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.500+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf2004d020 con 0x7fbf2c101fc0 2024-04-22T06:44:27.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.687+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf2c1a2c70 con 0x7fbf2c101fc0 2024-04-22T06:44:27.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.688+0000 7fbf297fa640 1 -- 172.21.15.79:0/279678836 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7fbf200a7e20 con 0x7fbf2c101fc0 2024-04-22T06:44:27.687 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:27.687 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf140a3db0 msgr2=0x7fbf140a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf140a3db0 0x7fbf140a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009bd0 tx=0x7fbf1c005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 msgr2=0x7fbf2c19a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c19a940 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbf20000c00 tx=0x7fbf20005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 shutdown_connections 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf2c102980 0x7fbf2c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbf2c101fc0 0x7fbf2c19a940 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf140a3db0 0x7fbf140a6270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 --2- 172.21.15.79:0/279678836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf2c100e70 0x7fbf2c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 >> 172.21.15.79:0/279678836 conn(0x7fbf2c0fc3e0 msgr2=0x7fbf2c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 shutdown_connections 2024-04-22T06:44:27.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.690+0000 7fbf329b1640 1 -- 172.21.15.79:0/279678836 wait complete. 2024-04-22T06:44:27.705 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:27.705 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:27.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.815+0000 7f3253bb6640 1 -- 172.21.15.79:0/1446326551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102a00 msgr2=0x7f324c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.815+0000 7f3253bb6640 1 --2- 172.21.15.79:0/1446326551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102a00 0x7f324c109320 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3244009ca0 tx=0x7f324402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 -- 172.21.15.79:0/1446326551 shutdown_connections 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 --2- 172.21.15.79:0/1446326551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102a00 0x7f324c109320 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 --2- 172.21.15.79:0/1446326551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c102040 0x7f324c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 --2- 172.21.15.79:0/1446326551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c100e60 0x7f324c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 -- 172.21.15.79:0/1446326551 >> 172.21.15.79:0/1446326551 conn(0x7f324c0fc420 msgr2=0x7f324c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 -- 172.21.15.79:0/1446326551 shutdown_connections 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 -- 172.21.15.79:0/1446326551 wait complete. 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 Processor -- start 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.816+0000 7f3253bb6640 1 -- start start 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3253bb6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c100e60 0x7f324c1970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3253bb6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 0x7f324c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3253bb6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c102a00 0x7f324c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3253bb6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f324c10ca00 con 0x7f324c102040 2024-04-22T06:44:27.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3253bb6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f324c10c880 con 0x7f324c102a00 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3253bb6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f324c10cb80 con 0x7f324c100e60 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 0x7f324c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 0x7f324c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33164/0 (socket says 172.21.15.79:33164) 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 -- 172.21.15.79:0/2793382726 learned_addr learned my addr 172.21.15.79:0/2793382726 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251952640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c100e60 0x7f324c1970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3252153640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c102a00 0x7f324c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 -- 172.21.15.79:0/2793382726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c100e60 msgr2=0x7f324c1970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c100e60 0x7f324c1970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 -- 172.21.15.79:0/2793382726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c102a00 msgr2=0x7f324c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c102a00 0x7f324c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 -- 172.21.15.79:0/2793382726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c06a550 con 0x7f324c102040 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251952640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c100e60 0x7f324c1970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3252153640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c102a00 0x7f324c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:27.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3251151640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 0x7f324c1975e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f323c00e9f0 tx=0x7f323c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.817+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c00cd80 con 0x7f324c102040 2024-04-22T06:44:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.818+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f323c0054d0 con 0x7f324c102040 2024-04-22T06:44:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.818+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324c06a840 con 0x7f324c102040 2024-04-22T06:44:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.818+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c010690 con 0x7f324c102040 2024-04-22T06:44:27.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.818+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324c06ad40 con 0x7f324c102040 2024-04-22T06:44:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.819+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f324c108220 con 0x7f324c102040 2024-04-22T06:44:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.820+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f323c004e90 con 0x7f324c102040 2024-04-22T06:44:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.820+0000 7f3242ffd640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f32380a3db0 0x7f32380a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.821+0000 7f3251952640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f32380a3db0 0x7f32380a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.821+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f323c0d0d70 con 0x7f324c102040 2024-04-22T06:44:27.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.821+0000 7f3251952640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f32380a3db0 0x7f32380a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f324c101bb0 tx=0x7f3248005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:27.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:27.825+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f323c09d2d0 con 0x7f324c102040 2024-04-22T06:44:28.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.008+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f324c02d770 con 0x7f324c102040 2024-04-22T06:44:28.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.008+0000 7f3242ffd640 1 -- 172.21.15.79:0/2793382726 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f323c09cbd0 con 0x7f324c102040 2024-04-22T06:44:28.007 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:28.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f32380a3db0 msgr2=0x7f32380a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f32380a3db0 0x7f32380a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f324c101bb0 tx=0x7f3248005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 msgr2=0x7f324c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 0x7f324c1975e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f323c00e9f0 tx=0x7f323c00edf0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 shutdown_connections 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f32380a3db0 0x7f32380a6270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f324c102a00 0x7f324c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f324c102040 0x7f324c1975e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 --2- 172.21.15.79:0/2793382726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f324c100e60 0x7f324c1970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.010+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 >> 172.21.15.79:0/2793382726 conn(0x7f324c0fc420 msgr2=0x7f324c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.011+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 shutdown_connections 2024-04-22T06:44:28.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.011+0000 7f3253bb6640 1 -- 172.21.15.79:0/2793382726 wait complete. 2024-04-22T06:44:28.020 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:28.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 -- 172.21.15.79:0/2119201230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4101fe0 msgr2=0x7f56c4102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/2119201230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4101fe0 0x7f56c4102460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009860 tx=0x7f56c002bb70 comp rx=0 tx=0).stop 2024-04-22T06:44:28.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 -- 172.21.15.79:0/2119201230 shutdown_connections 2024-04-22T06:44:28.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/2119201230 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4101fe0 0x7f56c4102460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/2119201230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c41029a0 0x7f56c41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/2119201230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c4100e90 0x7f56c41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 -- 172.21.15.79:0/2119201230 >> 172.21.15.79:0/2119201230 conn(0x7f56c40fc400 msgr2=0x7f56c40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:28.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 -- 172.21.15.79:0/2119201230 shutdown_connections 2024-04-22T06:44:28.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.127+0000 7f56cbfbd640 1 -- 172.21.15.79:0/2119201230 wait complete. 2024-04-22T06:44:28.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 Processor -- start 2024-04-22T06:44:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 -- start start 2024-04-22T06:44:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4100e90 0x7f56c4197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c4101fe0 0x7f56c41976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c41029a0 0x7f56c419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f56c410c9a0 con 0x7f56c4101fe0 2024-04-22T06:44:28.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56c410c820 con 0x7f56c4100e90 2024-04-22T06:44:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56cbfbd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56c410cb20 con 0x7f56c41029a0 2024-04-22T06:44:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9558640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c4101fe0 0x7f56c41976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9d59640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4100e90 0x7f56c4197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9558640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c4101fe0 0x7f56c41976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33168/0 (socket says 172.21.15.79:33168) 2024-04-22T06:44:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9558640 1 -- 172.21.15.79:0/1313800874 learned_addr learned my addr 172.21.15.79:0/1313800874 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:28.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56ca55a640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c41029a0 0x7f56c419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9d59640 1 -- 172.21.15.79:0/1313800874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c41029a0 msgr2=0x7f56c419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9d59640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c41029a0 0x7f56c419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9d59640 1 -- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c4101fe0 msgr2=0x7f56c41976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9d59640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c4101fe0 0x7f56c41976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.128+0000 7f56c9d59640 1 -- 172.21.15.79:0/1313800874 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c419ed80 con 0x7f56c4100e90 2024-04-22T06:44:28.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.129+0000 7f56c9d59640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4100e90 0x7f56c4197160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56b400d720 tx=0x7f56b400dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.129+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b4017070 con 0x7f56c4100e90 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.129+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56b400de30 con 0x7f56c4100e90 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.129+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b40047a0 con 0x7f56c4100e90 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.129+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c419f070 con 0x7f56c4100e90 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.129+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c419f5c0 con 0x7f56c4100e90 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.130+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c4108330 con 0x7f56c4100e90 2024-04-22T06:44:28.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.132+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f56b40099d0 con 0x7f56c4100e90 2024-04-22T06:44:28.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.132+0000 7f56baffd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b00a3e30 0x7f56b00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.132+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f56b40d0dd0 con 0x7f56c4100e90 2024-04-22T06:44:28.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.132+0000 7f56c9558640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b00a3e30 0x7f56b00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.133+0000 7f56c9558640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b00a3e30 0x7f56b00a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56c4198670 tx=0x7f56c00349b0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:28.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.135+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56b4020020 con 0x7f56c4100e90 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2516653263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1625007237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:28.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/279678836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:28.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.327+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56c4062b50 con 0x7f56c4100e90 2024-04-22T06:44:28.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.328+0000 7f56baffd640 1 -- 172.21.15.79:0/1313800874 <== mon.2 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f56b409d3b0 con 0x7f56c4100e90 2024-04-22T06:44:28.326 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:28.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b00a3e30 msgr2=0x7f56b00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b00a3e30 0x7f56b00a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56c4198670 tx=0x7f56c00349b0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4100e90 msgr2=0x7f56c4197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4100e90 0x7f56c4197160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56b400d720 tx=0x7f56b400dbe0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 shutdown_connections 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c41029a0 0x7f56c419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c4101fe0 0x7f56c41976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b00a3e30 0x7f56b00a62f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 --2- 172.21.15.79:0/1313800874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c4100e90 0x7f56c4197160 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 >> 172.21.15.79:0/1313800874 conn(0x7f56c40fc400 msgr2=0x7f56c4105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:28.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 shutdown_connections 2024-04-22T06:44:28.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.330+0000 7f56cbfbd640 1 -- 172.21.15.79:0/1313800874 wait complete. 2024-04-22T06:44:28.340 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:28.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2516653263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1625007237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:28.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/279678836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/1635583827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f8101fe0 msgr2=0x7ff3f8102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/1635583827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f8101fe0 0x7ff3f8102460 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009900 tx=0x7ff3e802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/1635583827 shutdown_connections 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/1635583827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f81029a0 0x7ff3f81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/1635583827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f8101fe0 0x7ff3f8102460 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/1635583827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3f8100e90 0x7ff3f81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/1635583827 >> 172.21.15.79:0/1635583827 conn(0x7ff3f80fc400 msgr2=0x7ff3f80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/1635583827 shutdown_connections 2024-04-22T06:44:28.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.445+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/1635583827 wait complete. 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 Processor -- start 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 -- start start 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100e90 0x7ff3f81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3f8101fe0 0x7ff3f8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 0x7ff3f819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810c9d0 con 0x7ff3f81029a0 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810c850 con 0x7ff3f8100e90 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3ff9c7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810cb50 con 0x7ff3f8101fe0 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fdf64640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 0x7ff3f819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fdf64640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 0x7ff3f819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33192/0 (socket says 172.21.15.79:33192) 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fdf64640 1 -- 172.21.15.79:0/2193154763 learned_addr learned my addr 172.21.15.79:0/2193154763 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fd763640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100e90 0x7ff3f81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fcf62640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3f8101fe0 0x7ff3f8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fdf64640 1 -- 172.21.15.79:0/2193154763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3f8101fe0 msgr2=0x7ff3f8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.446+0000 7ff3fdf64640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3f8101fe0 0x7ff3f8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3fdf64640 1 -- 172.21.15.79:0/2193154763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100e90 msgr2=0x7ff3f81981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3fdf64640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100e90 0x7ff3f81981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3fdf64640 1 -- 172.21.15.79:0/2193154763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f819fe10 con 0x7ff3f81029a0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3fd763640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100e90 0x7ff3f81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3fdf64640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 0x7ff3f819f7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000eb10 tx=0x7ff3f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0004b20 con 0x7ff3f81029a0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f0004cc0 con 0x7ff3f81029a0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f81a0100 con 0x7ff3f81029a0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0010670 con 0x7ff3f81029a0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.447+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f81a05d0 con 0x7ff3f81029a0 2024-04-22T06:44:28.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.448+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f81a0890 con 0x7ff3f81029a0 2024-04-22T06:44:28.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.449+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7ff3f0005020 con 0x7ff3f81029a0 2024-04-22T06:44:28.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.450+0000 7ff3ee7fc640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff3e40a3db0 0x7ff3e40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.450+0000 7ff3fd763640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff3e40a3db0 0x7ff3e40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.450+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7ff3f00d0f20 con 0x7ff3f81029a0 2024-04-22T06:44:28.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.450+0000 7ff3fd763640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff3e40a3db0 0x7ff3e40a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4005d00 tx=0x7ff3f4005c70 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:28.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:28.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-22T06:44:28.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2516653263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:28.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:28.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1625007237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:28.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/279678836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:28.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.455+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3f009d480 con 0x7ff3f81029a0 2024-04-22T06:44:28.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.641+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff3f81a0a60 con 0x7ff3f81029a0 2024-04-22T06:44:28.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.641+0000 7ff3ee7fc640 1 -- 172.21.15.79:0/2193154763 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7ff3f009cd80 con 0x7ff3f81029a0 2024-04-22T06:44:28.640 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff3e40a3db0 msgr2=0x7ff3e40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff3e40a3db0 0x7ff3e40a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4005d00 tx=0x7ff3f4005c70 comp rx=0 tx=0).stop 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 msgr2=0x7ff3f819f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 0x7ff3f819f7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000eb10 tx=0x7ff3f000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 shutdown_connections 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3f81029a0 0x7ff3f819f7a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3f8101fe0 0x7ff3f8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff3e40a3db0 0x7ff3e40a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 --2- 172.21.15.79:0/2193154763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3f8100e90 0x7ff3f81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 >> 172.21.15.79:0/2193154763 conn(0x7ff3f80fc400 msgr2=0x7ff3f80690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:28.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 shutdown_connections 2024-04-22T06:44:28.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.643+0000 7ff3ff9c7640 1 -- 172.21.15.79:0/2193154763 wait complete. 2024-04-22T06:44:28.653 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-22T06:44:28.653 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:28.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 -- 172.21.15.79:0/2504807597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664100e70 msgr2=0x7f5664101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2504807597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664100e70 0x7f5664101290 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5660009900 tx=0x7f566002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:28.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 -- 172.21.15.79:0/2504807597 shutdown_connections 2024-04-22T06:44:28.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2504807597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664102980 0x7f56641092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2504807597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5664101fc0 0x7f5664102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2504807597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664100e70 0x7f5664101290 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 -- 172.21.15.79:0/2504807597 >> 172.21.15.79:0/2504807597 conn(0x7f56640fc3e0 msgr2=0x7f56640fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:28.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 -- 172.21.15.79:0/2504807597 shutdown_connections 2024-04-22T06:44:28.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.756+0000 7f566b7c7640 1 -- 172.21.15.79:0/2504807597 wait complete. 2024-04-22T06:44:28.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 Processor -- start 2024-04-22T06:44:28.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 -- start start 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5664100e70 0x7f5664069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 0x7f566419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664102980 0x7f566419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f566410c9a0 con 0x7f5664101fc0 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f566410c820 con 0x7f5664102980 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f566410cb20 con 0x7f5664100e70 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 0x7f566419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 0x7f566419ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33212/0 (socket says 172.21.15.79:33212) 2024-04-22T06:44:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 -- 172.21.15.79:0/2219793967 learned_addr learned my addr 172.21.15.79:0/2219793967 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5669563640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5664100e70 0x7f5664069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5669d64640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664102980 0x7f566419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 -- 172.21.15.79:0/2219793967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5664100e70 msgr2=0x7f5664069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5664100e70 0x7f5664069c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 -- 172.21.15.79:0/2219793967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664102980 msgr2=0x7f566419f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664102980 0x7f566419f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:28.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 -- 172.21.15.79:0/2219793967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566419fdd0 con 0x7f5664101fc0 2024-04-22T06:44:28.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f5668d62640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 0x7f566419ced0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f565400d770 tx=0x7f565400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:28.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654017070 con 0x7f5664101fc0 2024-04-22T06:44:28.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56641a00c0 con 0x7f5664101fc0 2024-04-22T06:44:28.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5654009c30 con 0x7f5664101fc0 2024-04-22T06:44:28.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.757+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654003cc0 con 0x7f5664101fc0 2024-04-22T06:44:28.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.758+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641a0660 con 0x7f5664101fc0 2024-04-22T06:44:28.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.759+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56641081f0 con 0x7f5664101fc0 2024-04-22T06:44:28.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.760+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f5654003e60 con 0x7f5664101fc0 2024-04-22T06:44:28.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.760+0000 7f565a7fc640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56500a3db0 0x7f56500a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:28.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.760+0000 7f5669563640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56500a3db0 0x7f56500a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:28.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.760+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f56540d0310 con 0x7f5664101fc0 2024-04-22T06:44:28.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.761+0000 7f5669563640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56500a3db0 0x7f56500a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5660009230 tx=0x7f5660005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:28.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:28.765+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f565409c870 con 0x7f5664101fc0 2024-04-22T06:44:29.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.004+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f566402d7c0 con 0x7f5664101fc0 2024-04-22T06:44:29.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.005+0000 7f565a7fc640 1 -- 172.21.15.79:0/2219793967 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7f5654009de0 con 0x7f5664101fc0 2024-04-22T06:44:29.003 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:29.004 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:44:29.004 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:44:29.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56500a3db0 msgr2=0x7f56500a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56500a3db0 0x7f56500a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5660009230 tx=0x7f5660005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 msgr2=0x7f566419ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 0x7f566419ced0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f565400d770 tx=0x7f565400dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 shutdown_connections 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56500a3db0 0x7f56500a6270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664102980 0x7f566419f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5664101fc0 0x7f566419ced0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 --2- 172.21.15.79:0/2219793967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5664100e70 0x7f5664069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 >> 172.21.15.79:0/2219793967 conn(0x7f56640fc3e0 msgr2=0x7f56641053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 shutdown_connections 2024-04-22T06:44:29.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.007+0000 7f566b7c7640 1 -- 172.21.15.79:0/2219793967 wait complete. 2024-04-22T06:44:29.017 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-22T06:44:29.017 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-22T06:44:29.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:29.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 -- 172.21.15.79:0/827005745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28101fc0 msgr2=0x7f7a28102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 --2- 172.21.15.79:0/827005745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28101fc0 0x7f7a28102440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009900 tx=0x7f7a2402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 -- 172.21.15.79:0/827005745 shutdown_connections 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 --2- 172.21.15.79:0/827005745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a28102980 0x7f7a281092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 --2- 172.21.15.79:0/827005745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28101fc0 0x7f7a28102440 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 --2- 172.21.15.79:0/827005745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a28100e70 0x7f7a28101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 -- 172.21.15.79:0/827005745 >> 172.21.15.79:0/827005745 conn(0x7f7a280fc3e0 msgr2=0x7f7a280fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 -- 172.21.15.79:0/827005745 shutdown_connections 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 -- 172.21.15.79:0/827005745 wait complete. 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 Processor -- start 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.120+0000 7f7a302df640 1 -- start start 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a28100e70 0x7f7a28069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a28101fc0 0x7f7a2819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 0x7f7a2819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2810c9a0 con 0x7f7a28102980 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2810c820 con 0x7f7a28100e70 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2810cb20 con 0x7f7a28101fc0 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 0x7f7a2819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 0x7f7a2819f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33226/0 (socket says 172.21.15.79:33226) 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 -- 172.21.15.79:0/1191586371 learned_addr learned my addr 172.21.15.79:0/1191586371 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2d87a640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a28101fc0 0x7f7a2819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e07b640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a28100e70 0x7f7a28069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 -- 172.21.15.79:0/1191586371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a28101fc0 msgr2=0x7f7a2819cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a28101fc0 0x7f7a2819cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 -- 172.21.15.79:0/1191586371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a28100e70 msgr2=0x7f7a28069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a28100e70 0x7f7a28069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 -- 172.21.15.79:0/1191586371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2819fe20 con 0x7f7a28102980 2024-04-22T06:44:29.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a2e87c640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 0x7f7a2819f510 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000eb10 tx=0x7f7a2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20004b20 con 0x7f7a28102980 2024-04-22T06:44:29.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a20004cc0 con 0x7f7a28102980 2024-04-22T06:44:29.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a281a00b0 con 0x7f7a28102980 2024-04-22T06:44:29.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.121+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20010670 con 0x7f7a28102980 2024-04-22T06:44:29.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.122+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a281a05d0 con 0x7f7a28102980 2024-04-22T06:44:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.123+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a28108250 con 0x7f7a28102980 2024-04-22T06:44:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.124+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f7a20005000 con 0x7f7a28102980 2024-04-22T06:44:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.125+0000 7f7a1f7fe640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7a140a3e30 0x7f7a140a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.125+0000 7f7a2e07b640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7a140a3e30 0x7f7a140a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.125+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f7a200d10b0 con 0x7f7a28102980 2024-04-22T06:44:29.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.125+0000 7f7a2e07b640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7a140a3e30 0x7f7a140a62f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a180095f0 tx=0x7f7a18005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.128+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a20053ff0 con 0x7f7a28102980 2024-04-22T06:44:29.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2793382726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Unable to set osd_memory_target on smithi190 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi079 to 176.2M 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Unable to set osd_memory_target on smithi079 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:44:29.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:44:29.243 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:44:29.243 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1313800874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.243 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.243 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.243 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.243 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2193154763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.300+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7a20099580 con 0x7f7a28102980 2024-04-22T06:44:29.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.318+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7a28062390 con 0x7f7a140a3e30 2024-04-22T06:44:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.325+0000 7f7a1f7fe640 1 -- 172.21.15.79:0/1191586371 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258005 (secure 0 0 0) 0x7f7a28062390 con 0x7f7a140a3e30 2024-04-22T06:44:29.328 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7a140a3e30 msgr2=0x7f7a140a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7a140a3e30 0x7f7a140a62f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a180095f0 tx=0x7f7a18005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 msgr2=0x7f7a2819f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 0x7f7a2819f510 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000eb10 tx=0x7f7a2000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 shutdown_connections 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a28102980 0x7f7a2819f510 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a28101fc0 0x7f7a2819cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7a140a3e30 0x7f7a140a62f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 --2- 172.21.15.79:0/1191586371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a28100e70 0x7f7a28069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 >> 172.21.15.79:0/1191586371 conn(0x7f7a280fc3e0 msgr2=0x7f7a28105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 shutdown_connections 2024-04-22T06:44:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.329+0000 7f7a302df640 1 -- 172.21.15.79:0/1191586371 wait complete. 2024-04-22T06:44:29.339 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:44:28.964824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186818,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":968,"num_read_kb":642,"num_write":471,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381756,"kb_used_data":11776,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560068,"statfs":{"total":1151940427776,"available":1151549509632,"internally_reserved":0,"allocated":12058624,"data_stored":8776841,"data_compressed":199080,"data_compressed_allocated":262144,"data_compressed_original":3096576,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990794"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953228+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:44:24.953228+0000","last_peered":"2024-04-22T06:44:24.953228+0000","last_clean":"2024-04-22T06:44:24.953228+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:44:24.953228+0000","last_undegraded":"2024-04-22T06:44:24.953228+0000","last_fullsized":"2024-04-22T06:44:24.953228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194515+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:44:25.194515+0000","last_peered":"2024-04-22T06:44:25.194515+0000","last_clean":"2024-04-22T06:44:25.194515+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:44:25.194515+0000","last_undegraded":"2024-04-22T06:44:25.194515+0000","last_fullsized":"2024-04-22T06:44:25.194515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194358+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:44:25.194358+0000","last_peered":"2024-04-22T06:44:25.194358+0000","last_clean":"2024-04-22T06:44:25.194358+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.194358+0000","last_undegraded":"2024-04-22T06:44:25.194358+0000","last_fullsized":"2024-04-22T06:44:25.194358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193440+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:44:25.193440+0000","last_peered":"2024-04-22T06:44:25.193440+0000","last_clean":"2024-04-22T06:44:25.193440+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:44:25.193440+0000","last_undegraded":"2024-04-22T06:44:25.193440+0000","last_fullsized":"2024-04-22T06:44:25.193440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044020+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:44:25.044020+0000","last_peered":"2024-04-22T06:44:25.044020+0000","last_clean":"2024-04-22T06:44:25.044020+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:44:25.044020+0000","last_undegraded":"2024-04-22T06:44:25.044020+0000","last_fullsized":"2024-04-22T06:44:25.044020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951989+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:44:24.951989+0000","last_peered":"2024-04-22T06:44:24.951989+0000","last_clean":"2024-04-22T06:44:24.951989+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:44:24.951989+0000","last_undegraded":"2024-04-22T06:44:24.951989+0000","last_fullsized":"2024-04-22T06:44:24.951989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194834+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:44:25.194834+0000","last_peered":"2024-04-22T06:44:25.194834+0000","last_clean":"2024-04-22T06:44:25.194834+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:44:25.194834+0000","last_undegraded":"2024-04-22T06:44:25.194834+0000","last_fullsized":"2024-04-22T06:44:25.194834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044085+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:44:25.044085+0000","last_peered":"2024-04-22T06:44:25.044085+0000","last_clean":"2024-04-22T06:44:25.044085+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:44:25.044085+0000","last_undegraded":"2024-04-22T06:44:25.044085+0000","last_fullsized":"2024-04-22T06:44:25.044085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023622+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:44:25.023622+0000","last_peered":"2024-04-22T06:44:25.023622+0000","last_clean":"2024-04-22T06:44:25.023622+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:44:25.023622+0000","last_undegraded":"2024-04-22T06:44:25.023622+0000","last_fullsized":"2024-04-22T06:44:25.023622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952422+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:44:24.952422+0000","last_peered":"2024-04-22T06:44:24.952422+0000","last_clean":"2024-04-22T06:44:24.952422+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:44:24.952422+0000","last_undegraded":"2024-04-22T06:44:24.952422+0000","last_fullsized":"2024-04-22T06:44:24.952422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195320+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:44:25.195320+0000","last_peered":"2024-04-22T06:44:25.195320+0000","last_clean":"2024-04-22T06:44:25.195320+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:44:25.195320+0000","last_undegraded":"2024-04-22T06:44:25.195320+0000","last_fullsized":"2024-04-22T06:44:25.195320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193429+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:44:25.193429+0000","last_peered":"2024-04-22T06:44:25.193429+0000","last_clean":"2024-04-22T06:44:25.193429+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:44:25.193429+0000","last_undegraded":"2024-04-22T06:44:25.193429+0000","last_fullsized":"2024-04-22T06:44:25.193429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046192+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:44:25.046192+0000","last_peered":"2024-04-22T06:44:25.046192+0000","last_clean":"2024-04-22T06:44:25.046192+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:44:25.046192+0000","last_undegraded":"2024-04-22T06:44:25.046192+0000","last_fullsized":"2024-04-22T06:44:25.046192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194092+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:44:25.194092+0000","last_peered":"2024-04-22T06:44:25.194092+0000","last_clean":"2024-04-22T06:44:25.194092+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:44:25.194092+0000","last_undegraded":"2024-04-22T06:44:25.194092+0000","last_fullsized":"2024-04-22T06:44:25.194092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193990+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:44:25.193990+0000","last_peered":"2024-04-22T06:44:25.193990+0000","last_clean":"2024-04-22T06:44:25.193990+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:44:25.193990+0000","last_undegraded":"2024-04-22T06:44:25.193990+0000","last_fullsized":"2024-04-22T06:44:25.193990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023187+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:44:25.023187+0000","last_peered":"2024-04-22T06:44:25.023187+0000","last_clean":"2024-04-22T06:44:25.023187+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:44:25.023187+0000","last_undegraded":"2024-04-22T06:44:25.023187+0000","last_fullsized":"2024-04-22T06:44:25.023187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953321+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:44:24.953321+0000","last_peered":"2024-04-22T06:44:24.953321+0000","last_clean":"2024-04-22T06:44:24.953321+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:44:24.953321+0000","last_undegraded":"2024-04-22T06:44:24.953321+0000","last_fullsized":"2024-04-22T06:44:24.953321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948506+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:44:24.948506+0000","last_peered":"2024-04-22T06:44:24.948506+0000","last_clean":"2024-04-22T06:44:24.948506+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:44:24.948506+0000","last_undegraded":"2024-04-22T06:44:24.948506+0000","last_fullsized":"2024-04-22T06:44:24.948506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195467+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:44:25.195467+0000","last_peered":"2024-04-22T06:44:25.195467+0000","last_clean":"2024-04-22T06:44:25.195467+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:44:25.195467+0000","last_undegraded":"2024-04-22T06:44:25.195467+0000","last_fullsized":"2024-04-22T06:44:25.195467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948609+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:44:24.948609+0000","last_peered":"2024-04-22T06:44:24.948609+0000","last_clean":"2024-04-22T06:44:24.948609+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:44:24.948609+0000","last_undegraded":"2024-04-22T06:44:24.948609+0000","last_fullsized":"2024-04-22T06:44:24.948609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044653+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:44:25.044653+0000","last_peered":"2024-04-22T06:44:25.044653+0000","last_clean":"2024-04-22T06:44:25.044653+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:44:25.044653+0000","last_undegraded":"2024-04-22T06:44:25.044653+0000","last_fullsized":"2024-04-22T06:44:25.044653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044718+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:44:25.044718+0000","last_peered":"2024-04-22T06:44:25.044718+0000","last_clean":"2024-04-22T06:44:25.044718+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:44:25.044718+0000","last_undegraded":"2024-04-22T06:44:25.044718+0000","last_fullsized":"2024-04-22T06:44:25.044718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194508+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:44:25.194508+0000","last_peered":"2024-04-22T06:44:25.194508+0000","last_clean":"2024-04-22T06:44:25.194508+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:44:25.194508+0000","last_undegraded":"2024-04-22T06:44:25.194508+0000","last_fullsized":"2024-04-22T06:44:25.194508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044333+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:44:25.044333+0000","last_peered":"2024-04-22T06:44:25.044333+0000","last_clean":"2024-04-22T06:44:25.044333+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:44:25.044333+0000","last_undegraded":"2024-04-22T06:44:25.044333+0000","last_fullsized":"2024-04-22T06:44:25.044333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023387+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:44:25.023387+0000","last_peered":"2024-04-22T06:44:25.023387+0000","last_clean":"2024-04-22T06:44:25.023387+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:44:25.023387+0000","last_undegraded":"2024-04-22T06:44:25.023387+0000","last_fullsized":"2024-04-22T06:44:25.023387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948348+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:44:24.948348+0000","last_peered":"2024-04-22T06:44:24.948348+0000","last_clean":"2024-04-22T06:44:24.948348+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:44:24.948348+0000","last_undegraded":"2024-04-22T06:44:24.948348+0000","last_fullsized":"2024-04-22T06:44:24.948348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'9","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.889110+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:44:25.889110+0000","last_peered":"2024-04-22T06:44:25.889110+0000","last_clean":"2024-04-22T06:44:25.889110+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:44:25.889110+0000","last_undegraded":"2024-04-22T06:44:25.889110+0000","last_fullsized":"2024-04-22T06:44:25.889110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043961+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:44:25.043961+0000","last_peered":"2024-04-22T06:44:25.043961+0000","last_clean":"2024-04-22T06:44:25.043961+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:44:25.043961+0000","last_undegraded":"2024-04-22T06:44:25.043961+0000","last_fullsized":"2024-04-22T06:44:25.043961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951916+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:44:24.951916+0000","last_peered":"2024-04-22T06:44:24.951916+0000","last_clean":"2024-04-22T06:44:24.951916+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:44:24.951916+0000","last_undegraded":"2024-04-22T06:44:24.951916+0000","last_fullsized":"2024-04-22T06:44:24.951916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050784+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:44:25.050784+0000","last_peered":"2024-04-22T06:44:25.050784+0000","last_clean":"2024-04-22T06:44:25.050784+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:44:25.050784+0000","last_undegraded":"2024-04-22T06:44:25.050784+0000","last_fullsized":"2024-04-22T06:44:25.050784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045482+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:44:25.045482+0000","last_peered":"2024-04-22T06:44:25.045482+0000","last_clean":"2024-04-22T06:44:25.045482+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:44:25.045482+0000","last_undegraded":"2024-04-22T06:44:25.045482+0000","last_fullsized":"2024-04-22T06:44:25.045482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044260+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:44:25.044260+0000","last_peered":"2024-04-22T06:44:25.044260+0000","last_clean":"2024-04-22T06:44:25.044260+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:44:25.044260+0000","last_undegraded":"2024-04-22T06:44:25.044260+0000","last_fullsized":"2024-04-22T06:44:25.044260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050589+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:44:25.050589+0000","last_peered":"2024-04-22T06:44:25.050589+0000","last_clean":"2024-04-22T06:44:25.050589+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:44:25.050589+0000","last_undegraded":"2024-04-22T06:44:25.050589+0000","last_fullsized":"2024-04-22T06:44:25.050589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948311+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:44:24.948311+0000","last_peered":"2024-04-22T06:44:24.948311+0000","last_clean":"2024-04-22T06:44:24.948311+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:44:24.948311+0000","last_undegraded":"2024-04-22T06:44:24.948311+0000","last_fullsized":"2024-04-22T06:44:24.948311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.197928+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:44:25.197928+0000","last_peered":"2024-04-22T06:44:25.197928+0000","last_clean":"2024-04-22T06:44:25.197928+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:44:25.197928+0000","last_undegraded":"2024-04-22T06:44:25.197928+0000","last_fullsized":"2024-04-22T06:44:25.197928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023518+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:44:25.023518+0000","last_peered":"2024-04-22T06:44:25.023518+0000","last_clean":"2024-04-22T06:44:25.023518+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:44:25.023518+0000","last_undegraded":"2024-04-22T06:44:25.023518+0000","last_fullsized":"2024-04-22T06:44:25.023518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195175+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:44:25.195175+0000","last_peered":"2024-04-22T06:44:25.195175+0000","last_clean":"2024-04-22T06:44:25.195175+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:44:25.195175+0000","last_undegraded":"2024-04-22T06:44:25.195175+0000","last_fullsized":"2024-04-22T06:44:25.195175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194428+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:44:25.194428+0000","last_peered":"2024-04-22T06:44:25.194428+0000","last_clean":"2024-04-22T06:44:25.194428+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:44:25.194428+0000","last_undegraded":"2024-04-22T06:44:25.194428+0000","last_fullsized":"2024-04-22T06:44:25.194428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050921+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:44:25.050921+0000","last_peered":"2024-04-22T06:44:25.050921+0000","last_clean":"2024-04-22T06:44:25.050921+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:44:25.050921+0000","last_undegraded":"2024-04-22T06:44:25.050921+0000","last_fullsized":"2024-04-22T06:44:25.050921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023508+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:44:25.023508+0000","last_peered":"2024-04-22T06:44:25.023508+0000","last_clean":"2024-04-22T06:44:25.023508+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:44:25.023508+0000","last_undegraded":"2024-04-22T06:44:25.023508+0000","last_fullsized":"2024-04-22T06:44:25.023508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":301,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.889056+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:44:25.889056+0000","last_peered":"2024-04-22T06:44:25.889056+0000","last_clean":"2024-04-22T06:44:25.889056+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:44:25.889056+0000","last_undegraded":"2024-04-22T06:44:25.889056+0000","last_fullsized":"2024-04-22T06:44:25.889056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65553,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":186,"num_write":35,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044125+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:44:25.044125+0000","last_peered":"2024-04-22T06:44:25.044125+0000","last_clean":"2024-04-22T06:44:25.044125+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:44:25.044125+0000","last_undegraded":"2024-04-22T06:44:25.044125+0000","last_fullsized":"2024-04-22T06:44:25.044125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195568+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:44:25.195568+0000","last_peered":"2024-04-22T06:44:25.195568+0000","last_clean":"2024-04-22T06:44:25.195568+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:44:25.195568+0000","last_undegraded":"2024-04-22T06:44:25.195568+0000","last_fullsized":"2024-04-22T06:44:25.195568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045677+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:44:25.045677+0000","last_peered":"2024-04-22T06:44:25.045677+0000","last_clean":"2024-04-22T06:44:25.045677+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:44:25.045677+0000","last_undegraded":"2024-04-22T06:44:25.045677+0000","last_fullsized":"2024-04-22T06:44:25.045677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050734+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:44:25.050734+0000","last_peered":"2024-04-22T06:44:25.050734+0000","last_clean":"2024-04-22T06:44:25.050734+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:44:25.050734+0000","last_undegraded":"2024-04-22T06:44:25.050734+0000","last_fullsized":"2024-04-22T06:44:25.050734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195155+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:44:25.195155+0000","last_peered":"2024-04-22T06:44:25.195155+0000","last_clean":"2024-04-22T06:44:25.195155+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:44:25.195155+0000","last_undegraded":"2024-04-22T06:44:25.195155+0000","last_fullsized":"2024-04-22T06:44:25.195155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":150,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.888981+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:44:25.888981+0000","last_peered":"2024-04-22T06:44:25.888981+0000","last_clean":"2024-04-22T06:44:25.888981+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:44:25.888981+0000","last_undegraded":"2024-04-22T06:44:25.888981+0000","last_fullsized":"2024-04-22T06:44:25.888981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194722+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:44:25.194722+0000","last_peered":"2024-04-22T06:44:25.194722+0000","last_clean":"2024-04-22T06:44:25.194722+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:44:25.194722+0000","last_undegraded":"2024-04-22T06:44:25.194722+0000","last_fullsized":"2024-04-22T06:44:25.194722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043891+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:44:25.043891+0000","last_peered":"2024-04-22T06:44:25.043891+0000","last_clean":"2024-04-22T06:44:25.043891+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:44:25.043891+0000","last_undegraded":"2024-04-22T06:44:25.043891+0000","last_fullsized":"2024-04-22T06:44:25.043891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050670+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:44:25.050670+0000","last_peered":"2024-04-22T06:44:25.050670+0000","last_clean":"2024-04-22T06:44:25.050670+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:44:25.050670+0000","last_undegraded":"2024-04-22T06:44:25.050670+0000","last_fullsized":"2024-04-22T06:44:25.050670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044215+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:44:25.044215+0000","last_peered":"2024-04-22T06:44:25.044215+0000","last_clean":"2024-04-22T06:44:25.044215+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:44:25.044215+0000","last_undegraded":"2024-04-22T06:44:25.044215+0000","last_fullsized":"2024-04-22T06:44:25.044215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953070+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:44:24.953070+0000","last_peered":"2024-04-22T06:44:24.953070+0000","last_clean":"2024-04-22T06:44:24.953070+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:44:24.953070+0000","last_undegraded":"2024-04-22T06:44:24.953070+0000","last_fullsized":"2024-04-22T06:44:24.953070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948439+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:44:24.948439+0000","last_peered":"2024-04-22T06:44:24.948439+0000","last_clean":"2024-04-22T06:44:24.948439+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:44:24.948439+0000","last_undegraded":"2024-04-22T06:44:24.948439+0000","last_fullsized":"2024-04-22T06:44:24.948439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952192+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:44:24.952192+0000","last_peered":"2024-04-22T06:44:24.952192+0000","last_clean":"2024-04-22T06:44:24.952192+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:44:24.952192+0000","last_undegraded":"2024-04-22T06:44:24.952192+0000","last_fullsized":"2024-04-22T06:44:24.952192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195007+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:44:25.195007+0000","last_peered":"2024-04-22T06:44:25.195007+0000","last_clean":"2024-04-22T06:44:25.195007+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:44:25.195007+0000","last_undegraded":"2024-04-22T06:44:25.195007+0000","last_fullsized":"2024-04-22T06:44:25.195007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952314+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:44:24.952314+0000","last_peered":"2024-04-22T06:44:24.952314+0000","last_clean":"2024-04-22T06:44:24.952314+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:44:24.952314+0000","last_undegraded":"2024-04-22T06:44:24.952314+0000","last_fullsized":"2024-04-22T06:44:24.952314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050823+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:44:25.050823+0000","last_peered":"2024-04-22T06:44:25.050823+0000","last_clean":"2024-04-22T06:44:25.050823+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:44:25.050823+0000","last_undegraded":"2024-04-22T06:44:25.050823+0000","last_fullsized":"2024-04-22T06:44:25.050823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193813+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:44:25.193813+0000","last_peered":"2024-04-22T06:44:25.193813+0000","last_clean":"2024-04-22T06:44:25.193813+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:44:25.193813+0000","last_undegraded":"2024-04-22T06:44:25.193813+0000","last_fullsized":"2024-04-22T06:44:25.193813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045411+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:44:25.045411+0000","last_peered":"2024-04-22T06:44:25.045411+0000","last_clean":"2024-04-22T06:44:25.045411+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:44:25.045411+0000","last_undegraded":"2024-04-22T06:44:25.045411+0000","last_fullsized":"2024-04-22T06:44:25.045411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193355+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:44:25.193355+0000","last_peered":"2024-04-22T06:44:25.193355+0000","last_clean":"2024-04-22T06:44:25.193355+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:44:25.193355+0000","last_undegraded":"2024-04-22T06:44:25.193355+0000","last_fullsized":"2024-04-22T06:44:25.193355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194796+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:44:25.194796+0000","last_peered":"2024-04-22T06:44:25.194796+0000","last_clean":"2024-04-22T06:44:25.194796+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:44:25.194796+0000","last_undegraded":"2024-04-22T06:44:25.194796+0000","last_fullsized":"2024-04-22T06:44:25.194796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045286+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:44:25.045286+0000","last_peered":"2024-04-22T06:44:25.045286+0000","last_clean":"2024-04-22T06:44:25.045286+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:44:25.045286+0000","last_undegraded":"2024-04-22T06:44:25.045286+0000","last_fullsized":"2024-04-22T06:44:25.045286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194945+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:44:25.194945+0000","last_peered":"2024-04-22T06:44:25.194945+0000","last_clean":"2024-04-22T06:44:25.194945+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:44:25.194945+0000","last_undegraded":"2024-04-22T06:44:25.194945+0000","last_fullsized":"2024-04-22T06:44:25.194945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044263+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:44:25.044263+0000","last_peered":"2024-04-22T06:44:25.044263+0000","last_clean":"2024-04-22T06:44:25.044263+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:44:25.044263+0000","last_undegraded":"2024-04-22T06:44:25.044263+0000","last_fullsized":"2024-04-22T06:44:25.044263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045949+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:44:25.045949+0000","last_peered":"2024-04-22T06:44:25.045949+0000","last_clean":"2024-04-22T06:44:25.045949+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:44:25.045949+0000","last_undegraded":"2024-04-22T06:44:25.045949+0000","last_fullsized":"2024-04-22T06:44:25.045949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.156832+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:44:25.156832+0000","last_peered":"2024-04-22T06:44:25.156832+0000","last_clean":"2024-04-22T06:44:25.156832+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:44:25.156832+0000","last_undegraded":"2024-04-22T06:44:25.156832+0000","last_fullsized":"2024-04-22T06:44:25.156832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044531+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:44:25.044531+0000","last_peered":"2024-04-22T06:44:25.044531+0000","last_clean":"2024-04-22T06:44:25.044531+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:44:25.044531+0000","last_undegraded":"2024-04-22T06:44:25.044531+0000","last_fullsized":"2024-04-22T06:44:25.044531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193293+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:44:25.193293+0000","last_peered":"2024-04-22T06:44:25.193293+0000","last_clean":"2024-04-22T06:44:25.193293+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:44:25.193293+0000","last_undegraded":"2024-04-22T06:44:25.193293+0000","last_fullsized":"2024-04-22T06:44:25.193293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194323+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:44:25.194323+0000","last_peered":"2024-04-22T06:44:25.194323+0000","last_clean":"2024-04-22T06:44:25.194323+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:44:25.194323+0000","last_undegraded":"2024-04-22T06:44:25.194323+0000","last_fullsized":"2024-04-22T06:44:25.194323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023242+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:44:25.023242+0000","last_peered":"2024-04-22T06:44:25.023242+0000","last_clean":"2024-04-22T06:44:25.023242+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:44:25.023242+0000","last_undegraded":"2024-04-22T06:44:25.023242+0000","last_fullsized":"2024-04-22T06:44:25.023242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043912+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:44:25.043912+0000","last_peered":"2024-04-22T06:44:25.043912+0000","last_clean":"2024-04-22T06:44:25.043912+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:44:25.043912+0000","last_undegraded":"2024-04-22T06:44:25.043912+0000","last_fullsized":"2024-04-22T06:44:25.043912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044235+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:44:25.044235+0000","last_peered":"2024-04-22T06:44:25.044235+0000","last_clean":"2024-04-22T06:44:25.044235+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:44:25.044235+0000","last_undegraded":"2024-04-22T06:44:25.044235+0000","last_fullsized":"2024-04-22T06:44:25.044235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952874+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:44:24.952874+0000","last_peered":"2024-04-22T06:44:24.952874+0000","last_clean":"2024-04-22T06:44:24.952874+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:44:24.952874+0000","last_undegraded":"2024-04-22T06:44:24.952874+0000","last_fullsized":"2024-04-22T06:44:24.952874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953015+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:44:24.953015+0000","last_peered":"2024-04-22T06:44:24.953015+0000","last_clean":"2024-04-22T06:44:24.953015+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:44:24.953015+0000","last_undegraded":"2024-04-22T06:44:24.953015+0000","last_fullsized":"2024-04-22T06:44:24.953015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044498+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:44:25.044498+0000","last_peered":"2024-04-22T06:44:25.044498+0000","last_clean":"2024-04-22T06:44:25.044498+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:44:25.044498+0000","last_undegraded":"2024-04-22T06:44:25.044498+0000","last_fullsized":"2024-04-22T06:44:25.044498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952952+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:44:24.952952+0000","last_peered":"2024-04-22T06:44:24.952952+0000","last_clean":"2024-04-22T06:44:24.952952+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:44:24.952952+0000","last_undegraded":"2024-04-22T06:44:24.952952+0000","last_fullsized":"2024-04-22T06:44:24.952952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195201+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:44:25.195201+0000","last_peered":"2024-04-22T06:44:25.195201+0000","last_clean":"2024-04-22T06:44:25.195201+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:44:25.195201+0000","last_undegraded":"2024-04-22T06:44:25.195201+0000","last_fullsized":"2024-04-22T06:44:25.195201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195645+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:44:25.195645+0000","last_peered":"2024-04-22T06:44:25.195645+0000","last_clean":"2024-04-22T06:44:25.195645+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:44:25.195645+0000","last_undegraded":"2024-04-22T06:44:25.195645+0000","last_fullsized":"2024-04-22T06:44:25.195645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195419+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:44:25.195419+0000","last_peered":"2024-04-22T06:44:25.195419+0000","last_clean":"2024-04-22T06:44:25.195419+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:44:25.195419+0000","last_undegraded":"2024-04-22T06:44:25.195419+0000","last_fullsized":"2024-04-22T06:44:25.195419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.150698+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:44:25.150698+0000","last_peered":"2024-04-22T06:44:25.150698+0000","last_clean":"2024-04-22T06:44:25.150698+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:44:25.150698+0000","last_undegraded":"2024-04-22T06:44:25.150698+0000","last_fullsized":"2024-04-22T06:44:25.150698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195126+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:44:25.195126+0000","last_peered":"2024-04-22T06:44:25.195126+0000","last_clean":"2024-04-22T06:44:25.195126+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:44:25.195126+0000","last_undegraded":"2024-04-22T06:44:25.195126+0000","last_fullsized":"2024-04-22T06:44:25.195126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046237+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:44:25.046237+0000","last_peered":"2024-04-22T06:44:25.046237+0000","last_clean":"2024-04-22T06:44:25.046237+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:44:25.046237+0000","last_undegraded":"2024-04-22T06:44:25.046237+0000","last_fullsized":"2024-04-22T06:44:25.046237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195405+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:44:25.195405+0000","last_peered":"2024-04-22T06:44:25.195405+0000","last_clean":"2024-04-22T06:44:25.195405+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:44:25.195405+0000","last_undegraded":"2024-04-22T06:44:25.195405+0000","last_fullsized":"2024-04-22T06:44:25.195405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952951+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:44:24.952951+0000","last_peered":"2024-04-22T06:44:24.952951+0000","last_clean":"2024-04-22T06:44:24.952951+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:44:24.952951+0000","last_undegraded":"2024-04-22T06:44:24.952951+0000","last_fullsized":"2024-04-22T06:44:24.952951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195560+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:44:25.195560+0000","last_peered":"2024-04-22T06:44:25.195560+0000","last_clean":"2024-04-22T06:44:25.195560+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:44:25.195560+0000","last_undegraded":"2024-04-22T06:44:25.195560+0000","last_fullsized":"2024-04-22T06:44:25.195560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195368+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:44:25.195368+0000","last_peered":"2024-04-22T06:44:25.195368+0000","last_clean":"2024-04-22T06:44:25.195368+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:44:25.195368+0000","last_undegraded":"2024-04-22T06:44:25.195368+0000","last_fullsized":"2024-04-22T06:44:25.195368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194250+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:44:25.194250+0000","last_peered":"2024-04-22T06:44:25.194250+0000","last_clean":"2024-04-22T06:44:25.194250+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:44:25.194250+0000","last_undegraded":"2024-04-22T06:44:25.194250+0000","last_fullsized":"2024-04-22T06:44:25.194250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045832+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:44:25.045832+0000","last_peered":"2024-04-22T06:44:25.045832+0000","last_clean":"2024-04-22T06:44:25.045832+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:44:25.045832+0000","last_undegraded":"2024-04-22T06:44:25.045832+0000","last_fullsized":"2024-04-22T06:44:25.045832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953143+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:44:24.953143+0000","last_peered":"2024-04-22T06:44:24.953143+0000","last_clean":"2024-04-22T06:44:24.953143+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:44:24.953143+0000","last_undegraded":"2024-04-22T06:44:24.953143+0000","last_fullsized":"2024-04-22T06:44:24.953143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195715+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:44:25.195715+0000","last_peered":"2024-04-22T06:44:25.195715+0000","last_clean":"2024-04-22T06:44:25.195715+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:44:25.195715+0000","last_undegraded":"2024-04-22T06:44:25.195715+0000","last_fullsized":"2024-04-22T06:44:25.195715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044467+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:44:25.044467+0000","last_peered":"2024-04-22T06:44:25.044467+0000","last_clean":"2024-04-22T06:44:25.044467+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:44:25.044467+0000","last_undegraded":"2024-04-22T06:44:25.044467+0000","last_fullsized":"2024-04-22T06:44:25.044467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050646+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:44:25.050646+0000","last_peered":"2024-04-22T06:44:25.050646+0000","last_clean":"2024-04-22T06:44:25.050646+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:44:25.050646+0000","last_undegraded":"2024-04-22T06:44:25.050646+0000","last_fullsized":"2024-04-22T06:44:25.050646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044351+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:44:25.044351+0000","last_peered":"2024-04-22T06:44:25.044351+0000","last_clean":"2024-04-22T06:44:25.044351+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:44:25.044351+0000","last_undegraded":"2024-04-22T06:44:25.044351+0000","last_fullsized":"2024-04-22T06:44:25.044351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023389+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:44:25.023389+0000","last_peered":"2024-04-22T06:44:25.023389+0000","last_clean":"2024-04-22T06:44:25.023389+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:44:25.023389+0000","last_undegraded":"2024-04-22T06:44:25.023389+0000","last_fullsized":"2024-04-22T06:44:25.023389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045996+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:44:25.045996+0000","last_peered":"2024-04-22T06:44:25.045996+0000","last_clean":"2024-04-22T06:44:25.045996+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.045996+0000","last_undegraded":"2024-04-22T06:44:25.045996+0000","last_fullsized":"2024-04-22T06:44:25.045996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953024+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:44:24.953024+0000","last_peered":"2024-04-22T06:44:24.953024+0000","last_clean":"2024-04-22T06:44:24.953024+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:44:24.953024+0000","last_undegraded":"2024-04-22T06:44:24.953024+0000","last_fullsized":"2024-04-22T06:44:24.953024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":596216,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":832,"num_read_kb":523,"num_write":288,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1788648,"data_compressed":81528,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":465124,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":1253508,"data_compressed":46537,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858576,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":661888,"data_compressed":12626,"data_compressed_allocated":16384,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1048956,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":464749,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":530095,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":532759,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989876,"data_compressed":28203,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466109,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65553,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68217,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134390,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199654,"data_compressed":8030,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:29.343 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2793382726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Unable to set osd_memory_target on smithi190 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.345 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi079 to 176.2M 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Unable to set osd_memory_target on smithi079 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1313800874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.346 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2193154763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2793382726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Unable to set osd_memory_target on smithi190 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi079 to 176.2M 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Unable to set osd_memory_target on smithi079 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Updating smithi079:/etc/ceph/ceph.conf 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Updating smithi190:/etc/ceph/ceph.conf 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1313800874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:29.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2193154763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:29.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 -- 172.21.15.79:0/1233685901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930102b50 msgr2=0x7f0930106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1233685901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930102b50 0x7f0930106b90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0924009900 tx=0x7f092402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:29.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 -- 172.21.15.79:0/1233685901 shutdown_connections 2024-04-22T06:44:29.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1233685901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0930107270 0x7f093010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1233685901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930102b50 0x7f0930106b90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1233685901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09301021a0 0x7f0930102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 -- 172.21.15.79:0/1233685901 >> 172.21.15.79:0/1233685901 conn(0x7f09300793a0 msgr2=0x7f09300797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 -- 172.21.15.79:0/1233685901 shutdown_connections 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 -- 172.21.15.79:0/1233685901 wait complete. 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.483+0000 7f0935cd5640 1 Processor -- start 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 -- start start 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09301021a0 0x7f09301986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 0x7f0930198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093019cfb0 0x7f093019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0930112190 con 0x7f0930107270 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0930112010 con 0x7f09301021a0 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f0935cd5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0930112310 con 0x7f093019cfb0 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 0x7f0930198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 0x7f0930198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33238/0 (socket says 172.21.15.79:33238) 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 -- 172.21.15.79:0/1910628401 learned_addr learned my addr 172.21.15.79:0/1910628401 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092ffff640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093019cfb0 0x7f093019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092f7fe640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09301021a0 0x7f09301986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 -- 172.21.15.79:0/1910628401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093019cfb0 msgr2=0x7f093019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093019cfb0 0x7f093019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 -- 172.21.15.79:0/1910628401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09301021a0 msgr2=0x7f09301986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09301021a0 0x7f09301986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 -- 172.21.15.79:0/1910628401 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093019daa0 con 0x7f0930107270 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.484+0000 7f092effd640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 0x7f0930198c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0924036040 tx=0x7f09240045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.485+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0924043020 con 0x7f0930107270 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.485+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f092402cb70 con 0x7f0930107270 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.485+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093019dd30 con 0x7f0930107270 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.485+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0924003bf0 con 0x7f0930107270 2024-04-22T06:44:29.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.485+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09301a5550 con 0x7f0930107270 2024-04-22T06:44:29.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.486+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0930069230 con 0x7f0930107270 2024-04-22T06:44:29.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.487+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f0924046020 con 0x7f0930107270 2024-04-22T06:44:29.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.487+0000 7f092cff9640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f09180a3e40 0x7f09180a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.487+0000 7f092f7fe640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f09180a3e40 0x7f09180a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.487+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f09240f2b20 con 0x7f0930107270 2024-04-22T06:44:29.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.488+0000 7f092f7fe640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f09180a3e40 0x7f09180a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0920005fd0 tx=0x7f092000b590 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.492+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f092404d020 con 0x7f0930107270 2024-04-22T06:44:29.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.682+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0930102b50 con 0x7f09180a3e40 2024-04-22T06:44:29.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.689+0000 7f092cff9640 1 -- 172.21.15.79:0/1910628401 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258005 (secure 0 0 0) 0x7f0930102b50 con 0x7f09180a3e40 2024-04-22T06:44:29.690 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:29.691 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f09180a3e40 msgr2=0x7f09180a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f09180a3e40 0x7f09180a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0920005fd0 tx=0x7f092000b590 comp rx=0 tx=0).stop 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 msgr2=0x7f0930198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 0x7f0930198c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0924036040 tx=0x7f09240045e0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 shutdown_connections 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f093019cfb0 0x7f093019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0930107270 0x7f0930198c30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f09180a3e40 0x7f09180a6300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 --2- 172.21.15.79:0/1910628401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09301021a0 0x7f09301986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 >> 172.21.15.79:0/1910628401 conn(0x7f09300793a0 msgr2=0x7f09300fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 shutdown_connections 2024-04-22T06:44:29.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.692+0000 7f0935cd5640 1 -- 172.21.15.79:0/1910628401 wait complete. 2024-04-22T06:44:29.702 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:44:28.964824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186818,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":968,"num_read_kb":642,"num_write":471,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381756,"kb_used_data":11776,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560068,"statfs":{"total":1151940427776,"available":1151549509632,"internally_reserved":0,"allocated":12058624,"data_stored":8776841,"data_compressed":199080,"data_compressed_allocated":262144,"data_compressed_original":3096576,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990794"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953228+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:44:24.953228+0000","last_peered":"2024-04-22T06:44:24.953228+0000","last_clean":"2024-04-22T06:44:24.953228+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:44:24.953228+0000","last_undegraded":"2024-04-22T06:44:24.953228+0000","last_fullsized":"2024-04-22T06:44:24.953228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194515+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:44:25.194515+0000","last_peered":"2024-04-22T06:44:25.194515+0000","last_clean":"2024-04-22T06:44:25.194515+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:44:25.194515+0000","last_undegraded":"2024-04-22T06:44:25.194515+0000","last_fullsized":"2024-04-22T06:44:25.194515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194358+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:44:25.194358+0000","last_peered":"2024-04-22T06:44:25.194358+0000","last_clean":"2024-04-22T06:44:25.194358+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.194358+0000","last_undegraded":"2024-04-22T06:44:25.194358+0000","last_fullsized":"2024-04-22T06:44:25.194358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193440+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:44:25.193440+0000","last_peered":"2024-04-22T06:44:25.193440+0000","last_clean":"2024-04-22T06:44:25.193440+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:44:25.193440+0000","last_undegraded":"2024-04-22T06:44:25.193440+0000","last_fullsized":"2024-04-22T06:44:25.193440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044020+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:44:25.044020+0000","last_peered":"2024-04-22T06:44:25.044020+0000","last_clean":"2024-04-22T06:44:25.044020+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:44:25.044020+0000","last_undegraded":"2024-04-22T06:44:25.044020+0000","last_fullsized":"2024-04-22T06:44:25.044020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951989+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:44:24.951989+0000","last_peered":"2024-04-22T06:44:24.951989+0000","last_clean":"2024-04-22T06:44:24.951989+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:44:24.951989+0000","last_undegraded":"2024-04-22T06:44:24.951989+0000","last_fullsized":"2024-04-22T06:44:24.951989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194834+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:44:25.194834+0000","last_peered":"2024-04-22T06:44:25.194834+0000","last_clean":"2024-04-22T06:44:25.194834+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:44:25.194834+0000","last_undegraded":"2024-04-22T06:44:25.194834+0000","last_fullsized":"2024-04-22T06:44:25.194834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044085+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:44:25.044085+0000","last_peered":"2024-04-22T06:44:25.044085+0000","last_clean":"2024-04-22T06:44:25.044085+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:44:25.044085+0000","last_undegraded":"2024-04-22T06:44:25.044085+0000","last_fullsized":"2024-04-22T06:44:25.044085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023622+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:44:25.023622+0000","last_peered":"2024-04-22T06:44:25.023622+0000","last_clean":"2024-04-22T06:44:25.023622+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:44:25.023622+0000","last_undegraded":"2024-04-22T06:44:25.023622+0000","last_fullsized":"2024-04-22T06:44:25.023622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952422+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:44:24.952422+0000","last_peered":"2024-04-22T06:44:24.952422+0000","last_clean":"2024-04-22T06:44:24.952422+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:44:24.952422+0000","last_undegraded":"2024-04-22T06:44:24.952422+0000","last_fullsized":"2024-04-22T06:44:24.952422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195320+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:44:25.195320+0000","last_peered":"2024-04-22T06:44:25.195320+0000","last_clean":"2024-04-22T06:44:25.195320+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:44:25.195320+0000","last_undegraded":"2024-04-22T06:44:25.195320+0000","last_fullsized":"2024-04-22T06:44:25.195320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193429+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:44:25.193429+0000","last_peered":"2024-04-22T06:44:25.193429+0000","last_clean":"2024-04-22T06:44:25.193429+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:44:25.193429+0000","last_undegraded":"2024-04-22T06:44:25.193429+0000","last_fullsized":"2024-04-22T06:44:25.193429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046192+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:44:25.046192+0000","last_peered":"2024-04-22T06:44:25.046192+0000","last_clean":"2024-04-22T06:44:25.046192+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:44:25.046192+0000","last_undegraded":"2024-04-22T06:44:25.046192+0000","last_fullsized":"2024-04-22T06:44:25.046192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194092+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:44:25.194092+0000","last_peered":"2024-04-22T06:44:25.194092+0000","last_clean":"2024-04-22T06:44:25.194092+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:44:25.194092+0000","last_undegraded":"2024-04-22T06:44:25.194092+0000","last_fullsized":"2024-04-22T06:44:25.194092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193990+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:44:25.193990+0000","last_peered":"2024-04-22T06:44:25.193990+0000","last_clean":"2024-04-22T06:44:25.193990+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:44:25.193990+0000","last_undegraded":"2024-04-22T06:44:25.193990+0000","last_fullsized":"2024-04-22T06:44:25.193990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023187+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:44:25.023187+0000","last_peered":"2024-04-22T06:44:25.023187+0000","last_clean":"2024-04-22T06:44:25.023187+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:44:25.023187+0000","last_undegraded":"2024-04-22T06:44:25.023187+0000","last_fullsized":"2024-04-22T06:44:25.023187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953321+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:44:24.953321+0000","last_peered":"2024-04-22T06:44:24.953321+0000","last_clean":"2024-04-22T06:44:24.953321+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:44:24.953321+0000","last_undegraded":"2024-04-22T06:44:24.953321+0000","last_fullsized":"2024-04-22T06:44:24.953321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948506+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:44:24.948506+0000","last_peered":"2024-04-22T06:44:24.948506+0000","last_clean":"2024-04-22T06:44:24.948506+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:44:24.948506+0000","last_undegraded":"2024-04-22T06:44:24.948506+0000","last_fullsized":"2024-04-22T06:44:24.948506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195467+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:44:25.195467+0000","last_peered":"2024-04-22T06:44:25.195467+0000","last_clean":"2024-04-22T06:44:25.195467+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:44:25.195467+0000","last_undegraded":"2024-04-22T06:44:25.195467+0000","last_fullsized":"2024-04-22T06:44:25.195467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948609+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:44:24.948609+0000","last_peered":"2024-04-22T06:44:24.948609+0000","last_clean":"2024-04-22T06:44:24.948609+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:44:24.948609+0000","last_undegraded":"2024-04-22T06:44:24.948609+0000","last_fullsized":"2024-04-22T06:44:24.948609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044653+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:44:25.044653+0000","last_peered":"2024-04-22T06:44:25.044653+0000","last_clean":"2024-04-22T06:44:25.044653+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:44:25.044653+0000","last_undegraded":"2024-04-22T06:44:25.044653+0000","last_fullsized":"2024-04-22T06:44:25.044653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044718+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:44:25.044718+0000","last_peered":"2024-04-22T06:44:25.044718+0000","last_clean":"2024-04-22T06:44:25.044718+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:44:25.044718+0000","last_undegraded":"2024-04-22T06:44:25.044718+0000","last_fullsized":"2024-04-22T06:44:25.044718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194508+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:44:25.194508+0000","last_peered":"2024-04-22T06:44:25.194508+0000","last_clean":"2024-04-22T06:44:25.194508+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:44:25.194508+0000","last_undegraded":"2024-04-22T06:44:25.194508+0000","last_fullsized":"2024-04-22T06:44:25.194508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044333+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:44:25.044333+0000","last_peered":"2024-04-22T06:44:25.044333+0000","last_clean":"2024-04-22T06:44:25.044333+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:44:25.044333+0000","last_undegraded":"2024-04-22T06:44:25.044333+0000","last_fullsized":"2024-04-22T06:44:25.044333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023387+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:44:25.023387+0000","last_peered":"2024-04-22T06:44:25.023387+0000","last_clean":"2024-04-22T06:44:25.023387+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:44:25.023387+0000","last_undegraded":"2024-04-22T06:44:25.023387+0000","last_fullsized":"2024-04-22T06:44:25.023387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948348+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:44:24.948348+0000","last_peered":"2024-04-22T06:44:24.948348+0000","last_clean":"2024-04-22T06:44:24.948348+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:44:24.948348+0000","last_undegraded":"2024-04-22T06:44:24.948348+0000","last_fullsized":"2024-04-22T06:44:24.948348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'9","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.889110+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:44:25.889110+0000","last_peered":"2024-04-22T06:44:25.889110+0000","last_clean":"2024-04-22T06:44:25.889110+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:44:25.889110+0000","last_undegraded":"2024-04-22T06:44:25.889110+0000","last_fullsized":"2024-04-22T06:44:25.889110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043961+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:44:25.043961+0000","last_peered":"2024-04-22T06:44:25.043961+0000","last_clean":"2024-04-22T06:44:25.043961+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:44:25.043961+0000","last_undegraded":"2024-04-22T06:44:25.043961+0000","last_fullsized":"2024-04-22T06:44:25.043961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951916+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:44:24.951916+0000","last_peered":"2024-04-22T06:44:24.951916+0000","last_clean":"2024-04-22T06:44:24.951916+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:44:24.951916+0000","last_undegraded":"2024-04-22T06:44:24.951916+0000","last_fullsized":"2024-04-22T06:44:24.951916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050784+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:44:25.050784+0000","last_peered":"2024-04-22T06:44:25.050784+0000","last_clean":"2024-04-22T06:44:25.050784+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:44:25.050784+0000","last_undegraded":"2024-04-22T06:44:25.050784+0000","last_fullsized":"2024-04-22T06:44:25.050784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045482+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:44:25.045482+0000","last_peered":"2024-04-22T06:44:25.045482+0000","last_clean":"2024-04-22T06:44:25.045482+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:44:25.045482+0000","last_undegraded":"2024-04-22T06:44:25.045482+0000","last_fullsized":"2024-04-22T06:44:25.045482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044260+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:44:25.044260+0000","last_peered":"2024-04-22T06:44:25.044260+0000","last_clean":"2024-04-22T06:44:25.044260+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:44:25.044260+0000","last_undegraded":"2024-04-22T06:44:25.044260+0000","last_fullsized":"2024-04-22T06:44:25.044260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050589+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:44:25.050589+0000","last_peered":"2024-04-22T06:44:25.050589+0000","last_clean":"2024-04-22T06:44:25.050589+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:44:25.050589+0000","last_undegraded":"2024-04-22T06:44:25.050589+0000","last_fullsized":"2024-04-22T06:44:25.050589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948311+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:44:24.948311+0000","last_peered":"2024-04-22T06:44:24.948311+0000","last_clean":"2024-04-22T06:44:24.948311+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:44:24.948311+0000","last_undegraded":"2024-04-22T06:44:24.948311+0000","last_fullsized":"2024-04-22T06:44:24.948311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.197928+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:44:25.197928+0000","last_peered":"2024-04-22T06:44:25.197928+0000","last_clean":"2024-04-22T06:44:25.197928+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:44:25.197928+0000","last_undegraded":"2024-04-22T06:44:25.197928+0000","last_fullsized":"2024-04-22T06:44:25.197928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023518+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:44:25.023518+0000","last_peered":"2024-04-22T06:44:25.023518+0000","last_clean":"2024-04-22T06:44:25.023518+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:44:25.023518+0000","last_undegraded":"2024-04-22T06:44:25.023518+0000","last_fullsized":"2024-04-22T06:44:25.023518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195175+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:44:25.195175+0000","last_peered":"2024-04-22T06:44:25.195175+0000","last_clean":"2024-04-22T06:44:25.195175+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:44:25.195175+0000","last_undegraded":"2024-04-22T06:44:25.195175+0000","last_fullsized":"2024-04-22T06:44:25.195175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194428+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:44:25.194428+0000","last_peered":"2024-04-22T06:44:25.194428+0000","last_clean":"2024-04-22T06:44:25.194428+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:44:25.194428+0000","last_undegraded":"2024-04-22T06:44:25.194428+0000","last_fullsized":"2024-04-22T06:44:25.194428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050921+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:44:25.050921+0000","last_peered":"2024-04-22T06:44:25.050921+0000","last_clean":"2024-04-22T06:44:25.050921+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:44:25.050921+0000","last_undegraded":"2024-04-22T06:44:25.050921+0000","last_fullsized":"2024-04-22T06:44:25.050921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023508+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:44:25.023508+0000","last_peered":"2024-04-22T06:44:25.023508+0000","last_clean":"2024-04-22T06:44:25.023508+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:44:25.023508+0000","last_undegraded":"2024-04-22T06:44:25.023508+0000","last_fullsized":"2024-04-22T06:44:25.023508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":301,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.889056+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:44:25.889056+0000","last_peered":"2024-04-22T06:44:25.889056+0000","last_clean":"2024-04-22T06:44:25.889056+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:44:25.889056+0000","last_undegraded":"2024-04-22T06:44:25.889056+0000","last_fullsized":"2024-04-22T06:44:25.889056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65553,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":186,"num_write":35,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044125+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:44:25.044125+0000","last_peered":"2024-04-22T06:44:25.044125+0000","last_clean":"2024-04-22T06:44:25.044125+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:44:25.044125+0000","last_undegraded":"2024-04-22T06:44:25.044125+0000","last_fullsized":"2024-04-22T06:44:25.044125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195568+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:44:25.195568+0000","last_peered":"2024-04-22T06:44:25.195568+0000","last_clean":"2024-04-22T06:44:25.195568+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:44:25.195568+0000","last_undegraded":"2024-04-22T06:44:25.195568+0000","last_fullsized":"2024-04-22T06:44:25.195568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045677+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:44:25.045677+0000","last_peered":"2024-04-22T06:44:25.045677+0000","last_clean":"2024-04-22T06:44:25.045677+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:44:25.045677+0000","last_undegraded":"2024-04-22T06:44:25.045677+0000","last_fullsized":"2024-04-22T06:44:25.045677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050734+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:44:25.050734+0000","last_peered":"2024-04-22T06:44:25.050734+0000","last_clean":"2024-04-22T06:44:25.050734+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:44:25.050734+0000","last_undegraded":"2024-04-22T06:44:25.050734+0000","last_fullsized":"2024-04-22T06:44:25.050734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195155+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:44:25.195155+0000","last_peered":"2024-04-22T06:44:25.195155+0000","last_clean":"2024-04-22T06:44:25.195155+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:44:25.195155+0000","last_undegraded":"2024-04-22T06:44:25.195155+0000","last_fullsized":"2024-04-22T06:44:25.195155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":150,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.888981+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:44:25.888981+0000","last_peered":"2024-04-22T06:44:25.888981+0000","last_clean":"2024-04-22T06:44:25.888981+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:44:25.888981+0000","last_undegraded":"2024-04-22T06:44:25.888981+0000","last_fullsized":"2024-04-22T06:44:25.888981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194722+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:44:25.194722+0000","last_peered":"2024-04-22T06:44:25.194722+0000","last_clean":"2024-04-22T06:44:25.194722+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:44:25.194722+0000","last_undegraded":"2024-04-22T06:44:25.194722+0000","last_fullsized":"2024-04-22T06:44:25.194722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043891+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:44:25.043891+0000","last_peered":"2024-04-22T06:44:25.043891+0000","last_clean":"2024-04-22T06:44:25.043891+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:44:25.043891+0000","last_undegraded":"2024-04-22T06:44:25.043891+0000","last_fullsized":"2024-04-22T06:44:25.043891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050670+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:44:25.050670+0000","last_peered":"2024-04-22T06:44:25.050670+0000","last_clean":"2024-04-22T06:44:25.050670+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:44:25.050670+0000","last_undegraded":"2024-04-22T06:44:25.050670+0000","last_fullsized":"2024-04-22T06:44:25.050670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044215+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:44:25.044215+0000","last_peered":"2024-04-22T06:44:25.044215+0000","last_clean":"2024-04-22T06:44:25.044215+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:44:25.044215+0000","last_undegraded":"2024-04-22T06:44:25.044215+0000","last_fullsized":"2024-04-22T06:44:25.044215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953070+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:44:24.953070+0000","last_peered":"2024-04-22T06:44:24.953070+0000","last_clean":"2024-04-22T06:44:24.953070+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:44:24.953070+0000","last_undegraded":"2024-04-22T06:44:24.953070+0000","last_fullsized":"2024-04-22T06:44:24.953070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948439+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:44:24.948439+0000","last_peered":"2024-04-22T06:44:24.948439+0000","last_clean":"2024-04-22T06:44:24.948439+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:44:24.948439+0000","last_undegraded":"2024-04-22T06:44:24.948439+0000","last_fullsized":"2024-04-22T06:44:24.948439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952192+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:44:24.952192+0000","last_peered":"2024-04-22T06:44:24.952192+0000","last_clean":"2024-04-22T06:44:24.952192+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:44:24.952192+0000","last_undegraded":"2024-04-22T06:44:24.952192+0000","last_fullsized":"2024-04-22T06:44:24.952192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195007+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:44:25.195007+0000","last_peered":"2024-04-22T06:44:25.195007+0000","last_clean":"2024-04-22T06:44:25.195007+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:44:25.195007+0000","last_undegraded":"2024-04-22T06:44:25.195007+0000","last_fullsized":"2024-04-22T06:44:25.195007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952314+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:44:24.952314+0000","last_peered":"2024-04-22T06:44:24.952314+0000","last_clean":"2024-04-22T06:44:24.952314+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:44:24.952314+0000","last_undegraded":"2024-04-22T06:44:24.952314+0000","last_fullsized":"2024-04-22T06:44:24.952314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050823+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:44:25.050823+0000","last_peered":"2024-04-22T06:44:25.050823+0000","last_clean":"2024-04-22T06:44:25.050823+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:44:25.050823+0000","last_undegraded":"2024-04-22T06:44:25.050823+0000","last_fullsized":"2024-04-22T06:44:25.050823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193813+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:44:25.193813+0000","last_peered":"2024-04-22T06:44:25.193813+0000","last_clean":"2024-04-22T06:44:25.193813+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:44:25.193813+0000","last_undegraded":"2024-04-22T06:44:25.193813+0000","last_fullsized":"2024-04-22T06:44:25.193813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045411+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:44:25.045411+0000","last_peered":"2024-04-22T06:44:25.045411+0000","last_clean":"2024-04-22T06:44:25.045411+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:44:25.045411+0000","last_undegraded":"2024-04-22T06:44:25.045411+0000","last_fullsized":"2024-04-22T06:44:25.045411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193355+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:44:25.193355+0000","last_peered":"2024-04-22T06:44:25.193355+0000","last_clean":"2024-04-22T06:44:25.193355+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:44:25.193355+0000","last_undegraded":"2024-04-22T06:44:25.193355+0000","last_fullsized":"2024-04-22T06:44:25.193355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194796+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:44:25.194796+0000","last_peered":"2024-04-22T06:44:25.194796+0000","last_clean":"2024-04-22T06:44:25.194796+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:44:25.194796+0000","last_undegraded":"2024-04-22T06:44:25.194796+0000","last_fullsized":"2024-04-22T06:44:25.194796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045286+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:44:25.045286+0000","last_peered":"2024-04-22T06:44:25.045286+0000","last_clean":"2024-04-22T06:44:25.045286+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:44:25.045286+0000","last_undegraded":"2024-04-22T06:44:25.045286+0000","last_fullsized":"2024-04-22T06:44:25.045286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194945+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:44:25.194945+0000","last_peered":"2024-04-22T06:44:25.194945+0000","last_clean":"2024-04-22T06:44:25.194945+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:44:25.194945+0000","last_undegraded":"2024-04-22T06:44:25.194945+0000","last_fullsized":"2024-04-22T06:44:25.194945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044263+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:44:25.044263+0000","last_peered":"2024-04-22T06:44:25.044263+0000","last_clean":"2024-04-22T06:44:25.044263+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:44:25.044263+0000","last_undegraded":"2024-04-22T06:44:25.044263+0000","last_fullsized":"2024-04-22T06:44:25.044263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045949+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:44:25.045949+0000","last_peered":"2024-04-22T06:44:25.045949+0000","last_clean":"2024-04-22T06:44:25.045949+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:44:25.045949+0000","last_undegraded":"2024-04-22T06:44:25.045949+0000","last_fullsized":"2024-04-22T06:44:25.045949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.156832+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:44:25.156832+0000","last_peered":"2024-04-22T06:44:25.156832+0000","last_clean":"2024-04-22T06:44:25.156832+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:44:25.156832+0000","last_undegraded":"2024-04-22T06:44:25.156832+0000","last_fullsized":"2024-04-22T06:44:25.156832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044531+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:44:25.044531+0000","last_peered":"2024-04-22T06:44:25.044531+0000","last_clean":"2024-04-22T06:44:25.044531+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:44:25.044531+0000","last_undegraded":"2024-04-22T06:44:25.044531+0000","last_fullsized":"2024-04-22T06:44:25.044531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193293+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:44:25.193293+0000","last_peered":"2024-04-22T06:44:25.193293+0000","last_clean":"2024-04-22T06:44:25.193293+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:44:25.193293+0000","last_undegraded":"2024-04-22T06:44:25.193293+0000","last_fullsized":"2024-04-22T06:44:25.193293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194323+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:44:25.194323+0000","last_peered":"2024-04-22T06:44:25.194323+0000","last_clean":"2024-04-22T06:44:25.194323+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:44:25.194323+0000","last_undegraded":"2024-04-22T06:44:25.194323+0000","last_fullsized":"2024-04-22T06:44:25.194323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023242+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:44:25.023242+0000","last_peered":"2024-04-22T06:44:25.023242+0000","last_clean":"2024-04-22T06:44:25.023242+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:44:25.023242+0000","last_undegraded":"2024-04-22T06:44:25.023242+0000","last_fullsized":"2024-04-22T06:44:25.023242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043912+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:44:25.043912+0000","last_peered":"2024-04-22T06:44:25.043912+0000","last_clean":"2024-04-22T06:44:25.043912+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:44:25.043912+0000","last_undegraded":"2024-04-22T06:44:25.043912+0000","last_fullsized":"2024-04-22T06:44:25.043912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044235+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:44:25.044235+0000","last_peered":"2024-04-22T06:44:25.044235+0000","last_clean":"2024-04-22T06:44:25.044235+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:44:25.044235+0000","last_undegraded":"2024-04-22T06:44:25.044235+0000","last_fullsized":"2024-04-22T06:44:25.044235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952874+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:44:24.952874+0000","last_peered":"2024-04-22T06:44:24.952874+0000","last_clean":"2024-04-22T06:44:24.952874+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:44:24.952874+0000","last_undegraded":"2024-04-22T06:44:24.952874+0000","last_fullsized":"2024-04-22T06:44:24.952874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953015+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:44:24.953015+0000","last_peered":"2024-04-22T06:44:24.953015+0000","last_clean":"2024-04-22T06:44:24.953015+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:44:24.953015+0000","last_undegraded":"2024-04-22T06:44:24.953015+0000","last_fullsized":"2024-04-22T06:44:24.953015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044498+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:44:25.044498+0000","last_peered":"2024-04-22T06:44:25.044498+0000","last_clean":"2024-04-22T06:44:25.044498+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:44:25.044498+0000","last_undegraded":"2024-04-22T06:44:25.044498+0000","last_fullsized":"2024-04-22T06:44:25.044498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952952+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:44:24.952952+0000","last_peered":"2024-04-22T06:44:24.952952+0000","last_clean":"2024-04-22T06:44:24.952952+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:44:24.952952+0000","last_undegraded":"2024-04-22T06:44:24.952952+0000","last_fullsized":"2024-04-22T06:44:24.952952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195201+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:44:25.195201+0000","last_peered":"2024-04-22T06:44:25.195201+0000","last_clean":"2024-04-22T06:44:25.195201+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:44:25.195201+0000","last_undegraded":"2024-04-22T06:44:25.195201+0000","last_fullsized":"2024-04-22T06:44:25.195201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195645+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:44:25.195645+0000","last_peered":"2024-04-22T06:44:25.195645+0000","last_clean":"2024-04-22T06:44:25.195645+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:44:25.195645+0000","last_undegraded":"2024-04-22T06:44:25.195645+0000","last_fullsized":"2024-04-22T06:44:25.195645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195419+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:44:25.195419+0000","last_peered":"2024-04-22T06:44:25.195419+0000","last_clean":"2024-04-22T06:44:25.195419+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:44:25.195419+0000","last_undegraded":"2024-04-22T06:44:25.195419+0000","last_fullsized":"2024-04-22T06:44:25.195419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.150698+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:44:25.150698+0000","last_peered":"2024-04-22T06:44:25.150698+0000","last_clean":"2024-04-22T06:44:25.150698+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:44:25.150698+0000","last_undegraded":"2024-04-22T06:44:25.150698+0000","last_fullsized":"2024-04-22T06:44:25.150698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195126+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:44:25.195126+0000","last_peered":"2024-04-22T06:44:25.195126+0000","last_clean":"2024-04-22T06:44:25.195126+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:44:25.195126+0000","last_undegraded":"2024-04-22T06:44:25.195126+0000","last_fullsized":"2024-04-22T06:44:25.195126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046237+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:44:25.046237+0000","last_peered":"2024-04-22T06:44:25.046237+0000","last_clean":"2024-04-22T06:44:25.046237+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:44:25.046237+0000","last_undegraded":"2024-04-22T06:44:25.046237+0000","last_fullsized":"2024-04-22T06:44:25.046237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195405+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:44:25.195405+0000","last_peered":"2024-04-22T06:44:25.195405+0000","last_clean":"2024-04-22T06:44:25.195405+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:44:25.195405+0000","last_undegraded":"2024-04-22T06:44:25.195405+0000","last_fullsized":"2024-04-22T06:44:25.195405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952951+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:44:24.952951+0000","last_peered":"2024-04-22T06:44:24.952951+0000","last_clean":"2024-04-22T06:44:24.952951+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:44:24.952951+0000","last_undegraded":"2024-04-22T06:44:24.952951+0000","last_fullsized":"2024-04-22T06:44:24.952951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195560+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:44:25.195560+0000","last_peered":"2024-04-22T06:44:25.195560+0000","last_clean":"2024-04-22T06:44:25.195560+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:44:25.195560+0000","last_undegraded":"2024-04-22T06:44:25.195560+0000","last_fullsized":"2024-04-22T06:44:25.195560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195368+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:44:25.195368+0000","last_peered":"2024-04-22T06:44:25.195368+0000","last_clean":"2024-04-22T06:44:25.195368+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:44:25.195368+0000","last_undegraded":"2024-04-22T06:44:25.195368+0000","last_fullsized":"2024-04-22T06:44:25.195368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194250+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:44:25.194250+0000","last_peered":"2024-04-22T06:44:25.194250+0000","last_clean":"2024-04-22T06:44:25.194250+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:44:25.194250+0000","last_undegraded":"2024-04-22T06:44:25.194250+0000","last_fullsized":"2024-04-22T06:44:25.194250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045832+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:44:25.045832+0000","last_peered":"2024-04-22T06:44:25.045832+0000","last_clean":"2024-04-22T06:44:25.045832+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:44:25.045832+0000","last_undegraded":"2024-04-22T06:44:25.045832+0000","last_fullsized":"2024-04-22T06:44:25.045832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953143+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:44:24.953143+0000","last_peered":"2024-04-22T06:44:24.953143+0000","last_clean":"2024-04-22T06:44:24.953143+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:44:24.953143+0000","last_undegraded":"2024-04-22T06:44:24.953143+0000","last_fullsized":"2024-04-22T06:44:24.953143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195715+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:44:25.195715+0000","last_peered":"2024-04-22T06:44:25.195715+0000","last_clean":"2024-04-22T06:44:25.195715+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:44:25.195715+0000","last_undegraded":"2024-04-22T06:44:25.195715+0000","last_fullsized":"2024-04-22T06:44:25.195715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044467+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:44:25.044467+0000","last_peered":"2024-04-22T06:44:25.044467+0000","last_clean":"2024-04-22T06:44:25.044467+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:44:25.044467+0000","last_undegraded":"2024-04-22T06:44:25.044467+0000","last_fullsized":"2024-04-22T06:44:25.044467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050646+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:44:25.050646+0000","last_peered":"2024-04-22T06:44:25.050646+0000","last_clean":"2024-04-22T06:44:25.050646+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:44:25.050646+0000","last_undegraded":"2024-04-22T06:44:25.050646+0000","last_fullsized":"2024-04-22T06:44:25.050646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044351+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:44:25.044351+0000","last_peered":"2024-04-22T06:44:25.044351+0000","last_clean":"2024-04-22T06:44:25.044351+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:44:25.044351+0000","last_undegraded":"2024-04-22T06:44:25.044351+0000","last_fullsized":"2024-04-22T06:44:25.044351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023389+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:44:25.023389+0000","last_peered":"2024-04-22T06:44:25.023389+0000","last_clean":"2024-04-22T06:44:25.023389+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:44:25.023389+0000","last_undegraded":"2024-04-22T06:44:25.023389+0000","last_fullsized":"2024-04-22T06:44:25.023389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045996+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:44:25.045996+0000","last_peered":"2024-04-22T06:44:25.045996+0000","last_clean":"2024-04-22T06:44:25.045996+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.045996+0000","last_undegraded":"2024-04-22T06:44:25.045996+0000","last_fullsized":"2024-04-22T06:44:25.045996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953024+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:44:24.953024+0000","last_peered":"2024-04-22T06:44:24.953024+0000","last_clean":"2024-04-22T06:44:24.953024+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:44:24.953024+0000","last_undegraded":"2024-04-22T06:44:24.953024+0000","last_fullsized":"2024-04-22T06:44:24.953024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":596216,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":832,"num_read_kb":523,"num_write":288,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1788648,"data_compressed":81528,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":465124,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":1253508,"data_compressed":46537,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858576,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":661888,"data_compressed":12626,"data_compressed_allocated":16384,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1048956,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":464749,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":530095,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":532759,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989876,"data_compressed":28203,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466109,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65553,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68217,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134390,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199654,"data_compressed":8030,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:29.706 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-22T06:44:29.706 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 -- 172.21.15.79:0/1925193710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8101b60 msgr2=0x7fe9e810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 --2- 172.21.15.79:0/1925193710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8101b60 0x7fe9e810dc90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009cd0 tx=0x7fe9e002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 -- 172.21.15.79:0/1925193710 shutdown_connections 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 --2- 172.21.15.79:0/1925193710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9e81011a0 0x7fe9e8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 --2- 172.21.15.79:0/1925193710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8101b60 0x7fe9e810dc90 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 --2- 172.21.15.79:0/1925193710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8105e30 0x7fe9e8106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 -- 172.21.15.79:0/1925193710 >> 172.21.15.79:0/1925193710 conn(0x7fe9e8077e30 msgr2=0x7fe9e8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 -- 172.21.15.79:0/1925193710 shutdown_connections 2024-04-22T06:44:29.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.842+0000 7fe9eda93640 1 -- 172.21.15.79:0/1925193710 wait complete. 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 Processor -- start 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 -- start start 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e81011a0 0x7fe9e8197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9e8101b60 0x7fe9e81978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 0x7fe9e8197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e81110e0 con 0x7fe9e8105e30 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e8110f60 con 0x7fe9e8101b60 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9eda93640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e8111260 con 0x7fe9e81011a0 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9e7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 0x7fe9e8197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9e7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 0x7fe9e8197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33254/0 (socket says 172.21.15.79:33254) 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.843+0000 7fe9e7fff640 1 -- 172.21.15.79:0/474041601 learned_addr learned my addr 172.21.15.79:0/474041601 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e77fe640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e81011a0 0x7fe9e8197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9deffd640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9e8101b60 0x7fe9e81978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e7fff640 1 -- 172.21.15.79:0/474041601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e81011a0 msgr2=0x7fe9e8197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e7fff640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e81011a0 0x7fe9e8197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e7fff640 1 -- 172.21.15.79:0/474041601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9e8101b60 msgr2=0x7fe9e81978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e7fff640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9e8101b60 0x7fe9e81978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e7fff640 1 -- 172.21.15.79:0/474041601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e81a6b70 con 0x7fe9e8105e30 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e7fff640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 0x7fe9e8197e10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0037040 tx=0x7fe9e0034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0041020 con 0x7fe9e8105e30 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e0034ca0 con 0x7fe9e8105e30 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e81a6e00 con 0x7fe9e8105e30 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0039040 con 0x7fe9e8105e30 2024-04-22T06:44:29.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.844+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e81a72c0 con 0x7fe9e8105e30 2024-04-22T06:44:29.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.845+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e8068de0 con 0x7fe9e8105e30 2024-04-22T06:44:29.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.846+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe9e0044020 con 0x7fe9e8105e30 2024-04-22T06:44:29.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.847+0000 7fe9e57fa640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9d40a3e40 0x7fe9d40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.847+0000 7fe9e77fe640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9d40a3e40 0x7fe9d40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.847+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fe9e00f26d0 con 0x7fe9e8105e30 2024-04-22T06:44:29.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.847+0000 7fe9e77fe640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9d40a3e40 0x7fe9d40a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009bd0 tx=0x7fe9d8005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:29.851+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9e004b020 con 0x7fe9e8105e30 2024-04-22T06:44:30.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.030+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9e8062390 con 0x7fe9d40a3e40 2024-04-22T06:44:30.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.037+0000 7fe9e57fa640 1 -- 172.21.15.79:0/474041601 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258005 (secure 0 0 0) 0x7fe9e8062390 con 0x7fe9d40a3e40 2024-04-22T06:44:30.038 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:30.039 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:44:30.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9d40a3e40 msgr2=0x7fe9d40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9d40a3e40 0x7fe9d40a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009bd0 tx=0x7fe9d8005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 msgr2=0x7fe9e8197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 0x7fe9e8197e10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0037040 tx=0x7fe9e0034650 comp rx=0 tx=0).stop 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 shutdown_connections 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9d40a3e40 0x7fe9d40a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe9e8101b60 0x7fe9e81978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe9e8105e30 0x7fe9e8197e10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 --2- 172.21.15.79:0/474041601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e81011a0 0x7fe9e8197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 >> 172.21.15.79:0/474041601 conn(0x7fe9e8077e30 msgr2=0x7fe9e80fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 shutdown_connections 2024-04-22T06:44:30.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.040+0000 7fe9eda93640 1 -- 172.21.15.79:0/474041601 wait complete. 2024-04-22T06:44:30.049 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:44:28.964824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186818,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":968,"num_read_kb":642,"num_write":471,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":866,"ondisk_log_size":866,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381756,"kb_used_data":11776,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560068,"statfs":{"total":1151940427776,"available":1151549509632,"internally_reserved":0,"allocated":12058624,"data_stored":8776841,"data_compressed":199080,"data_compressed_allocated":262144,"data_compressed_original":3096576,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990794"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953228+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:44:24.953228+0000","last_peered":"2024-04-22T06:44:24.953228+0000","last_clean":"2024-04-22T06:44:24.953228+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:44:24.953228+0000","last_undegraded":"2024-04-22T06:44:24.953228+0000","last_fullsized":"2024-04-22T06:44:24.953228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194515+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:44:25.194515+0000","last_peered":"2024-04-22T06:44:25.194515+0000","last_clean":"2024-04-22T06:44:25.194515+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:44:25.194515+0000","last_undegraded":"2024-04-22T06:44:25.194515+0000","last_fullsized":"2024-04-22T06:44:25.194515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194358+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:44:25.194358+0000","last_peered":"2024-04-22T06:44:25.194358+0000","last_clean":"2024-04-22T06:44:25.194358+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.194358+0000","last_undegraded":"2024-04-22T06:44:25.194358+0000","last_fullsized":"2024-04-22T06:44:25.194358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193440+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:44:25.193440+0000","last_peered":"2024-04-22T06:44:25.193440+0000","last_clean":"2024-04-22T06:44:25.193440+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:44:25.193440+0000","last_undegraded":"2024-04-22T06:44:25.193440+0000","last_fullsized":"2024-04-22T06:44:25.193440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044020+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:44:25.044020+0000","last_peered":"2024-04-22T06:44:25.044020+0000","last_clean":"2024-04-22T06:44:25.044020+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:44:25.044020+0000","last_undegraded":"2024-04-22T06:44:25.044020+0000","last_fullsized":"2024-04-22T06:44:25.044020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951989+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:44:24.951989+0000","last_peered":"2024-04-22T06:44:24.951989+0000","last_clean":"2024-04-22T06:44:24.951989+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:44:24.951989+0000","last_undegraded":"2024-04-22T06:44:24.951989+0000","last_fullsized":"2024-04-22T06:44:24.951989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194834+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:44:25.194834+0000","last_peered":"2024-04-22T06:44:25.194834+0000","last_clean":"2024-04-22T06:44:25.194834+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:44:25.194834+0000","last_undegraded":"2024-04-22T06:44:25.194834+0000","last_fullsized":"2024-04-22T06:44:25.194834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044085+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:44:25.044085+0000","last_peered":"2024-04-22T06:44:25.044085+0000","last_clean":"2024-04-22T06:44:25.044085+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:44:25.044085+0000","last_undegraded":"2024-04-22T06:44:25.044085+0000","last_fullsized":"2024-04-22T06:44:25.044085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023622+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:44:25.023622+0000","last_peered":"2024-04-22T06:44:25.023622+0000","last_clean":"2024-04-22T06:44:25.023622+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:44:25.023622+0000","last_undegraded":"2024-04-22T06:44:25.023622+0000","last_fullsized":"2024-04-22T06:44:25.023622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952422+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:44:24.952422+0000","last_peered":"2024-04-22T06:44:24.952422+0000","last_clean":"2024-04-22T06:44:24.952422+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:44:24.952422+0000","last_undegraded":"2024-04-22T06:44:24.952422+0000","last_fullsized":"2024-04-22T06:44:24.952422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195320+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:44:25.195320+0000","last_peered":"2024-04-22T06:44:25.195320+0000","last_clean":"2024-04-22T06:44:25.195320+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:44:25.195320+0000","last_undegraded":"2024-04-22T06:44:25.195320+0000","last_fullsized":"2024-04-22T06:44:25.195320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193429+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:44:25.193429+0000","last_peered":"2024-04-22T06:44:25.193429+0000","last_clean":"2024-04-22T06:44:25.193429+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:44:25.193429+0000","last_undegraded":"2024-04-22T06:44:25.193429+0000","last_fullsized":"2024-04-22T06:44:25.193429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046192+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:44:25.046192+0000","last_peered":"2024-04-22T06:44:25.046192+0000","last_clean":"2024-04-22T06:44:25.046192+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:44:25.046192+0000","last_undegraded":"2024-04-22T06:44:25.046192+0000","last_fullsized":"2024-04-22T06:44:25.046192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194092+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:44:25.194092+0000","last_peered":"2024-04-22T06:44:25.194092+0000","last_clean":"2024-04-22T06:44:25.194092+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:44:25.194092+0000","last_undegraded":"2024-04-22T06:44:25.194092+0000","last_fullsized":"2024-04-22T06:44:25.194092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193990+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:44:25.193990+0000","last_peered":"2024-04-22T06:44:25.193990+0000","last_clean":"2024-04-22T06:44:25.193990+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:44:25.193990+0000","last_undegraded":"2024-04-22T06:44:25.193990+0000","last_fullsized":"2024-04-22T06:44:25.193990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023187+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:44:25.023187+0000","last_peered":"2024-04-22T06:44:25.023187+0000","last_clean":"2024-04-22T06:44:25.023187+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:44:25.023187+0000","last_undegraded":"2024-04-22T06:44:25.023187+0000","last_fullsized":"2024-04-22T06:44:25.023187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953321+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:44:24.953321+0000","last_peered":"2024-04-22T06:44:24.953321+0000","last_clean":"2024-04-22T06:44:24.953321+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:44:24.953321+0000","last_undegraded":"2024-04-22T06:44:24.953321+0000","last_fullsized":"2024-04-22T06:44:24.953321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948506+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:44:24.948506+0000","last_peered":"2024-04-22T06:44:24.948506+0000","last_clean":"2024-04-22T06:44:24.948506+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:44:24.948506+0000","last_undegraded":"2024-04-22T06:44:24.948506+0000","last_fullsized":"2024-04-22T06:44:24.948506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195467+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:44:25.195467+0000","last_peered":"2024-04-22T06:44:25.195467+0000","last_clean":"2024-04-22T06:44:25.195467+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:44:25.195467+0000","last_undegraded":"2024-04-22T06:44:25.195467+0000","last_fullsized":"2024-04-22T06:44:25.195467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948609+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:44:24.948609+0000","last_peered":"2024-04-22T06:44:24.948609+0000","last_clean":"2024-04-22T06:44:24.948609+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:44:24.948609+0000","last_undegraded":"2024-04-22T06:44:24.948609+0000","last_fullsized":"2024-04-22T06:44:24.948609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044653+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:44:25.044653+0000","last_peered":"2024-04-22T06:44:25.044653+0000","last_clean":"2024-04-22T06:44:25.044653+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:44:25.044653+0000","last_undegraded":"2024-04-22T06:44:25.044653+0000","last_fullsized":"2024-04-22T06:44:25.044653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044718+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:44:25.044718+0000","last_peered":"2024-04-22T06:44:25.044718+0000","last_clean":"2024-04-22T06:44:25.044718+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:44:25.044718+0000","last_undegraded":"2024-04-22T06:44:25.044718+0000","last_fullsized":"2024-04-22T06:44:25.044718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194508+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:44:25.194508+0000","last_peered":"2024-04-22T06:44:25.194508+0000","last_clean":"2024-04-22T06:44:25.194508+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:44:25.194508+0000","last_undegraded":"2024-04-22T06:44:25.194508+0000","last_fullsized":"2024-04-22T06:44:25.194508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044333+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:44:25.044333+0000","last_peered":"2024-04-22T06:44:25.044333+0000","last_clean":"2024-04-22T06:44:25.044333+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:44:25.044333+0000","last_undegraded":"2024-04-22T06:44:25.044333+0000","last_fullsized":"2024-04-22T06:44:25.044333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023387+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:44:25.023387+0000","last_peered":"2024-04-22T06:44:25.023387+0000","last_clean":"2024-04-22T06:44:25.023387+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:44:25.023387+0000","last_undegraded":"2024-04-22T06:44:25.023387+0000","last_fullsized":"2024-04-22T06:44:25.023387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948348+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:44:24.948348+0000","last_peered":"2024-04-22T06:44:24.948348+0000","last_clean":"2024-04-22T06:44:24.948348+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:44:24.948348+0000","last_undegraded":"2024-04-22T06:44:24.948348+0000","last_fullsized":"2024-04-22T06:44:24.948348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'9","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.889110+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:44:25.889110+0000","last_peered":"2024-04-22T06:44:25.889110+0000","last_clean":"2024-04-22T06:44:25.889110+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:44:25.889110+0000","last_undegraded":"2024-04-22T06:44:25.889110+0000","last_fullsized":"2024-04-22T06:44:25.889110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043961+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:44:25.043961+0000","last_peered":"2024-04-22T06:44:25.043961+0000","last_clean":"2024-04-22T06:44:25.043961+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:44:25.043961+0000","last_undegraded":"2024-04-22T06:44:25.043961+0000","last_fullsized":"2024-04-22T06:44:25.043961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951916+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:44:24.951916+0000","last_peered":"2024-04-22T06:44:24.951916+0000","last_clean":"2024-04-22T06:44:24.951916+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:44:24.951916+0000","last_undegraded":"2024-04-22T06:44:24.951916+0000","last_fullsized":"2024-04-22T06:44:24.951916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050784+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:44:25.050784+0000","last_peered":"2024-04-22T06:44:25.050784+0000","last_clean":"2024-04-22T06:44:25.050784+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:44:25.050784+0000","last_undegraded":"2024-04-22T06:44:25.050784+0000","last_fullsized":"2024-04-22T06:44:25.050784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045482+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:44:25.045482+0000","last_peered":"2024-04-22T06:44:25.045482+0000","last_clean":"2024-04-22T06:44:25.045482+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:44:25.045482+0000","last_undegraded":"2024-04-22T06:44:25.045482+0000","last_fullsized":"2024-04-22T06:44:25.045482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044260+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:44:25.044260+0000","last_peered":"2024-04-22T06:44:25.044260+0000","last_clean":"2024-04-22T06:44:25.044260+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:44:25.044260+0000","last_undegraded":"2024-04-22T06:44:25.044260+0000","last_fullsized":"2024-04-22T06:44:25.044260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050589+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:44:25.050589+0000","last_peered":"2024-04-22T06:44:25.050589+0000","last_clean":"2024-04-22T06:44:25.050589+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:44:25.050589+0000","last_undegraded":"2024-04-22T06:44:25.050589+0000","last_fullsized":"2024-04-22T06:44:25.050589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948311+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:44:24.948311+0000","last_peered":"2024-04-22T06:44:24.948311+0000","last_clean":"2024-04-22T06:44:24.948311+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:44:24.948311+0000","last_undegraded":"2024-04-22T06:44:24.948311+0000","last_fullsized":"2024-04-22T06:44:24.948311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.197928+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:44:25.197928+0000","last_peered":"2024-04-22T06:44:25.197928+0000","last_clean":"2024-04-22T06:44:25.197928+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:44:25.197928+0000","last_undegraded":"2024-04-22T06:44:25.197928+0000","last_fullsized":"2024-04-22T06:44:25.197928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023518+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:44:25.023518+0000","last_peered":"2024-04-22T06:44:25.023518+0000","last_clean":"2024-04-22T06:44:25.023518+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:44:25.023518+0000","last_undegraded":"2024-04-22T06:44:25.023518+0000","last_fullsized":"2024-04-22T06:44:25.023518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195175+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:44:25.195175+0000","last_peered":"2024-04-22T06:44:25.195175+0000","last_clean":"2024-04-22T06:44:25.195175+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:44:25.195175+0000","last_undegraded":"2024-04-22T06:44:25.195175+0000","last_fullsized":"2024-04-22T06:44:25.195175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194428+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:44:25.194428+0000","last_peered":"2024-04-22T06:44:25.194428+0000","last_clean":"2024-04-22T06:44:25.194428+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:44:25.194428+0000","last_undegraded":"2024-04-22T06:44:25.194428+0000","last_fullsized":"2024-04-22T06:44:25.194428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050921+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:44:25.050921+0000","last_peered":"2024-04-22T06:44:25.050921+0000","last_clean":"2024-04-22T06:44:25.050921+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:44:25.050921+0000","last_undegraded":"2024-04-22T06:44:25.050921+0000","last_fullsized":"2024-04-22T06:44:25.050921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023508+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:44:25.023508+0000","last_peered":"2024-04-22T06:44:25.023508+0000","last_clean":"2024-04-22T06:44:25.023508+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:44:25.023508+0000","last_undegraded":"2024-04-22T06:44:25.023508+0000","last_fullsized":"2024-04-22T06:44:25.023508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":301,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.889056+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:44:25.889056+0000","last_peered":"2024-04-22T06:44:25.889056+0000","last_clean":"2024-04-22T06:44:25.889056+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:44:25.889056+0000","last_undegraded":"2024-04-22T06:44:25.889056+0000","last_fullsized":"2024-04-22T06:44:25.889056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65553,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":186,"num_write":35,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044125+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:44:25.044125+0000","last_peered":"2024-04-22T06:44:25.044125+0000","last_clean":"2024-04-22T06:44:25.044125+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:44:25.044125+0000","last_undegraded":"2024-04-22T06:44:25.044125+0000","last_fullsized":"2024-04-22T06:44:25.044125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195568+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:44:25.195568+0000","last_peered":"2024-04-22T06:44:25.195568+0000","last_clean":"2024-04-22T06:44:25.195568+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:44:25.195568+0000","last_undegraded":"2024-04-22T06:44:25.195568+0000","last_fullsized":"2024-04-22T06:44:25.195568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045677+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:44:25.045677+0000","last_peered":"2024-04-22T06:44:25.045677+0000","last_clean":"2024-04-22T06:44:25.045677+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:44:25.045677+0000","last_undegraded":"2024-04-22T06:44:25.045677+0000","last_fullsized":"2024-04-22T06:44:25.045677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050734+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:44:25.050734+0000","last_peered":"2024-04-22T06:44:25.050734+0000","last_clean":"2024-04-22T06:44:25.050734+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:44:25.050734+0000","last_undegraded":"2024-04-22T06:44:25.050734+0000","last_fullsized":"2024-04-22T06:44:25.050734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195155+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:44:25.195155+0000","last_peered":"2024-04-22T06:44:25.195155+0000","last_clean":"2024-04-22T06:44:25.195155+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:44:25.195155+0000","last_undegraded":"2024-04-22T06:44:25.195155+0000","last_fullsized":"2024-04-22T06:44:25.195155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":150,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.888981+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:44:25.888981+0000","last_peered":"2024-04-22T06:44:25.888981+0000","last_clean":"2024-04-22T06:44:25.888981+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:44:25.888981+0000","last_undegraded":"2024-04-22T06:44:25.888981+0000","last_fullsized":"2024-04-22T06:44:25.888981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194722+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:44:25.194722+0000","last_peered":"2024-04-22T06:44:25.194722+0000","last_clean":"2024-04-22T06:44:25.194722+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:44:25.194722+0000","last_undegraded":"2024-04-22T06:44:25.194722+0000","last_fullsized":"2024-04-22T06:44:25.194722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043891+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:44:25.043891+0000","last_peered":"2024-04-22T06:44:25.043891+0000","last_clean":"2024-04-22T06:44:25.043891+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:44:25.043891+0000","last_undegraded":"2024-04-22T06:44:25.043891+0000","last_fullsized":"2024-04-22T06:44:25.043891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050670+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:44:25.050670+0000","last_peered":"2024-04-22T06:44:25.050670+0000","last_clean":"2024-04-22T06:44:25.050670+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:44:25.050670+0000","last_undegraded":"2024-04-22T06:44:25.050670+0000","last_fullsized":"2024-04-22T06:44:25.050670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044215+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:44:25.044215+0000","last_peered":"2024-04-22T06:44:25.044215+0000","last_clean":"2024-04-22T06:44:25.044215+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:44:25.044215+0000","last_undegraded":"2024-04-22T06:44:25.044215+0000","last_fullsized":"2024-04-22T06:44:25.044215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953070+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:44:24.953070+0000","last_peered":"2024-04-22T06:44:24.953070+0000","last_clean":"2024-04-22T06:44:24.953070+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:44:24.953070+0000","last_undegraded":"2024-04-22T06:44:24.953070+0000","last_fullsized":"2024-04-22T06:44:24.953070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948439+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:44:24.948439+0000","last_peered":"2024-04-22T06:44:24.948439+0000","last_clean":"2024-04-22T06:44:24.948439+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:44:24.948439+0000","last_undegraded":"2024-04-22T06:44:24.948439+0000","last_fullsized":"2024-04-22T06:44:24.948439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952192+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:44:24.952192+0000","last_peered":"2024-04-22T06:44:24.952192+0000","last_clean":"2024-04-22T06:44:24.952192+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:44:24.952192+0000","last_undegraded":"2024-04-22T06:44:24.952192+0000","last_fullsized":"2024-04-22T06:44:24.952192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195007+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:44:25.195007+0000","last_peered":"2024-04-22T06:44:25.195007+0000","last_clean":"2024-04-22T06:44:25.195007+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:44:25.195007+0000","last_undegraded":"2024-04-22T06:44:25.195007+0000","last_fullsized":"2024-04-22T06:44:25.195007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952314+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:44:24.952314+0000","last_peered":"2024-04-22T06:44:24.952314+0000","last_clean":"2024-04-22T06:44:24.952314+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:44:24.952314+0000","last_undegraded":"2024-04-22T06:44:24.952314+0000","last_fullsized":"2024-04-22T06:44:24.952314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050823+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:44:25.050823+0000","last_peered":"2024-04-22T06:44:25.050823+0000","last_clean":"2024-04-22T06:44:25.050823+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:44:25.050823+0000","last_undegraded":"2024-04-22T06:44:25.050823+0000","last_fullsized":"2024-04-22T06:44:25.050823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":145,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193813+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:44:25.193813+0000","last_peered":"2024-04-22T06:44:25.193813+0000","last_clean":"2024-04-22T06:44:25.193813+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:44:25.193813+0000","last_undegraded":"2024-04-22T06:44:25.193813+0000","last_fullsized":"2024-04-22T06:44:25.193813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045411+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:44:25.045411+0000","last_peered":"2024-04-22T06:44:25.045411+0000","last_clean":"2024-04-22T06:44:25.045411+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:44:25.045411+0000","last_undegraded":"2024-04-22T06:44:25.045411+0000","last_fullsized":"2024-04-22T06:44:25.045411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193355+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:44:25.193355+0000","last_peered":"2024-04-22T06:44:25.193355+0000","last_clean":"2024-04-22T06:44:25.193355+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:44:25.193355+0000","last_undegraded":"2024-04-22T06:44:25.193355+0000","last_fullsized":"2024-04-22T06:44:25.193355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194796+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:44:25.194796+0000","last_peered":"2024-04-22T06:44:25.194796+0000","last_clean":"2024-04-22T06:44:25.194796+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:44:25.194796+0000","last_undegraded":"2024-04-22T06:44:25.194796+0000","last_fullsized":"2024-04-22T06:44:25.194796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045286+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:44:25.045286+0000","last_peered":"2024-04-22T06:44:25.045286+0000","last_clean":"2024-04-22T06:44:25.045286+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:44:25.045286+0000","last_undegraded":"2024-04-22T06:44:25.045286+0000","last_fullsized":"2024-04-22T06:44:25.045286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194945+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:44:25.194945+0000","last_peered":"2024-04-22T06:44:25.194945+0000","last_clean":"2024-04-22T06:44:25.194945+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:44:25.194945+0000","last_undegraded":"2024-04-22T06:44:25.194945+0000","last_fullsized":"2024-04-22T06:44:25.194945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044263+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:44:25.044263+0000","last_peered":"2024-04-22T06:44:25.044263+0000","last_clean":"2024-04-22T06:44:25.044263+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:44:25.044263+0000","last_undegraded":"2024-04-22T06:44:25.044263+0000","last_fullsized":"2024-04-22T06:44:25.044263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045949+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:44:25.045949+0000","last_peered":"2024-04-22T06:44:25.045949+0000","last_clean":"2024-04-22T06:44:25.045949+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:44:25.045949+0000","last_undegraded":"2024-04-22T06:44:25.045949+0000","last_fullsized":"2024-04-22T06:44:25.045949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.156832+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:44:25.156832+0000","last_peered":"2024-04-22T06:44:25.156832+0000","last_clean":"2024-04-22T06:44:25.156832+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:44:25.156832+0000","last_undegraded":"2024-04-22T06:44:25.156832+0000","last_fullsized":"2024-04-22T06:44:25.156832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044531+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:44:25.044531+0000","last_peered":"2024-04-22T06:44:25.044531+0000","last_clean":"2024-04-22T06:44:25.044531+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:44:25.044531+0000","last_undegraded":"2024-04-22T06:44:25.044531+0000","last_fullsized":"2024-04-22T06:44:25.044531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193293+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:44:25.193293+0000","last_peered":"2024-04-22T06:44:25.193293+0000","last_clean":"2024-04-22T06:44:25.193293+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:44:25.193293+0000","last_undegraded":"2024-04-22T06:44:25.193293+0000","last_fullsized":"2024-04-22T06:44:25.193293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194323+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:44:25.194323+0000","last_peered":"2024-04-22T06:44:25.194323+0000","last_clean":"2024-04-22T06:44:25.194323+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:44:25.194323+0000","last_undegraded":"2024-04-22T06:44:25.194323+0000","last_fullsized":"2024-04-22T06:44:25.194323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023242+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:44:25.023242+0000","last_peered":"2024-04-22T06:44:25.023242+0000","last_clean":"2024-04-22T06:44:25.023242+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:44:25.023242+0000","last_undegraded":"2024-04-22T06:44:25.023242+0000","last_fullsized":"2024-04-22T06:44:25.023242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043912+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:44:25.043912+0000","last_peered":"2024-04-22T06:44:25.043912+0000","last_clean":"2024-04-22T06:44:25.043912+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:44:25.043912+0000","last_undegraded":"2024-04-22T06:44:25.043912+0000","last_fullsized":"2024-04-22T06:44:25.043912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044235+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:44:25.044235+0000","last_peered":"2024-04-22T06:44:25.044235+0000","last_clean":"2024-04-22T06:44:25.044235+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:44:25.044235+0000","last_undegraded":"2024-04-22T06:44:25.044235+0000","last_fullsized":"2024-04-22T06:44:25.044235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952874+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:44:24.952874+0000","last_peered":"2024-04-22T06:44:24.952874+0000","last_clean":"2024-04-22T06:44:24.952874+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:44:24.952874+0000","last_undegraded":"2024-04-22T06:44:24.952874+0000","last_fullsized":"2024-04-22T06:44:24.952874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953015+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:44:24.953015+0000","last_peered":"2024-04-22T06:44:24.953015+0000","last_clean":"2024-04-22T06:44:24.953015+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:44:24.953015+0000","last_undegraded":"2024-04-22T06:44:24.953015+0000","last_fullsized":"2024-04-22T06:44:24.953015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044498+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:44:25.044498+0000","last_peered":"2024-04-22T06:44:25.044498+0000","last_clean":"2024-04-22T06:44:25.044498+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:44:25.044498+0000","last_undegraded":"2024-04-22T06:44:25.044498+0000","last_fullsized":"2024-04-22T06:44:25.044498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952952+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:44:24.952952+0000","last_peered":"2024-04-22T06:44:24.952952+0000","last_clean":"2024-04-22T06:44:24.952952+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:44:24.952952+0000","last_undegraded":"2024-04-22T06:44:24.952952+0000","last_fullsized":"2024-04-22T06:44:24.952952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195201+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:44:25.195201+0000","last_peered":"2024-04-22T06:44:25.195201+0000","last_clean":"2024-04-22T06:44:25.195201+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:44:25.195201+0000","last_undegraded":"2024-04-22T06:44:25.195201+0000","last_fullsized":"2024-04-22T06:44:25.195201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195645+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:44:25.195645+0000","last_peered":"2024-04-22T06:44:25.195645+0000","last_clean":"2024-04-22T06:44:25.195645+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:44:25.195645+0000","last_undegraded":"2024-04-22T06:44:25.195645+0000","last_fullsized":"2024-04-22T06:44:25.195645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195419+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:44:25.195419+0000","last_peered":"2024-04-22T06:44:25.195419+0000","last_clean":"2024-04-22T06:44:25.195419+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:44:25.195419+0000","last_undegraded":"2024-04-22T06:44:25.195419+0000","last_fullsized":"2024-04-22T06:44:25.195419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.150698+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:44:25.150698+0000","last_peered":"2024-04-22T06:44:25.150698+0000","last_clean":"2024-04-22T06:44:25.150698+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:44:25.150698+0000","last_undegraded":"2024-04-22T06:44:25.150698+0000","last_fullsized":"2024-04-22T06:44:25.150698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195126+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:44:25.195126+0000","last_peered":"2024-04-22T06:44:25.195126+0000","last_clean":"2024-04-22T06:44:25.195126+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:44:25.195126+0000","last_undegraded":"2024-04-22T06:44:25.195126+0000","last_fullsized":"2024-04-22T06:44:25.195126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046237+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:44:25.046237+0000","last_peered":"2024-04-22T06:44:25.046237+0000","last_clean":"2024-04-22T06:44:25.046237+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:44:25.046237+0000","last_undegraded":"2024-04-22T06:44:25.046237+0000","last_fullsized":"2024-04-22T06:44:25.046237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195405+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:44:25.195405+0000","last_peered":"2024-04-22T06:44:25.195405+0000","last_clean":"2024-04-22T06:44:25.195405+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:44:25.195405+0000","last_undegraded":"2024-04-22T06:44:25.195405+0000","last_fullsized":"2024-04-22T06:44:25.195405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952951+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:44:24.952951+0000","last_peered":"2024-04-22T06:44:24.952951+0000","last_clean":"2024-04-22T06:44:24.952951+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:44:24.952951+0000","last_undegraded":"2024-04-22T06:44:24.952951+0000","last_fullsized":"2024-04-22T06:44:24.952951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195560+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:44:25.195560+0000","last_peered":"2024-04-22T06:44:25.195560+0000","last_clean":"2024-04-22T06:44:25.195560+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:44:25.195560+0000","last_undegraded":"2024-04-22T06:44:25.195560+0000","last_fullsized":"2024-04-22T06:44:25.195560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195368+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:44:25.195368+0000","last_peered":"2024-04-22T06:44:25.195368+0000","last_clean":"2024-04-22T06:44:25.195368+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:44:25.195368+0000","last_undegraded":"2024-04-22T06:44:25.195368+0000","last_fullsized":"2024-04-22T06:44:25.195368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194250+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:44:25.194250+0000","last_peered":"2024-04-22T06:44:25.194250+0000","last_clean":"2024-04-22T06:44:25.194250+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:44:25.194250+0000","last_undegraded":"2024-04-22T06:44:25.194250+0000","last_fullsized":"2024-04-22T06:44:25.194250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045832+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:44:25.045832+0000","last_peered":"2024-04-22T06:44:25.045832+0000","last_clean":"2024-04-22T06:44:25.045832+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:44:25.045832+0000","last_undegraded":"2024-04-22T06:44:25.045832+0000","last_fullsized":"2024-04-22T06:44:25.045832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953143+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:44:24.953143+0000","last_peered":"2024-04-22T06:44:24.953143+0000","last_clean":"2024-04-22T06:44:24.953143+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:44:24.953143+0000","last_undegraded":"2024-04-22T06:44:24.953143+0000","last_fullsized":"2024-04-22T06:44:24.953143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195715+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:44:25.195715+0000","last_peered":"2024-04-22T06:44:25.195715+0000","last_clean":"2024-04-22T06:44:25.195715+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:44:25.195715+0000","last_undegraded":"2024-04-22T06:44:25.195715+0000","last_fullsized":"2024-04-22T06:44:25.195715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044467+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:44:25.044467+0000","last_peered":"2024-04-22T06:44:25.044467+0000","last_clean":"2024-04-22T06:44:25.044467+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:44:25.044467+0000","last_undegraded":"2024-04-22T06:44:25.044467+0000","last_fullsized":"2024-04-22T06:44:25.044467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050646+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:44:25.050646+0000","last_peered":"2024-04-22T06:44:25.050646+0000","last_clean":"2024-04-22T06:44:25.050646+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:44:25.050646+0000","last_undegraded":"2024-04-22T06:44:25.050646+0000","last_fullsized":"2024-04-22T06:44:25.050646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044351+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:44:25.044351+0000","last_peered":"2024-04-22T06:44:25.044351+0000","last_clean":"2024-04-22T06:44:25.044351+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:44:25.044351+0000","last_undegraded":"2024-04-22T06:44:25.044351+0000","last_fullsized":"2024-04-22T06:44:25.044351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023389+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:44:25.023389+0000","last_peered":"2024-04-22T06:44:25.023389+0000","last_clean":"2024-04-22T06:44:25.023389+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:44:25.023389+0000","last_undegraded":"2024-04-22T06:44:25.023389+0000","last_fullsized":"2024-04-22T06:44:25.023389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045996+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:44:25.045996+0000","last_peered":"2024-04-22T06:44:25.045996+0000","last_clean":"2024-04-22T06:44:25.045996+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.045996+0000","last_undegraded":"2024-04-22T06:44:25.045996+0000","last_fullsized":"2024-04-22T06:44:25.045996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953024+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:44:24.953024+0000","last_peered":"2024-04-22T06:44:24.953024+0000","last_clean":"2024-04-22T06:44:24.953024+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:44:24.953024+0000","last_undegraded":"2024-04-22T06:44:24.953024+0000","last_fullsized":"2024-04-22T06:44:24.953024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":596216,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":832,"num_read_kb":523,"num_write":288,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1788648,"data_compressed":81528,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":465124,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":1253508,"data_compressed":46537,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858576,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":661888,"data_compressed":12626,"data_compressed_allocated":16384,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1048956,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":464749,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":530095,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":457093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":532759,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989876,"data_compressed":28203,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466109,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1048108,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65553,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68217,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134390,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199654,"data_compressed":8030,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:30.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:30.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.189+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2373975807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf41011a0 msgr2=0x7f2bf4101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.189+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2373975807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf41011a0 0x7f2bf4101620 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009930 tx=0x7f2be402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:30.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.189+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2373975807 shutdown_connections 2024-04-22T06:44:30.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.189+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2373975807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bf4101b60 0x7f2bf410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.189+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2373975807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf41011a0 0x7f2bf4101620 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2373975807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf4105e30 0x7f2bf4106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2373975807 >> 172.21.15.79:0/2373975807 conn(0x7f2bf4077e30 msgr2=0x7f2bf4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2373975807 shutdown_connections 2024-04-22T06:44:30.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2373975807 wait complete. 2024-04-22T06:44:30.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 Processor -- start 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- start start 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 0x7f2bf41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bf4101b60 0x7f2bf4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 0x7f2bf419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf4112240 con 0x7f2bf4105e30 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41120c0 con 0x7f2bf41011a0 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.190+0000 7f2bf9f8e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41123c0 con 0x7f2bf4101b60 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 0x7f2bf419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 0x7f2bf41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 0x7f2bf41986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:52670/0 (socket says 172.21.15.79:52670) 2024-04-22T06:44:30.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf37fe640 1 -- 172.21.15.79:0/2142795881 learned_addr learned my addr 172.21.15.79:0/2142795881 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 0x7f2bf419cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33264/0 (socket says 172.21.15.79:33264) 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf2ffd640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bf4101b60 0x7f2bf4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 -- 172.21.15.79:0/2142795881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bf4101b60 msgr2=0x7f2bf4198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bf4101b60 0x7f2bf4198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 -- 172.21.15.79:0/2142795881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 msgr2=0x7f2bf41986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 0x7f2bf41986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 -- 172.21.15.79:0/2142795881 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf419d690 con 0x7f2bf4105e30 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf37fe640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 0x7f2bf41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf3fff640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 0x7f2bf419cf60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00cab0 tx=0x7f2bec00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec00ae20 con 0x7f2bf4105e30 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bec00f900 con 0x7f2bf4105e30 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf419d980 con 0x7f2bf4105e30 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0189f0 con 0x7f2bf4105e30 2024-04-22T06:44:30.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.191+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf41a5210 con 0x7f2bf4105e30 2024-04-22T06:44:30.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.192+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf4068de0 con 0x7f2bf4105e30 2024-04-22T06:44:30.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.193+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f2bec00f450 con 0x7f2bf4105e30 2024-04-22T06:44:30.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.194+0000 7f2bf0ff9640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2bdc0a3e40 0x7f2bdc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.194+0000 7f2bf37fe640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2bdc0a3e40 0x7f2bdc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.194+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f2bec0d10d0 con 0x7f2bf4105e30 2024-04-22T06:44:30.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.195+0000 7f2bf37fe640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2bdc0a3e40 0x7f2bdc0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2be8005fd0 tx=0x7f2be800b480 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:30.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.199+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bec014030 con 0x7f2bf4105e30 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2219793967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:30.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:29 smithi079 ceph-mon[27946]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:30.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:30.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2219793967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:30.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:30.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:30.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:30.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:29 smithi190 ceph-mon[31421]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:30.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.376+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bf4109390 con 0x7f2bf4105e30 2024-04-22T06:44:30.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.378+0000 7f2bf0ff9640 1 -- 172.21.15.79:0/2142795881 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7f2bec09d5f0 con 0x7f2bf4105e30 2024-04-22T06:44:30.376 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:30.377 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:30.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2bdc0a3e40 msgr2=0x7f2bdc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2bdc0a3e40 0x7f2bdc0a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2be8005fd0 tx=0x7f2be800b480 comp rx=0 tx=0).stop 2024-04-22T06:44:30.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 msgr2=0x7f2bf419cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 0x7f2bf419cf60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00cab0 tx=0x7f2bec00cf70 comp rx=0 tx=0).stop 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 shutdown_connections 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2bf4105e30 0x7f2bf419cf60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bf4101b60 0x7f2bf4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2bdc0a3e40 0x7f2bdc0a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 --2- 172.21.15.79:0/2142795881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bf41011a0 0x7f2bf41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 >> 172.21.15.79:0/2142795881 conn(0x7f2bf4077e30 msgr2=0x7f2bf40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 shutdown_connections 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.380+0000 7f2bf9f8e640 1 -- 172.21.15.79:0/2142795881 wait complete. 2024-04-22T06:44:30.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:30.390 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:30.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2219793967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: Updating smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: Updating smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: Updating smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:44:30.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:29 smithi193 ceph-mon[31815]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:30.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- 172.21.15.79:0/1939299873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 msgr2=0x7f3d4c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- 172.21.15.79:0/1939299873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c101480 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3d44009900 tx=0x7f3d4402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:30.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- 172.21.15.79:0/1939299873 shutdown_connections 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- 172.21.15.79:0/1939299873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d4c106170 0x7f3d4c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- 172.21.15.79:0/1939299873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d4c101a50 0x7f3d4c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- 172.21.15.79:0/1939299873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c101480 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- 172.21.15.79:0/1939299873 >> 172.21.15.79:0/1939299873 conn(0x7f3d4c077eb0 msgr2=0x7f3d4c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- 172.21.15.79:0/1939299873 shutdown_connections 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- 172.21.15.79:0/1939299873 wait complete. 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 Processor -- start 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- start start 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d4c106170 0x7f3d4c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d4c19bf10 0x7f3d4c19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c1110d0 con 0x7f3d4c1010a0 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c110f50 con 0x7f3d4c106170 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.493+0000 7f3d53be6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c111250 con 0x7f3d4c19bf10 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33276/0 (socket says 172.21.15.79:33276) 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 -- 172.21.15.79:0/2322964772 learned_addr learned my addr 172.21.15.79:0/2322964772 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51181640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d4c106170 0x7f3d4c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d52183640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d4c19bf10 0x7f3d4c19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 -- 172.21.15.79:0/2322964772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d4c19bf10 msgr2=0x7f3d4c19c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d4c19bf10 0x7f3d4c19c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 -- 172.21.15.79:0/2322964772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d4c106170 msgr2=0x7f3d4c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d4c106170 0x7f3d4c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 -- 172.21.15.79:0/2322964772 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4c19ca00 con 0x7f3d4c1010a0 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d51982640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c197650 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3d44005940 tx=0x7f3d4402c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:30.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d44043020 con 0x7f3d4c1010a0 2024-04-22T06:44:30.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4402ca70 con 0x7f3d4c1010a0 2024-04-22T06:44:30.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4c19cc90 con 0x7f3d4c1010a0 2024-04-22T06:44:30.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d440039f0 con 0x7f3d4c1010a0 2024-04-22T06:44:30.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.494+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4c1a44b0 con 0x7f3d4c1010a0 2024-04-22T06:44:30.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.495+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d4c0691c0 con 0x7f3d4c1010a0 2024-04-22T06:44:30.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.496+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3d44046020 con 0x7f3d4c1010a0 2024-04-22T06:44:30.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.497+0000 7f3d4affd640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3d340a3e40 0x7f3d340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.497+0000 7f3d51181640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3d340a3e40 0x7f3d340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.497+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3d440f29b0 con 0x7f3d4c1010a0 2024-04-22T06:44:30.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.497+0000 7f3d51181640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3d340a3e40 0x7f3d340a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0ff9a0 tx=0x7f3d40005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:30.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.501+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d4404d020 con 0x7f3d4c1010a0 2024-04-22T06:44:30.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.676+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d4c109400 con 0x7f3d4c1010a0 2024-04-22T06:44:30.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.676+0000 7f3d4affd640 1 -- 172.21.15.79:0/2322964772 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f3d4402cc20 con 0x7f3d4c1010a0 2024-04-22T06:44:30.675 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3d340a3e40 msgr2=0x7f3d340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3d340a3e40 0x7f3d340a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0ff9a0 tx=0x7f3d40005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 msgr2=0x7f3d4c197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c197650 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3d44005940 tx=0x7f3d4402c5f0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 shutdown_connections 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3d340a3e40 0x7f3d340a6300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d4c106170 0x7f3d4c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d4c19bf10 0x7f3d4c19c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 --2- 172.21.15.79:0/2322964772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d4c1010a0 0x7f3d4c197650 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 >> 172.21.15.79:0/2322964772 conn(0x7f3d4c077eb0 msgr2=0x7f3d4c107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 shutdown_connections 2024-04-22T06:44:30.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.678+0000 7f3d53be6640 1 -- 172.21.15.79:0/2322964772 wait complete. 2024-04-22T06:44:30.688 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 -- 172.21.15.79:0/1076716994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe8102b90 msgr2=0x7f3fe8106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1076716994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe8102b90 0x7f3fe8106bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009900 tx=0x7f3fe402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 -- 172.21.15.79:0/1076716994 shutdown_connections 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1076716994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81072b0 0x7f3fe810ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1076716994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe8102b90 0x7f3fe8106bd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1076716994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fe81021e0 0x7f3fe81025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 -- 172.21.15.79:0/1076716994 >> 172.21.15.79:0/1076716994 conn(0x7f3fe80fda60 msgr2=0x7f3fe80ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 -- 172.21.15.79:0/1076716994 shutdown_connections 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 -- 172.21.15.79:0/1076716994 wait complete. 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.790+0000 7f3feff3d640 1 Processor -- start 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 -- start start 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fe81021e0 0x7f3fe819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 0x7f3fe819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81985d0 0x7f3fe8198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81121f0 con 0x7f3fe81072b0 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8112070 con 0x7f3fe81021e0 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8112370 con 0x7f3fe81985d0 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 0x7f3fe819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 0x7f3fe819e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33292/0 (socket says 172.21.15.79:33292) 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 -- 172.21.15.79:0/1257916789 learned_addr learned my addr 172.21.15.79:0/1257916789 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:30.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fee4da640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81985d0 0x7f3fe8198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fedcd9640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fe81021e0 0x7f3fe819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 -- 172.21.15.79:0/1257916789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81985d0 msgr2=0x7f3fe8198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81985d0 0x7f3fe8198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 -- 172.21.15.79:0/1257916789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fe81021e0 msgr2=0x7f3fe819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fe81021e0 0x7f3fe819e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 -- 172.21.15.79:0/1257916789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe81990f0 con 0x7f3fe81072b0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fed4d8640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 0x7f3fe819e960 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4036040 tx=0x7f3fe402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4043020 con 0x7f3fe81072b0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe402cbe0 con 0x7f3fe81072b0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe8199380 con 0x7f3fe81072b0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.791+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4003bf0 con 0x7f3fe81072b0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.792+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe81a55d0 con 0x7f3fe81072b0 2024-04-22T06:44:30.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.792+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe80691c0 con 0x7f3fe81072b0 2024-04-22T06:44:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.793+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3fe4046020 con 0x7f3fe81072b0 2024-04-22T06:44:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.794+0000 7f3fdaffd640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3fd40a3e40 0x7f3fd40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:30.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.794+0000 7f3fedcd9640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3fd40a3e40 0x7f3fd40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:30.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.794+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3fe40f30e0 con 0x7f3fe81072b0 2024-04-22T06:44:30.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.795+0000 7f3fedcd9640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3fd40a3e40 0x7f3fd40a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc005d00 tx=0x7f3fdc00b410 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:30.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.798+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fe404d020 con 0x7f3fe81072b0 2024-04-22T06:44:30.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.978+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3fe810a4e0 con 0x7f3fe81072b0 2024-04-22T06:44:30.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.979+0000 7f3fdaffd640 1 -- 172.21.15.79:0/1257916789 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f3fe40bf710 con 0x7f3fe81072b0 2024-04-22T06:44:30.977 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.980+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3fd40a3e40 msgr2=0x7f3fd40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.980+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3fd40a3e40 0x7f3fd40a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc005d00 tx=0x7f3fdc00b410 comp rx=0 tx=0).stop 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.980+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 msgr2=0x7f3fe819e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.980+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 0x7f3fe819e960 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4036040 tx=0x7f3fe402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 shutdown_connections 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81985d0 0x7f3fe8198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fe81072b0 0x7f3fe819e960 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3fd40a3e40 0x7f3fd40a6300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 --2- 172.21.15.79:0/1257916789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fe81021e0 0x7f3fe819e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 >> 172.21.15.79:0/1257916789 conn(0x7f3fe80fda60 msgr2=0x7f3fe80fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:30.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 shutdown_connections 2024-04-22T06:44:30.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:30.981+0000 7f3feff3d640 1 -- 172.21.15.79:0/1257916789 wait complete. 2024-04-22T06:44:30.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 -- 172.21.15.79:0/3511356257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778100af0 msgr2=0x7fa778100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 --2- 172.21.15.79:0/3511356257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778100af0 0x7fa778100ed0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa768009900 tx=0x7fa76802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 -- 172.21.15.79:0/3511356257 shutdown_connections 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 --2- 172.21.15.79:0/3511356257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7781014a0 0x7fa778105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 --2- 172.21.15.79:0/3511356257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa778105970 0x7fa77810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 --2- 172.21.15.79:0/3511356257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778100af0 0x7fa778100ed0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 -- 172.21.15.79:0/3511356257 >> 172.21.15.79:0/3511356257 conn(0x7fa7780760e0 msgr2=0x7fa7780764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 -- 172.21.15.79:0/3511356257 shutdown_connections 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.092+0000 7fa77ce62640 1 -- 172.21.15.79:0/3511356257 wait complete. 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 Processor -- start 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 -- start start 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa778100af0 0x7fa778197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 0x7fa778197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77819bf00 0x7fa77819c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa7781108a0 con 0x7fa778105970 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa778110720 con 0x7fa778100af0 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77ce62640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa778110a20 con 0x7fa77819bf00 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa76ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 0x7fa778197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa76ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 0x7fa778197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33310/0 (socket says 172.21.15.79:33310) 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa76ffff640 1 -- 172.21.15.79:0/1385588522 learned_addr learned my addr 172.21.15.79:0/1385588522 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.093+0000 7fa77659c640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa778100af0 0x7fa778197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76ffff640 1 -- 172.21.15.79:0/1385588522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77819bf00 msgr2=0x7fa77819c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa776d9d640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77819bf00 0x7fa77819c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76ffff640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77819bf00 0x7fa77819c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76ffff640 1 -- 172.21.15.79:0/1385588522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa778100af0 msgr2=0x7fa778197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76ffff640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa778100af0 0x7fa778197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76ffff640 1 -- 172.21.15.79:0/1385588522 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77819c9f0 con 0x7fa778105970 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa776d9d640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77819bf00 0x7fa77819c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76ffff640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 0x7fa778197b80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa760005fd0 tx=0x7fa760009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760004020 con 0x7fa778105970 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa760007c50 con 0x7fa778105970 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760003a50 con 0x7fa778105970 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77819cce0 con 0x7fa778105970 2024-04-22T06:44:31.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.094+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7780feb70 con 0x7fa778105970 2024-04-22T06:44:31.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.095+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7780691c0 con 0x7fa778105970 2024-04-22T06:44:31.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.096+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fa760003bf0 con 0x7fa778105970 2024-04-22T06:44:31.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.097+0000 7fa76f7fe640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa7640a3ec0 0x7fa7640a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.097+0000 7fa77659c640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa7640a3ec0 0x7fa7640a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.097+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fa7600d2330 con 0x7fa778105970 2024-04-22T06:44:31.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.097+0000 7fa77659c640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa7640a3ec0 0x7fa7640a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa768009230 tx=0x7fa768005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.101+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa76009e7b0 con 0x7fa778105970 2024-04-22T06:44:31.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:30 smithi079 ceph-mon[27946]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:31.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:30 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2142795881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:31.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:30 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2322964772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:31.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.277+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa778101680 con 0x7fa778105970 2024-04-22T06:44:31.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.277+0000 7fa76f7fe640 1 -- 172.21.15.79:0/1385588522 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa76009e0b0 con 0x7fa778105970 2024-04-22T06:44:31.275 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa7640a3ec0 msgr2=0x7fa7640a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa7640a3ec0 0x7fa7640a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa768009230 tx=0x7fa768005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 msgr2=0x7fa778197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 0x7fa778197b80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa760005fd0 tx=0x7fa760009d10 comp rx=0 tx=0).stop 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 shutdown_connections 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77819bf00 0x7fa77819c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa778105970 0x7fa778197b80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa7640a3ec0 0x7fa7640a6380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 --2- 172.21.15.79:0/1385588522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa778100af0 0x7fa778197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 >> 172.21.15.79:0/1385588522 conn(0x7fa7780760e0 msgr2=0x7fa778104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 shutdown_connections 2024-04-22T06:44:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.279+0000 7fa77ce62640 1 -- 172.21.15.79:0/1385588522 wait complete. 2024-04-22T06:44:31.288 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 -- 172.21.15.79:0/1778970789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd708107340 msgr2=0x7fd70810eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 --2- 172.21.15.79:0/1778970789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd708107340 0x7fd70810eec0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd700009ca0 tx=0x7fd70002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 -- 172.21.15.79:0/1778970789 shutdown_connections 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 --2- 172.21.15.79:0/1778970789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd70806b260 0x7fd708106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 --2- 172.21.15.79:0/1778970789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd708107340 0x7fd70810eec0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 --2- 172.21.15.79:0/1778970789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd70806a8b0 0x7fd70806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 -- 172.21.15.79:0/1778970789 >> 172.21.15.79:0/1778970789 conn(0x7fd708077e30 msgr2=0x7fd708078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 -- 172.21.15.79:0/1778970789 shutdown_connections 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.387+0000 7fd70fba4640 1 -- 172.21.15.79:0/1778970789 wait complete. 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 Processor -- start 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 -- start start 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd70806a8b0 0x7fd708198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd708107340 0x7fd708198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 0x7fd70819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd708112260 con 0x7fd70819d020 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081120e0 con 0x7fd70806a8b0 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70fba4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081123e0 con 0x7fd708107340 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 0x7fd70819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 0x7fd70819d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33334/0 (socket says 172.21.15.79:33334) 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 -- 172.21.15.79:0/3622132386 learned_addr learned my addr 172.21.15.79:0/3622132386 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70d940640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd70806a8b0 0x7fd708198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 -- 172.21.15.79:0/3622132386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd708107340 msgr2=0x7fd708198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd708107340 0x7fd708198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 -- 172.21.15.79:0/3622132386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd70806a8b0 msgr2=0x7fd708198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd70806a8b0 0x7fd708198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.388+0000 7fd70e141640 1 -- 172.21.15.79:0/3622132386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70819db10 con 0x7fd70819d020 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.389+0000 7fd70e141640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 0x7fd70819d4d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd700005e30 tx=0x7fd700032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.389+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700041020 con 0x7fd70819d020 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.389+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd700032970 con 0x7fd70819d020 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.389+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70819dda0 con 0x7fd70819d020 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.389+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700039040 con 0x7fd70819d020 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.389+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7081a5560 con 0x7fd70819d020 2024-04-22T06:44:31.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.390+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd708069230 con 0x7fd70819d020 2024-04-22T06:44:31.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.391+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd700044020 con 0x7fd70819d020 2024-04-22T06:44:31.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.391+0000 7fd6feffd640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6f40a3e40 0x7fd6f40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.391+0000 7fd70d940640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6f40a3e40 0x7fd6f40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.391+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd7000f2740 con 0x7fd70819d020 2024-04-22T06:44:31.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.392+0000 7fd70d940640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6f40a3e40 0x7fd6f40a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd704005fd0 tx=0x7fd70400b390 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.396+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd70004b020 con 0x7fd70819d020 2024-04-22T06:44:31.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:30 smithi193 ceph-mon[31815]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:31.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:30 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2142795881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:31.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:30 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2322964772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:31.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:30 smithi190 ceph-mon[31421]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:31.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:30 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2142795881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:31.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:30 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2322964772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:31.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.611+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd70806b260 con 0x7fd70819d020 2024-04-22T06:44:31.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.611+0000 7fd6feffd640 1 -- 172.21.15.79:0/3622132386 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7fd7000becd0 con 0x7fd70819d020 2024-04-22T06:44:31.610 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:31.610 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:44:31.610 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:44:31.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.613+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6f40a3e40 msgr2=0x7fd6f40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.613+0000 7fd70fba4640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6f40a3e40 0x7fd6f40a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd704005fd0 tx=0x7fd70400b390 comp rx=0 tx=0).stop 2024-04-22T06:44:31.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.613+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 msgr2=0x7fd70819d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 0x7fd70819d4d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd700005e30 tx=0x7fd700032420 comp rx=0 tx=0).stop 2024-04-22T06:44:31.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 shutdown_connections 2024-04-22T06:44:31.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd70819d020 0x7fd70819d4d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd708107340 0x7fd708198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6f40a3e40 0x7fd6f40a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 --2- 172.21.15.79:0/3622132386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd70806a8b0 0x7fd708198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 >> 172.21.15.79:0/3622132386 conn(0x7fd708077e30 msgr2=0x7fd7080ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 shutdown_connections 2024-04-22T06:44:31.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.614+0000 7fd70fba4640 1 -- 172.21.15.79:0/3622132386 wait complete. 2024-04-22T06:44:31.623 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:31.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 -- 172.21.15.79:0/4192924009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 msgr2=0x7fb70010dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 --2- 172.21.15.79:0/4192924009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb70010dcb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009900 tx=0x7fb6f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 -- 172.21.15.79:0/4192924009 shutdown_connections 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 --2- 172.21.15.79:0/4192924009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb700101130 0x7fb700101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 --2- 172.21.15.79:0/4192924009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb70010dcb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 --2- 172.21.15.79:0/4192924009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb700106f10 0x7fb7001072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 -- 172.21.15.79:0/4192924009 >> 172.21.15.79:0/4192924009 conn(0x7fb700077e30 msgr2=0x7fb700078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 -- 172.21.15.79:0/4192924009 shutdown_connections 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 -- 172.21.15.79:0/4192924009 wait complete. 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.725+0000 7fb707740640 1 Processor -- start 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 -- start start 2024-04-22T06:44:31.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb700101130 0x7fb700199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb7001999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb700106f10 0x7fb7001a4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb7001110d0 con 0x7fb700101ad0 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb700110f50 con 0x7fb700106f10 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb700111250 con 0x7fb700101130 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb7001999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb7001999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33356/0 (socket says 172.21.15.79:33356) 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 -- 172.21.15.79:0/1058536433 learned_addr learned my addr 172.21.15.79:0/1058536433 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb705cdd640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb700106f10 0x7fb7001a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb7054dc640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb700101130 0x7fb700199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 -- 172.21.15.79:0/1058536433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb700101130 msgr2=0x7fb700199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb700101130 0x7fb700199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 -- 172.21.15.79:0/1058536433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb700106f10 msgr2=0x7fb7001a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb700106f10 0x7fb7001a4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 -- 172.21.15.79:0/1058536433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7001a4a50 con 0x7fb700101ad0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb704cdb640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb7001999d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400e9f0 tx=0x7fb6f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400cd80 con 0x7fb700101ad0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f40108e0 con 0x7fb700101ad0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7001a4c20 con 0x7fb700101ad0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.726+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4018980 con 0x7fb700101ad0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.727+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7001a5170 con 0x7fb700101ad0 2024-04-22T06:44:31.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.727+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb700069230 con 0x7fb700101ad0 2024-04-22T06:44:31.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.728+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb6f4010430 con 0x7fb700101ad0 2024-04-22T06:44:31.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.729+0000 7fb6fe7fc640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6e80a3e40 0x7fb6e80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.729+0000 7fb7054dc640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6e80a3e40 0x7fb6e80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.729+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fb6f40d12c0 con 0x7fb700101ad0 2024-04-22T06:44:31.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.729+0000 7fb7054dc640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6e80a3e40 0x7fb6e80a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007d50 tx=0x7fb6f8007730 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.733+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6f409d740 con 0x7fb700101ad0 2024-04-22T06:44:31.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.912+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7001a5340 con 0x7fb700101ad0 2024-04-22T06:44:31.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.913+0000 7fb6fe7fc640 1 -- 172.21.15.79:0/1058536433 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7fb6f409d040 con 0x7fb700101ad0 2024-04-22T06:44:31.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:31.912 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:31.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6e80a3e40 msgr2=0x7fb6e80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6e80a3e40 0x7fb6e80a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007d50 tx=0x7fb6f8007730 comp rx=0 tx=0).stop 2024-04-22T06:44:31.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 msgr2=0x7fb7001999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb7001999d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400e9f0 tx=0x7fb6f400eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 shutdown_connections 2024-04-22T06:44:31.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6e80a3e40 0x7fb6e80a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb700106f10 0x7fb7001a4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb700101ad0 0x7fb7001999d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.915+0000 7fb707740640 1 --2- 172.21.15.79:0/1058536433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb700101130 0x7fb700199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.916+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 >> 172.21.15.79:0/1058536433 conn(0x7fb700077e30 msgr2=0x7fb7000ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.916+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 shutdown_connections 2024-04-22T06:44:31.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:31.916+0000 7fb707740640 1 -- 172.21.15.79:0/1058536433 wait complete. 2024-04-22T06:44:31.925 DEBUG:tasks.fwd_scrub:Yielding 2024-04-22T06:44:31.926 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T06:44:31.935 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:44:31.935 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:44:31.935 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-22T06:44:31.936 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-22T06:44:31.936 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:44:31.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:31.936 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:44:32.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 -- 172.21.15.79:0/868839789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c101090 msgr2=0x7f926c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 --2- 172.21.15.79:0/868839789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c101090 0x7f926c101470 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9268009900 tx=0x7f926802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 -- 172.21.15.79:0/2371250620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 msgr2=0x7ffac81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2371250620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac81035c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009900 tx=0x7ffab802bd60 comp rx=0 tx=0).stop 2024-04-22T06:44:32.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 -- 172.21.15.79:0/868839789 shutdown_connections 2024-04-22T06:44:32.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 --2- 172.21.15.79:0/868839789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101a40 0x7f926c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 --2- 172.21.15.79:0/868839789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f926c106160 0x7f926c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 --2- 172.21.15.79:0/868839789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c101090 0x7f926c101470 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 -- 172.21.15.79:0/868839789 >> 172.21.15.79:0/868839789 conn(0x7f926c077f50 msgr2=0x7f926c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 -- 172.21.15.79:0/2371250620 shutdown_connections 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2371250620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac810c520 0x7ffac810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2371250620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac8103b00 0x7ffac810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2371250620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac81035c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 -- 172.21.15.79:0/2371250620 >> 172.21.15.79:0/2371250620 conn(0x7ffac8077fd0 msgr2=0x7ffac80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 -- 172.21.15.79:0/2371250620 shutdown_connections 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7ffacd46d640 1 -- 172.21.15.79:0/2371250620 wait complete. 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 -- 172.21.15.79:0/868839789 shutdown_connections 2024-04-22T06:44:32.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.041+0000 7f927354b640 1 -- 172.21.15.79:0/868839789 wait complete. 2024-04-22T06:44:32.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 Processor -- start 2024-04-22T06:44:32.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 Processor -- start 2024-04-22T06:44:32.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 -- start start 2024-04-22T06:44:32.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 -- start start 2024-04-22T06:44:32.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101090 0x7f926c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 0x7f926c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f926c1985f0 0x7f926c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f926c111120 con 0x7f926c106160 2024-04-22T06:44:32.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f926c110fa0 con 0x7f926c101090 2024-04-22T06:44:32.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f927354b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1112a0 con 0x7f926c1985f0 2024-04-22T06:44:32.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 0x7ffac8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac810c520 0x7ffac819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffac81121b0 con 0x7ffac81031e0 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffac8112030 con 0x7ffac8103b00 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffacd46d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffac8112330 con 0x7ffac810c520 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f9270ae6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 0x7f926c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.046 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f92712e7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101090 0x7f926c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f9270ae6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 0x7f926c19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33390/0 (socket says 172.21.15.79:33390) 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f9270ae6640 1 -- 172.21.15.79:0/316016113 learned_addr learned my addr 172.21.15.79:0/316016113 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f9271ae8640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f926c1985f0 0x7f926c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffabffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 0x7ffac8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffac6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffabffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 0x7ffac8198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:52770/0 (socket says 172.21.15.79:52770) 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffabffff640 1 -- 172.21.15.79:0/2998300507 learned_addr learned my addr 172.21.15.79:0/2998300507 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7ffac77fe640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac810c520 0x7ffac819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f92712e7640 1 -- 172.21.15.79:0/316016113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f926c1985f0 msgr2=0x7f926c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f92712e7640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f926c1985f0 0x7f926c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f92712e7640 1 -- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 msgr2=0x7f926c19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f92712e7640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 0x7f926c19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f92712e7640 1 -- 172.21.15.79:0/316016113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c199110 con 0x7f926c101090 2024-04-22T06:44:32.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.042+0000 7f9270ae6640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 0x7f926c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac6ffd640 1 -- 172.21.15.79:0/2998300507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac810c520 msgr2=0x7ffac819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac6ffd640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac810c520 0x7ffac819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac6ffd640 1 -- 172.21.15.79:0/2998300507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 msgr2=0x7ffac8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac6ffd640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 0x7ffac8198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac6ffd640 1 -- 172.21.15.79:0/2998300507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac819d720 con 0x7ffac81031e0 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffabffff640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 0x7ffac8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:32.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7f92712e7640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101090 0x7f926c19e3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f926802c170 tx=0x7f92680045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac6ffd640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac8198730 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffab8036040 tx=0x7ffab802d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7f92627fc640 1 -- 172.21.15.79:0/316016113 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9268043020 con 0x7f926c101090 2024-04-22T06:44:32.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7f92627fc640 1 -- 172.21.15.79:0/316016113 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f926802cb70 con 0x7f926c101090 2024-04-22T06:44:32.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c1993a0 con 0x7f926c101090 2024-04-22T06:44:32.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7f92627fc640 1 -- 172.21.15.79:0/316016113 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9268003ac0 con 0x7f926c101090 2024-04-22T06:44:32.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8043020 con 0x7ffac81031e0 2024-04-22T06:44:32.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c1a55a0 con 0x7f926c101090 2024-04-22T06:44:32.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac819d9b0 con 0x7ffac81031e0 2024-04-22T06:44:32.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab802dd20 con 0x7ffac81031e0 2024-04-22T06:44:32.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8003b50 con 0x7ffac81031e0 2024-04-22T06:44:32.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.043+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac81a52a0 con 0x7ffac81031e0 2024-04-22T06:44:32.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.044+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac804fce0 con 0x7ffac81031e0 2024-04-22T06:44:32.052 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.045+0000 7f92627fc640 1 -- 172.21.15.79:0/316016113 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9268046020 con 0x7f926c101090 2024-04-22T06:44:32.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.045+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ffab8046020 con 0x7ffac81031e0 2024-04-22T06:44:32.053 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7ffac4ff9640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffab40a3ee0 0x7ffab40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f92627fc640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92580a3e40 0x7f92580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 Processor -- start 2024-04-22T06:44:32.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7ffab80f3040 con 0x7ffac81031e0 2024-04-22T06:44:32.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7ffabffff640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffab40a3ee0 0x7ffab40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 -- start start 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9270ae6640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92580a3e40 0x7f92580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f92627fc640 1 -- 172.21.15.79:0/316016113 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f92680f2a20 con 0x7f926c101090 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a6290 0x557d389ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557d38909f00 con 0x557d389aee10 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557d388a7590 con 0x557d389a6290 2024-04-22T06:44:32.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.046+0000 7f9276213740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557d38905ec0 con 0x557d389a58e0 2024-04-22T06:44:32.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33412/0 (socket says 172.21.15.79:33412) 2024-04-22T06:44:32.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 -- 172.21.15.79:0/247410744 learned_addr learned my addr 172.21.15.79:0/247410744 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7ffabffff640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffab40a3ee0 0x7ffab40a63a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffac8100930 tx=0x7ffab000b410 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9270ae6640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92580a3e40 0x7f92580a6300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f926c06a1c0 tx=0x7f925c00b410 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f92712e7640 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9270ae6640 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a6290 0x557d389ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 -- 172.21.15.79:0/247410744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 msgr2=0x557d389a5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389a5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 -- 172.21.15.79:0/247410744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a6290 msgr2=0x557d389ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a6290 0x557d389ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 -- 172.21.15.79:0/247410744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d386e47e0 con 0x557d389aee10 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9270ae6640 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a6290 0x557d389ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:32.057 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.047+0000 7f9271ae8640 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389b1260 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9264009f50 tx=0x7f92640118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b2a0dae5f398a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.050+0000 7f92577fe640 1 -- 172.21.15.79:0/247410744 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264004d20 con 0x557d389aee10 2024-04-22T06:44:32.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.050+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffab804d020 con 0x7ffac81031e0 2024-04-22T06:44:32.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.050+0000 7f92577fe640 1 -- 172.21.15.79:0/247410744 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9264004050 con 0x557d389aee10 2024-04-22T06:44:32.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.050+0000 7f92577fe640 1 -- 172.21.15.79:0/247410744 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264005280 con 0x557d389aee10 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 -- 172.21.15.79:0/247410744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 msgr2=0x557d389b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389b1260 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9264009f50 tx=0x7f92640118d0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 -- 172.21.15.79:0/247410744 shutdown_connections 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a6290 0x557d389ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389b1260 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- 172.21.15.79:0/247410744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389a5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 -- 172.21.15.79:0/247410744 >> 172.21.15.79:0/247410744 conn(0x557d3899ffb0 msgr2=0x557d389a0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 -- 172.21.15.79:0/247410744 shutdown_connections 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 -- 172.21.15.79:0/247410744 wait complete. 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 -- start start 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 Processor -- start 2024-04-22T06:44:32.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 0x557d389ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.051+0000 7f9276213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d389ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557d389b1d80 con 0x557d389a6290 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557d388c3580 con 0x557d389aee10 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557d389b1f00 con 0x557d389a58e0 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 10 client.0 _ll_register_callbacks cb 0x557d388c9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 0x557d389ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 0x557d389ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33426/0 (socket says 172.21.15.79:33426) 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 -- 172.21.15.79:0/2435643718 learned_addr learned my addr 172.21.15.79:0/2435643718 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9271ae8640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d389ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f92712e7640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 -- 172.21.15.79:0/2435643718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 msgr2=0x557d389abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389abdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 -- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 msgr2=0x557d389ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d389ac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 -- 172.21.15.79:0/2435643718 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d38744f70 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9271ae8640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d389ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f92712e7640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 0x557d389ac2f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f925c011a30 tx=0x7f925c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b3b9acc7f7ee6b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9254ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9254ff9640 1 -- 172.21.15.79:0/2435643718 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c01ec00 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9264004050 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9254ff9640 1 -- 172.21.15.79:0/2435643718 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f925c01eda0 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9254ff9640 1 -- 172.21.15.79:0/2435643718 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f925c028430 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 10 client.15471 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.052+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557d385dc170 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9254ff9640 1 -- 172.21.15.79:0/2435643718 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9264004050 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9254ff9640 1 -- 172.21.15.79:0/2435643718 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f925c01f6a0 con 0x557d389a6290 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9276213740 10 client.15471 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:32.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9276213740 10 client.15471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9276213740 20 client.15471 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9276213740 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d389bf360 0x557d389c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9276213740 4 client.15471 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d38917920 con 0x557d389bf360 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f92712e7640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d389bf360 0x557d389c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f9254ff9640 10 client.15471 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.053+0000 7f92712e7640 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d389bf360 0x557d389c17f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad89c97fac3318 server_cookie=cd7b6edf8f6dce5e in_seq=0 out_seq=0 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9254ff9640 1 -- 172.21.15.79:0/2435643718 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d38917920 con 0x557d389bf360 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9254ff9640 10 client.15471 handle_command_reply: tid=0 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 client.15471 shutdown 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 msgr2=0x557d389ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 0x557d389ac2f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f925c011a30 tx=0x7f925c004730 comp rx=0 tx=0).stop 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 shutdown_connections 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d389bf360 0x557d389c17f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d389ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a6290 0x557d389ac2f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 --2- 172.21.15.79:0/2435643718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a58e0 0x557d389abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 >> 172.21.15.79:0/2435643718 conn(0x557d3899ffb0 msgr2=0x557d389a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9254ff9640 0 client.15471 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 shutdown_connections 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 1 -- 172.21.15.79:0/2435643718 wait complete. 2024-04-22T06:44:32.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.054+0000 7f9276213740 20 client.15471 trim_cache size 0 max 16384 2024-04-22T06:44:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 Processor -- start 2024-04-22T06:44:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 -- start start 2024-04-22T06:44:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a58e0 0x557d389ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389bac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1b03e0 con 0x557d389aee10 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1adcb0 con 0x557d389a58e0 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9276213740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1abb00 con 0x557d389a6290 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389bac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389bac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33428/0 (socket says 172.21.15.79:33428) 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 -- 172.21.15.79:0/481970039 learned_addr learned my addr 172.21.15.79:0/481970039 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9270ae6640 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f92712e7640 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a58e0 0x557d389ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 -- 172.21.15.79:0/481970039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 msgr2=0x557d389aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389aacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 -- 172.21.15.79:0/481970039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a58e0 msgr2=0x557d389ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a58e0 0x557d389ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f9271ae8640 1 -- 172.21.15.79:0/481970039 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d389bdf60 con 0x557d389aee10 2024-04-22T06:44:32.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.072+0000 7f92712e7640 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a58e0 0x557d389ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9271ae8640 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389bac40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9264009ef0 tx=0x7f9264003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6217091ba453614f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9256ffd640 1 -- 172.21.15.79:0/481970039 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264004050 con 0x557d389aee10 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9256ffd640 1 -- 172.21.15.79:0/481970039 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9264004780 con 0x557d389aee10 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9256ffd640 1 -- 172.21.15.79:0/481970039 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264005290 con 0x557d389aee10 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9276213740 1 -- 172.21.15.79:0/481970039 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 msgr2=0x557d389bac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9276213740 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389bac40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9264009ef0 tx=0x7f9264003a20 comp rx=0 tx=0).stop 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9276213740 1 -- 172.21.15.79:0/481970039 shutdown_connections 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9276213740 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389aee10 0x557d389bac40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9276213740 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.073+0000 7f9276213740 1 --2- 172.21.15.79:0/481970039 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389a58e0 0x557d389ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.074+0000 7f9276213740 1 -- 172.21.15.79:0/481970039 >> 172.21.15.79:0/481970039 conn(0x557d3899ffb0 msgr2=0x557d389a1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.074+0000 7f9276213740 1 -- 172.21.15.79:0/481970039 shutdown_connections 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.074+0000 7f9276213740 1 -- 172.21.15.79:0/481970039 wait complete. 2024-04-22T06:44:32.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.074+0000 7f9276213740 1 -- start start 2024-04-22T06:44:32.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.074+0000 7f9276213740 1 Processor -- start 2024-04-22T06:44:32.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 0x557d389b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d38a1d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1ac170 con 0x557d389a58e0 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1ad5e0 con 0x557d389aee10 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1ac840 con 0x557d389a6290 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 10 client.0 _ll_register_callbacks cb 0x7f925c011680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 0x557d389b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 0x557d389b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33432/0 (socket says 172.21.15.79:33432) 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 -- 172.21.15.79:0/2256778786 learned_addr learned my addr 172.21.15.79:0/2256778786 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9271ae8640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d38a1d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 -- 172.21.15.79:0/2256778786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 msgr2=0x557d389b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 -- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 msgr2=0x557d38a1d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d38a1d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 -- 172.21.15.79:0/2256778786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d38a1dfb0 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f92712e7640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 0x557d389b36e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f92680be550 tx=0x7f92680035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cba937fc7a4bef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9254ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9254ff9640 1 -- 172.21.15.79:0/2256778786 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680c2c00 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9254ff9640 1 -- 172.21.15.79:0/2256778786 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f92680a90b0 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9264004780 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9254ff9640 1 -- 172.21.15.79:0/2256778786 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f92680c74c0 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 10 client.15477 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.075+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x557d38a26400 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9254ff9640 1 -- 172.21.15.79:0/2256778786 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9264004780 con 0x557d389a58e0 2024-04-22T06:44:32.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9254ff9640 1 -- 172.21.15.79:0/2256778786 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f92680d4070 con 0x557d389a58e0 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9276213740 10 client.15477 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9276213740 10 client.15477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9276213740 20 client.15477 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9276213740 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d38a26ab0 0x557d38a28ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9276213740 4 client.15477 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557d38911790 con 0x557d38a26ab0 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d38a26ab0 0x557d38a28ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9254ff9640 10 client.15477 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.076+0000 7f9270ae6640 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d38a26ab0 0x557d38a28ed0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d09690575881c8 server_cookie=2e298a4999c64865 in_seq=0 out_seq=0 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.077+0000 7f9254ff9640 1 -- 172.21.15.79:0/2256778786 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557d38911790 con 0x557d38a26ab0 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.077+0000 7f9254ff9640 10 client.15477 handle_command_reply: tid=0 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.077+0000 7f9276213740 1 client.15477 shutdown 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.077+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 msgr2=0x557d389b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.077+0000 7f9276213740 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 0x557d389b36e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f92680be550 tx=0x7f92680035f0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 shutdown_connections 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557d38a26ab0 0x557d38a28ed0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557d389aee10 0x557d38a1d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557d389a6290 0x557d389b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9254ff9640 0 client.15477 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 --2- 172.21.15.79:0/2256778786 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557d389a58e0 0x557d389b36e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 >> 172.21.15.79:0/2256778786 conn(0x557d3899ffb0 msgr2=0x557d389abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 shutdown_connections 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 1 -- 172.21.15.79:0/2256778786 wait complete. 2024-04-22T06:44:32.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f9276213740 20 client.15477 trim_cache size 0 max 16384 2024-04-22T06:44:32.083 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:32.083 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:44:32.083 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "69493815-1af5-4335-b3c0-e48efca62ddf", 2024-04-22T06:44:32.083 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:44:32.083 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92580a3e40 msgr2=0x7f92580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92580a3e40 0x7f92580a6300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f926c06a1c0 tx=0x7f925c00b410 comp rx=0 tx=0).stop 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101090 msgr2=0x7f926c19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101090 0x7f926c19e3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f926802c170 tx=0x7f92680045e0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 shutdown_connections 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f926c1985f0 0x7f926c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f926c106160 0x7f926c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92580a3e40 0x7f92580a6300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 --2- 172.21.15.79:0/316016113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926c101090 0x7f926c19e3f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 >> 172.21.15.79:0/316016113 conn(0x7f926c077f50 msgr2=0x7f926c104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.078+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 shutdown_connections 2024-04-22T06:44:32.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.079+0000 7f927354b640 1 -- 172.21.15.79:0/316016113 wait complete. 2024-04-22T06:44:32.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:44:32.088 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 -- 172.21.15.79:0/1438454007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101a30 msgr2=0x7f2008105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 --2- 172.21.15.79:0/1438454007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101a30 0x7f2008105a70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009900 tx=0x7f1ffc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 -- 172.21.15.79:0/1438454007 shutdown_connections 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 --2- 172.21.15.79:0/1438454007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2008106150 0x7f200810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 --2- 172.21.15.79:0/1438454007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101a30 0x7f2008105a70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 --2- 172.21.15.79:0/1438454007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2008101080 0x7f2008101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 -- 172.21.15.79:0/1438454007 >> 172.21.15.79:0/1438454007 conn(0x7f2008078000 msgr2=0x7f2008078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.191+0000 7f200e681640 1 -- 172.21.15.79:0/1438454007 shutdown_connections 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.192+0000 7f200e681640 1 -- 172.21.15.79:0/1438454007 wait complete. 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.192+0000 7f200e681640 1 Processor -- start 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.192+0000 7f200e681640 1 -- start start 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200e681640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 0x7f20081a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200e681640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2008106150 0x7f20081a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200e681640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f200819a750 0x7f200819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200e681640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2008111090 con 0x7f2008101080 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200e681640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2008110f10 con 0x7f2008106150 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200e681640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2008111210 con 0x7f200819a750 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 0x7f20081a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 0x7f20081a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33448/0 (socket says 172.21.15.79:33448) 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 -- 172.21.15.79:0/4055178521 learned_addr learned my addr 172.21.15.79:0/4055178521 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f20077fe640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2008106150 0x7f20081a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f200cc1e640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f200819a750 0x7f200819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 -- 172.21.15.79:0/4055178521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f200819a750 msgr2=0x7f200819abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f200819a750 0x7f200819abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 -- 172.21.15.79:0/4055178521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2008106150 msgr2=0x7f20081a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2008106150 0x7f20081a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 -- 172.21.15.79:0/4055178521 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200819b350 con 0x7f2008101080 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f2007fff640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 0x7f20081a0690 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800e9f0 tx=0x7f1ff800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.193+0000 7f20057fa640 1 -- 172.21.15.79:0/4055178521 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff800cd80 con 0x7f2008101080 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.194+0000 7f20057fa640 1 -- 172.21.15.79:0/4055178521 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff80108e0 con 0x7f2008101080 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.194+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20081a7430 con 0x7f2008101080 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.194+0000 7f20057fa640 1 -- 172.21.15.79:0/4055178521 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8018980 con 0x7f2008101080 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.194+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20081a77e0 con 0x7f2008101080 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.196+0000 7f20057fa640 1 -- 172.21.15.79:0/4055178521 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1ff8010430 con 0x7f2008101080 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 Processor -- start 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 -- start start 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e3b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f20057fa640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ff00a3e40 0x7f1ff00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e44540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e46e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55fc63da01e0 con 0x55fc63e44a80 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55fc63d3d390 con 0x55fc63e3b580 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2011279740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fc63d9c1a0 con 0x55fc63e3bea0 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e3b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2007fff640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ff00a3e40 0x7f1ff00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e3b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:52816/0 (socket says 172.21.15.79:52816) 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 -- 172.21.15.79:0/1377226370 learned_addr learned my addr 172.21.15.79:0/1377226370 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f20057fa640 1 -- 172.21.15.79:0/4055178521 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f1ff80d0d40 con 0x7f2008101080 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e46e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f20077fe640 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e44540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 -- 172.21.15.79:0/1377226370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 msgr2=0x55fc63e44540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e44540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 -- 172.21.15.79:0/1377226370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 msgr2=0x55fc63e46e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e46e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f200cc1e640 1 -- 172.21.15.79:0/1377226370 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc63b7c7d0 con 0x55fc63e3b580 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.197+0000 7f2007fff640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ff00a3e40 0x7f1ff00a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80ce3a0 tx=0x7f1ff8013700 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.198+0000 7f200cc1e640 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e3b960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2000005f40 tx=0x7f2000005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10bd157d18a77ff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.198+0000 7f1fee7fc640 1 -- 172.21.15.79:0/1377226370 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000004b20 con 0x55fc63e3b580 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.198+0000 7f1fee7fc640 1 -- 172.21.15.79:0/1377226370 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2000004cc0 con 0x55fc63e3b580 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.198+0000 7f1fee7fc640 1 -- 172.21.15.79:0/1377226370 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000004030 con 0x55fc63e3b580 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 -- 172.21.15.79:0/1377226370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 msgr2=0x55fc63e3b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e3b960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2000005f40 tx=0x7f2000005bd0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 -- 172.21.15.79:0/1377226370 shutdown_connections 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e46e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e44540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 --2- 172.21.15.79:0/1377226370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e3b960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 -- 172.21.15.79:0/1377226370 >> 172.21.15.79:0/1377226370 conn(0x55fc63e35cb0 msgr2=0x55fc63e36090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 -- 172.21.15.79:0/1377226370 shutdown_connections 2024-04-22T06:44:32.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 -- 172.21.15.79:0/1377226370 wait complete. 2024-04-22T06:44:32.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 -- start start 2024-04-22T06:44:32.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.199+0000 7f2011279740 1 Processor -- start 2024-04-22T06:44:32.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e41ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e42000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e42540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55fc63e47870 con 0x55fc63e44a80 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55fc63d592f0 con 0x55fc63e3b580 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fc63e479f0 con 0x55fc63e3bea0 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 10 client.0 _ll_register_callbacks cb 0x55fc63d66580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e42540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e42540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33468/0 (socket says 172.21.15.79:33468) 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 -- 172.21.15.79:0/3233496445 learned_addr learned my addr 172.21.15.79:0/3233496445 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f20077fe640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e42000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e41ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 -- 172.21.15.79:0/3233496445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 msgr2=0x55fc63e42000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e42000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 -- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 msgr2=0x55fc63e41ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e41ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 -- 172.21.15.79:0/3233496445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc63ab23f0 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f200cc1e640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e42540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2000005cc0 tx=0x7f20000118b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ee01e683c94b73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f1fe77fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f1fe77fe640 1 -- 172.21.15.79:0/3233496445 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000023070 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f1fe77fe640 1 -- 172.21.15.79:0/3233496445 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f200000f880 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.200+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2000004cc0 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f1fe77fe640 1 -- 172.21.15.79:0/3233496445 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2000004b20 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 10 client.15489 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55fc63a720f0 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f1fe77fe640 1 -- 172.21.15.79:0/3233496445 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2000004cc0 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f1fe77fe640 1 -- 172.21.15.79:0/3233496445 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f2000038610 con 0x55fc63e44a80 2024-04-22T06:44:32.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 10 client.15489 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 10 client.15489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 20 client.15489 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63e54920 0x55fc63e56d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 4 client.15489 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc63dadc00 con 0x55fc63e54920 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.201+0000 7f20077fe640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63e54920 0x55fc63e56d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.202+0000 7f1fe77fe640 10 client.15489 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.202+0000 7f20077fe640 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63e54920 0x55fc63e56d10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddddaf7f92e6af12 server_cookie=f8432c90a33ce5b5 in_seq=0 out_seq=0 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.202+0000 7f1fe77fe640 1 -- 172.21.15.79:0/3233496445 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc63dadc00 con 0x55fc63e54920 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.202+0000 7f1fe77fe640 10 client.15489 handle_command_reply: tid=0 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.202+0000 7f2011279740 1 client.15489 shutdown 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 msgr2=0x55fc63e42540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e42540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2000005cc0 tx=0x7f20000118b0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 shutdown_connections 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63e54920 0x55fc63e56d10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e42540 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3bea0 0x55fc63e42000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f1fe77fe640 0 client.15489 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 --2- 172.21.15.79:0/3233496445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3b580 0x55fc63e41ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 >> 172.21.15.79:0/3233496445 conn(0x55fc63e35cb0 msgr2=0x55fc63e37a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 shutdown_connections 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 1 -- 172.21.15.79:0/3233496445 wait complete. 2024-04-22T06:44:32.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.203+0000 7f2011279740 20 client.15489 trim_cache size 0 max 16384 2024-04-22T06:44:32.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 Processor -- start 2024-04-22T06:44:32.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 -- start start 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 0x55fc63e441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e44a80 0x55fc63e42190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b25c0 con 0x55fc63e3b580 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f20081afe90 con 0x55fc63e3bea0 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f2011279740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f20081adce0 con 0x55fc63e44a80 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 0x55fc63e441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 0x55fc63e441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33474/0 (socket says 172.21.15.79:33474) 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.220+0000 7f200cc1e640 1 -- 172.21.15.79:0/2257079154 learned_addr learned my addr 172.21.15.79:0/2257079154 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f20077fe640 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f200cc1e640 1 -- 172.21.15.79:0/2257079154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e44a80 msgr2=0x55fc63e42190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f200cc1e640 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e44a80 0x55fc63e42190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f200cc1e640 1 -- 172.21.15.79:0/2257079154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 msgr2=0x55fc63e50730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f200cc1e640 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e50730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f200cc1e640 1 -- 172.21.15.79:0/2257079154 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc63e57080 con 0x55fc63e3b580 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f200cc1e640 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 0x55fc63e441f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2000005d70 tx=0x7f2000004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484d8b2a94a24a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f1fedffb640 1 -- 172.21.15.79:0/2257079154 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000023030 con 0x55fc63e3b580 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f1fedffb640 1 -- 172.21.15.79:0/2257079154 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2000024040 con 0x55fc63e3b580 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.221+0000 7f1fedffb640 1 -- 172.21.15.79:0/2257079154 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000011270 con 0x55fc63e3b580 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- 172.21.15.79:0/2257079154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 msgr2=0x55fc63e441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 0x55fc63e441f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2000005d70 tx=0x7f2000004ed0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- 172.21.15.79:0/2257079154 shutdown_connections 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e44a80 0x55fc63e42190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- 172.21.15.79:0/2257079154 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e3b580 0x55fc63e441f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- 172.21.15.79:0/2257079154 >> 172.21.15.79:0/2257079154 conn(0x55fc63e35cb0 msgr2=0x55fc63e41a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- 172.21.15.79:0/2257079154 shutdown_connections 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- 172.21.15.79:0/2257079154 wait complete. 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- start start 2024-04-22T06:44:32.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 Processor -- start 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3b580 0x55fc63e4cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e48220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e48760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f20081ae350 con 0x55fc63e44a80 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f20081af7c0 con 0x55fc63e3bea0 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f20081aea20 con 0x55fc63e3b580 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.222+0000 7f2011279740 10 client.0 _ll_register_callbacks cb 0x7f20000115d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e48760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e48760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33484/0 (socket says 172.21.15.79:33484) 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 -- 172.21.15.79:0/190322389 learned_addr learned my addr 172.21.15.79:0/190322389 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f20077fe640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e48220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3b580 0x55fc63e4cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 -- 172.21.15.79:0/190322389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3b580 msgr2=0x55fc63e4cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3b580 0x55fc63e4cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 -- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 msgr2=0x55fc63e48220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e48220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 -- 172.21.15.79:0/190322389 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc63e48d20 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f200cc1e640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e48760 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55fc63e49350 tx=0x7f2000038f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f8756bbde8efe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f1fe77fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f1fe77fe640 1 -- 172.21.15.79:0/190322389 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000023040 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2000024040 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f1fe77fe640 1 -- 172.21.15.79:0/190322389 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f2000011220 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f1fe77fe640 1 -- 172.21.15.79:0/190322389 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2000038510 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f2011279740 10 client.15495 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55fc63ebabf0 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.223+0000 7f1fe77fe640 1 -- 172.21.15.79:0/190322389 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2000024040 con 0x55fc63e44a80 2024-04-22T06:44:32.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f1fe77fe640 1 -- 172.21.15.79:0/190322389 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f200000fb20 con 0x55fc63e44a80 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 10 client.15495 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 20 client.15495 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63ebb230 0x55fc63ebd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 4 client.15495 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc63da7a70 con 0x55fc63ebb230 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f20077fe640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63ebb230 0x55fc63ebd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f1fe77fe640 10 client.15495 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f20077fe640 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63ebb230 0x55fc63ebd650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33b6c4d0f98c8fc2 server_cookie=de0ed73a48709aad in_seq=0 out_seq=0 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f1fe77fe640 1 -- 172.21.15.79:0/190322389 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fc63da7a70 con 0x55fc63ebb230 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f1fe77fe640 10 client.15495 handle_command_reply: tid=0 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 1 client.15495 shutdown 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 msgr2=0x55fc63e48760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.224+0000 7f2011279740 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e48760 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55fc63e49350 tx=0x7f2000038f90 comp rx=0 tx=0).stop 2024-04-22T06:44:32.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 shutdown_connections 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55fc63ebb230 0x55fc63ebd650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fc63e3bea0 0x55fc63e48220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55fc63e44a80 0x55fc63e48760 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 --2- 172.21.15.79:0/190322389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc63e3b580 0x55fc63e4cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f1fe77fe640 0 client.15495 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 >> 172.21.15.79:0/190322389 conn(0x55fc63e35cb0 msgr2=0x55fc63e42310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 shutdown_connections 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 1 -- 172.21.15.79:0/190322389 wait complete. 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f2011279740 20 client.15495 trim_cache size 0 max 16384 2024-04-22T06:44:32.229 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: "69493815-1af5-4335-b3c0-e48efca62ddf": { 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "69493815-1af5-4335-b3c0-e48efca62ddf", 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:32.230 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ff00a3e40 msgr2=0x7f1ff00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ff00a3e40 0x7f1ff00a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80ce3a0 tx=0x7f1ff8013700 comp rx=0 tx=0).stop 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 msgr2=0x7f20081a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 0x7f20081a0690 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800e9f0 tx=0x7f1ff800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 shutdown_connections 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ff00a3e40 0x7f1ff00a6300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2008106150 0x7f20081a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f200819a750 0x7f200819abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 --2- 172.21.15.79:0/4055178521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2008101080 0x7f20081a0690 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 >> 172.21.15.79:0/4055178521 conn(0x7f2008078000 msgr2=0x7f200810c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 shutdown_connections 2024-04-22T06:44:32.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.225+0000 7f200e681640 1 -- 172.21.15.79:0/4055178521 wait complete. 2024-04-22T06:44:32.233 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:31 smithi079 ceph-mon[27946]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 205 B/s wr, 26 op/s 2024-04-22T06:44:32.233 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1257916789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:32.233 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1385588522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:32.233 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3622132386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.233 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:31 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1058536433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.234 INFO:tasks.cephfs.filesystem:scrub status for tag:69493815-1af5-4335-b3c0-e48efca62ddf - {'path': '/', 'tag': '69493815-1af5-4335-b3c0-e48efca62ddf', 'options': 'recursive,force'} 2024-04-22T06:44:32.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.266+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ffac8107710 con 0x7ffab40a3ee0 2024-04-22T06:44:32.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.328+0000 7ffac4ff9640 1 -- 172.21.15.79:0/2998300507 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ffab000bce0 con 0x7ffab40a3ee0 2024-04-22T06:44:32.326 INFO:teuthology.orchestra.run.smithi079.stdout:Schedule set for path / 2024-04-22T06:44:32.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffab40a3ee0 msgr2=0x7ffab40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffab40a3ee0 0x7ffab40a63a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffac8100930 tx=0x7ffab000b410 comp rx=0 tx=0).stop 2024-04-22T06:44:32.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 msgr2=0x7ffac8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac8198730 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffab8036040 tx=0x7ffab802d920 comp rx=0 tx=0).stop 2024-04-22T06:44:32.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 shutdown_connections 2024-04-22T06:44:32.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffab40a3ee0 0x7ffab40a63a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffac8103b00 0x7ffac8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac810c520 0x7ffac819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 --2- 172.21.15.79:0/2998300507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffac81031e0 0x7ffac8198730 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 >> 172.21.15.79:0/2998300507 conn(0x7ffac8077fd0 msgr2=0x7ffac80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 shutdown_connections 2024-04-22T06:44:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.330+0000 7ffacd46d640 1 -- 172.21.15.79:0/2998300507 wait complete. 2024-04-22T06:44:32.339 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-22T06:44:32.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:31 smithi193 ceph-mon[31815]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 205 B/s wr, 26 op/s 2024-04-22T06:44:32.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1257916789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:32.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1385588522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:32.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3622132386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:31 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1058536433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:31 smithi190 ceph-mon[31421]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 205 B/s wr, 26 op/s 2024-04-22T06:44:32.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1257916789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:32.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1385588522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:32.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3622132386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:31 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1058536433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 -- 172.21.15.79:0/4136494085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c64101aa0 msgr2=0x7f4c64105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/4136494085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c64101aa0 0x7f4c64105a90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009900 tx=0x7f4c5002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 -- 172.21.15.79:0/4136494085 shutdown_connections 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/4136494085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c64106170 0x7f4c6410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/4136494085 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c64101aa0 0x7f4c64105a90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/4136494085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641010f0 0x7f4c641014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 -- 172.21.15.79:0/4136494085 >> 172.21.15.79:0/4136494085 conn(0x7f4c64077fd0 msgr2=0x7f4c640783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 -- 172.21.15.79:0/4136494085 shutdown_connections 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 -- 172.21.15.79:0/4136494085 wait complete. 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.485+0000 7f4c6aae0640 1 Processor -- start 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 -- start start 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641010f0 0x7f4c64197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c64106170 0x7f4c64197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 0x7f4c6419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641110b0 con 0x7f4c6419bde0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4c64110f30 con 0x7f4c641010f0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c64111230 con 0x7f4c64106170 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 0x7f4c6419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 0x7f4c6419c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53520/0 (socket says 172.21.15.79:53520) 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 -- 172.21.15.79:0/1786390449 learned_addr learned my addr 172.21.15.79:0/1786390449 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6887c640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641010f0 0x7f4c64197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c63fff640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c64106170 0x7f4c64197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 -- 172.21.15.79:0/1786390449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c64106170 msgr2=0x7f4c64197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c64106170 0x7f4c64197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 -- 172.21.15.79:0/1786390449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641010f0 msgr2=0x7f4c64197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641010f0 0x7f4c64197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 -- 172.21.15.79:0/1786390449 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6419c8d0 con 0x7f4c6419bde0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6907d640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 0x7f4c6419c290 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00cad0 tx=0x7f4c5c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00ad60 con 0x7f4c6419bde0 2024-04-22T06:44:32.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c5c005020 con 0x7f4c6419bde0 2024-04-22T06:44:32.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6419cbc0 con 0x7f4c6419bde0 2024-04-22T06:44:32.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.486+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00f6b0 con 0x7f4c6419bde0 2024-04-22T06:44:32.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.487+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c641a4380 con 0x7f4c6419bde0 2024-04-22T06:44:32.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.488+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c6404fce0 con 0x7f4c6419bde0 2024-04-22T06:44:32.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.489+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f4c5c0051d0 con 0x7f4c6419bde0 2024-04-22T06:44:32.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.489+0000 7f4c61ffb640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c540a3ee0 0x7f4c540a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.489+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f4c5c0d0980 con 0x7f4c6419bde0 2024-04-22T06:44:32.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.489+0000 7f4c6887c640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c540a3ee0 0x7f4c540a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.490+0000 7f4c6887c640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c540a3ee0 0x7f4c540a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800abe0 tx=0x7f4c5800a2d0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.494+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c5c014030 con 0x7f4c6419bde0 2024-04-22T06:44:32.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.694+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f4c64101aa0 con 0x7f4c540a3ee0 2024-04-22T06:44:32.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.753+0000 7f4c61ffb640 1 -- 172.21.15.79:0/1786390449 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f4c64101aa0 con 0x7f4c540a3ee0 2024-04-22T06:44:32.751 INFO:teuthology.orchestra.run.smithi079.stdout:Retention added to path / 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c540a3ee0 msgr2=0x7f4c540a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c540a3ee0 0x7f4c540a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800abe0 tx=0x7f4c5800a2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 msgr2=0x7f4c6419c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 0x7f4c6419c290 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00cad0 tx=0x7f4c5c00ced0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 shutdown_connections 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c6419bde0 0x7f4c6419c290 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c64106170 0x7f4c64197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c540a3ee0 0x7f4c540a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 --2- 172.21.15.79:0/1786390449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c641010f0 0x7f4c64197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 >> 172.21.15.79:0/1786390449 conn(0x7f4c64077fd0 msgr2=0x7f4c640fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 shutdown_connections 2024-04-22T06:44:32.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.755+0000 7f4c6aae0640 1 -- 172.21.15.79:0/1786390449 wait complete. 2024-04-22T06:44:32.764 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T06:44:32.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 -- 172.21.15.79:0/3762382638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806b3a0 msgr2=0x7f3a18106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/3762382638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806b3a0 0x7f3a18106b80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009900 tx=0x7f3a1402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 -- 172.21.15.79:0/3762382638 shutdown_connections 2024-04-22T06:44:32.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/3762382638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a18107260 0x7f3a1810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/3762382638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806b3a0 0x7f3a18106b80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/3762382638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1806a9f0 0x7f3a1806add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 -- 172.21.15.79:0/3762382638 >> 172.21.15.79:0/3762382638 conn(0x7f3a18076b80 msgr2=0x7f3a18076f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 -- 172.21.15.79:0/3762382638 shutdown_connections 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.865+0000 7f3a1fc26640 1 -- 172.21.15.79:0/3762382638 wait complete. 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 Processor -- start 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 -- start start 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 0x7f3a18198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a18107260 0x7f3a18198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1819ceb0 0x7f3a1819d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3a181121a0 con 0x7f3a1806a9f0 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3a18112020 con 0x7f3a1819ceb0 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1fc26640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a18112320 con 0x7f3a18107260 2024-04-22T06:44:32.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1d9c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 0x7f3a18198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1d9c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 0x7f3a18198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53540/0 (socket says 172.21.15.79:53540) 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1d9c2640 1 -- 172.21.15.79:0/204720738 learned_addr learned my addr 172.21.15.79:0/204720738 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.866+0000 7f3a1e1c3640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1819ceb0 0x7f3a1819d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d9c2640 1 -- 172.21.15.79:0/204720738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a18107260 msgr2=0x7f3a18198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d1c1640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a18107260 0x7f3a18198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d9c2640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a18107260 0x7f3a18198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d9c2640 1 -- 172.21.15.79:0/204720738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1819ceb0 msgr2=0x7f3a1819d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d9c2640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1819ceb0 0x7f3a1819d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d9c2640 1 -- 172.21.15.79:0/204720738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1819d9a0 con 0x7f3a1806a9f0 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1e1c3640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1819ceb0 0x7f3a1819d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d1c1640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a18107260 0x7f3a18198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1d9c2640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 0x7f3a18198640 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800c890 tx=0x7f3a0800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08004220 con 0x7f3a1806a9f0 2024-04-22T06:44:32.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a080043c0 con 0x7f3a1806a9f0 2024-04-22T06:44:32.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1819dc90 con 0x7f3a1806a9f0 2024-04-22T06:44:32.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08004a00 con 0x7f3a1806a9f0 2024-04-22T06:44:32.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.867+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a181a5480 con 0x7f3a1806a9f0 2024-04-22T06:44:32.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.868+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a18103660 con 0x7f3a1806a9f0 2024-04-22T06:44:32.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.869+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3a080079b0 con 0x7f3a1806a9f0 2024-04-22T06:44:32.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.869+0000 7f3a0effd640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3a040a3ee0 0x7f3a040a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.870+0000 7f3a1d1c1640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3a040a3ee0 0x7f3a040a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.870+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3a080d0880 con 0x7f3a1806a9f0 2024-04-22T06:44:32.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.870+0000 7f3a1d1c1640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3a040a3ee0 0x7f3a040a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a14036040 tx=0x7f3a14000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:32.874+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a08014030 con 0x7f3a1806a9f0 2024-04-22T06:44:33.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.078+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3a1806b3a0 con 0x7f3a040a3ee0 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.086+0000 7f3a0effd640 1 -- 172.21.15.79:0/204720738 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f3a1806b3a0 con 0x7f3a040a3ee0 2024-04-22T06:44:33.085 INFO:teuthology.orchestra.run.smithi079.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T06:44:32", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3a040a3ee0 msgr2=0x7f3a040a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3a040a3ee0 0x7f3a040a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a14036040 tx=0x7f3a14000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 msgr2=0x7f3a18198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 0x7f3a18198640 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800c890 tx=0x7f3a0800cd50 comp rx=0 tx=0).stop 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 shutdown_connections 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3a040a3ee0 0x7f3a040a63a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a1819ceb0 0x7f3a1819d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a18107260 0x7f3a18198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 --2- 172.21.15.79:0/204720738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3a1806a9f0 0x7f3a18198640 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 >> 172.21.15.79:0/204720738 conn(0x7f3a18076b80 msgr2=0x7f3a180ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 shutdown_connections 2024-04-22T06:44:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.088+0000 7f3a1fc26640 1 -- 172.21.15.79:0/204720738 wait complete. 2024-04-22T06:44:33.097 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:32 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:44:33.098 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:32 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:33.098 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:32 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:44:33.098 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:32 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:33.098 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:32 smithi079 ceph-mon[27946]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.098 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:32 smithi079 ceph-mon[27946]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.099 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- 172.21.15.79:0/1886747851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c10c560 msgr2=0x7f0d3c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- 172.21.15.79:0/1886747851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c10c560 0x7f0d3c10e9b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009ca0 tx=0x7f0d3402c180 comp rx=0 tx=0).stop 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- 172.21.15.79:0/1886747851 shutdown_connections 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- 172.21.15.79:0/1886747851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c10c560 0x7f0d3c10e9b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- 172.21.15.79:0/1886747851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c103b40 0x7f0d3c10c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- 172.21.15.79:0/1886747851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c103220 0x7f0d3c103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- 172.21.15.79:0/1886747851 >> 172.21.15.79:0/1886747851 conn(0x7f0d3c077fd0 msgr2=0x7f0d3c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- 172.21.15.79:0/1886747851 shutdown_connections 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- 172.21.15.79:0/1886747851 wait complete. 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 Processor -- start 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- start start 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c103220 0x7f0d3c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 0x7f0d3c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c10c560 0x7f0d3c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c1121a0 con 0x7f0d3c103b40 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c112020 con 0x7f0d3c103220 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.201+0000 7f0d4458e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c112320 con 0x7f0d3c10c560 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 0x7f0d3c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 0x7f0d3c198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53564/0 (socket says 172.21.15.79:53564) 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 -- 172.21.15.79:0/739659738 learned_addr learned my addr 172.21.15.79:0/739659738 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d4232a640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c103220 0x7f0d3c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 -- 172.21.15.79:0/739659738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c10c560 msgr2=0x7f0d3c19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c10c560 0x7f0d3c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 -- 172.21.15.79:0/739659738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c103220 msgr2=0x7f0d3c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c103220 0x7f0d3c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 -- 172.21.15.79:0/739659738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c19d710 con 0x7f0d3c103b40 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d41b29640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 0x7f0d3c198c60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00d770 tx=0x7f0d2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c017070 con 0x7f0d3c103b40 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d2c0054d0 con 0x7f0d3c103b40 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c004c30 con 0x7f0d3c103b40 2024-04-22T06:44:33.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c19da00 con 0x7f0d3c103b40 2024-04-22T06:44:33.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.202+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1a5290 con 0x7f0d3c103b40 2024-04-22T06:44:33.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.203+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3c04fce0 con 0x7f0d3c103b40 2024-04-22T06:44:33.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.204+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f0d2c004df0 con 0x7f0d3c103b40 2024-04-22T06:44:33.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.205+0000 7f0d337fe640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0d280a3f60 0x7f0d280a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.205+0000 7f0d4232a640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0d280a3f60 0x7f0d280a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.205+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f0d2c0d0520 con 0x7f0d3c103b40 2024-04-22T06:44:33.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.205+0000 7f0d4232a640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0d280a3f60 0x7f0d280a6420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005fd0 tx=0x7f0d38005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.209+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d2c09c9a0 con 0x7f0d3c103b40 2024-04-22T06:44:33.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.407+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f0d3c062390 con 0x7f0d280a3f60 2024-04-22T06:44:33.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.414+0000 7f0d337fe640 1 -- 172.21.15.79:0/739659738 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f0d3c062390 con 0x7f0d280a3f60 2024-04-22T06:44:33.413 INFO:teuthology.orchestra.run.smithi079.stdout:/ 1m 6m3h 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0d280a3f60 msgr2=0x7f0d280a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0d280a3f60 0x7f0d280a6420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005fd0 tx=0x7f0d38005850 comp rx=0 tx=0).stop 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 msgr2=0x7f0d3c198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 0x7f0d3c198c60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00d770 tx=0x7f0d2c00dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 shutdown_connections 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c10c560 0x7f0d3c19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d3c103b40 0x7f0d3c198c60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0d280a3f60 0x7f0d280a6420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 --2- 172.21.15.79:0/739659738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c103220 0x7f0d3c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 >> 172.21.15.79:0/739659738 conn(0x7f0d3c077fd0 msgr2=0x7f0d3c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 shutdown_connections 2024-04-22T06:44:33.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.416+0000 7f0d4458e640 1 -- 172.21.15.79:0/739659738 wait complete. 2024-04-22T06:44:33.426 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-22T06:44:33.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:32 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:44:33.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:32 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:33.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:32 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:44:33.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:32 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:33.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:32 smithi193 ceph-mon[31815]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:32 smithi193 ceph-mon[31815]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.450 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-22T06:44:33.461 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-22T06:44:33.500 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:44:33.500 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-22T06:44:33.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:32 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:44:33.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:32 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:33.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:32 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:44:33.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:32 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:33.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:32 smithi190 ceph-mon[31421]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:32 smithi190 ceph-mon[31421]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.503 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-22T06:44:33.553 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:33.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 -- 172.21.15.79:0/4154593258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c101110 msgr2=0x7f234c106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 --2- 172.21.15.79:0/4154593258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c101110 0x7f234c106bd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2334009900 tx=0x7f233402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:33.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 -- 172.21.15.79:0/4154593258 shutdown_connections 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 --2- 172.21.15.79:0/4154593258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f234c1072b0 0x7f234c10eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 --2- 172.21.15.79:0/4154593258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c101110 0x7f234c106bd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 --2- 172.21.15.79:0/4154593258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 0x7f234c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 -- 172.21.15.79:0/4154593258 >> 172.21.15.79:0/4154593258 conn(0x7f234c077480 msgr2=0x7f234c0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 -- 172.21.15.79:0/4154593258 shutdown_connections 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.653+0000 7f235125c640 1 -- 172.21.15.79:0/4154593258 wait complete. 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 Processor -- start 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 -- start start 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 0x7f234c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f234c1072b0 0x7f234c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 0x7f234c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f234c112230 con 0x7f234c1985d0 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f234c1120b0 con 0x7f234c1072b0 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f235125c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f234c1123b0 con 0x7f234c100760 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f234b59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 0x7f234c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f234b59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 0x7f234c198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53584/0 (socket says 172.21.15.79:53584) 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f234b59e640 1 -- 172.21.15.79:0/2171461492 learned_addr learned my addr 172.21.15.79:0/2171461492 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.654+0000 7f234ad9d640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 0x7f234c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f2343fff640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f234c1072b0 0x7f234c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234b59e640 1 -- 172.21.15.79:0/2171461492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 msgr2=0x7f234c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234b59e640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 0x7f234c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234b59e640 1 -- 172.21.15.79:0/2171461492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f234c1072b0 msgr2=0x7f234c19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234b59e640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f234c1072b0 0x7f234c19e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234b59e640 1 -- 172.21.15.79:0/2171461492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234c1990f0 con 0x7f234c1985d0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234ad9d640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 0x7f234c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f234b59e640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 0x7f234c198a80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f234400cad0 tx=0x7f234400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234400ae20 con 0x7f234c1985d0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f234c1993e0 con 0x7f234c1985d0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2344005020 con 0x7f234c1985d0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234400f6b0 con 0x7f234c1985d0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.655+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f234c1a5630 con 0x7f234c1985d0 2024-04-22T06:44:33.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.656+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f234c069230 con 0x7f234c1985d0 2024-04-22T06:44:33.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.657+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f23440051d0 con 0x7f234c1985d0 2024-04-22T06:44:33.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.658+0000 7f2348d99640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f23380a3e40 0x7f23380a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.658+0000 7f234ad9d640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f23380a3e40 0x7f23380a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.658+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f23440d0ee0 con 0x7f234c1985d0 2024-04-22T06:44:33.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.658+0000 7f234ad9d640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f23380a3e40 0x7f23380a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f233c0058c0 tx=0x7f233c005d30 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.661+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2344014030 con 0x7f234c1985d0 2024-04-22T06:44:33.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.840+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f234c199b00 con 0x7f234c1985d0 2024-04-22T06:44:33.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.841+0000 7f2348d99640 1 -- 172.21.15.79:0/2171461492 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7f234c199b00 con 0x7f234c1985d0 2024-04-22T06:44:33.840 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:33.840 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:33.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f23380a3e40 msgr2=0x7f23380a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f23380a3e40 0x7f23380a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f233c0058c0 tx=0x7f233c005d30 comp rx=0 tx=0).stop 2024-04-22T06:44:33.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 msgr2=0x7f234c198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 0x7f234c198a80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f234400cad0 tx=0x7f234400cf90 comp rx=0 tx=0).stop 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 shutdown_connections 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f23380a3e40 0x7f23380a6300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f234c1072b0 0x7f234c19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f234c1985d0 0x7f234c198a80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 --2- 172.21.15.79:0/2171461492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f234c100760 0x7f234c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 >> 172.21.15.79:0/2171461492 conn(0x7f234c077480 msgr2=0x7f234c1056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.843+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 shutdown_connections 2024-04-22T06:44:33.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.844+0000 7f235125c640 1 -- 172.21.15.79:0/2171461492 wait complete. 2024-04-22T06:44:33.854 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:33.854 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:33.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 -- 172.21.15.79:0/1955204195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c10c510 msgr2=0x7f127c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 --2- 172.21.15.79:0/1955204195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c10c510 0x7f127c10e9b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1274009900 tx=0x7f127402bd60 comp rx=0 tx=0).stop 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 -- 172.21.15.79:0/1955204195 shutdown_connections 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 --2- 172.21.15.79:0/1955204195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c10c510 0x7f127c10e9b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 --2- 172.21.15.79:0/1955204195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f127c103af0 0x7f127c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 --2- 172.21.15.79:0/1955204195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f127c1031d0 0x7f127c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.956+0000 7f1281a3f640 1 -- 172.21.15.79:0/1955204195 >> 172.21.15.79:0/1955204195 conn(0x7f127c077f50 msgr2=0x7f127c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 -- 172.21.15.79:0/1955204195 shutdown_connections 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 -- 172.21.15.79:0/1955204195 wait complete. 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 Processor -- start 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 -- start start 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 0x7f127c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f127c103af0 0x7f127c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f127c10c510 0x7f127c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f127c1121a0 con 0x7f127c1031d0 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f127c112020 con 0x7f127c10c510 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f1281a3f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f127c112320 con 0x7f127c103af0 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.957+0000 7f127affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 0x7f127c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 0x7f127c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53606/0 (socket says 172.21.15.79:53606) 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 -- 172.21.15.79:0/925175296 learned_addr learned my addr 172.21.15.79:0/925175296 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 -- 172.21.15.79:0/925175296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f127c103af0 msgr2=0x7f127c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.962 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f127c103af0 0x7f127c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 -- 172.21.15.79:0/925175296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f127c10c510 msgr2=0x7f127c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127b7fe640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f127c10c510 0x7f127c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f127c10c510 0x7f127c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 -- 172.21.15.79:0/925175296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c19d6d0 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f127affd640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 0x7f127c1986e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f126c00ebc0 tx=0x7f126c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c010630 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f126c010c80 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127c19d9c0 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c018b60 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.958+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f127c1a5250 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.959+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f127c068d70 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.960+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f126c018d00 con 0x7f127c1031d0 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.961+0000 7f1280a3d640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f12640a3e40 0x7f12640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.961+0000 7f127a7fc640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f12640a3e40 0x7f12640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.961+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f126c0d11c0 con 0x7f127c1031d0 2024-04-22T06:44:33.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.961+0000 7f127a7fc640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f12640a3e40 0x7f12640a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f127c1008a0 tx=0x7f1270005b60 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:33.965+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f126c09d640 con 0x7f127c1031d0 2024-04-22T06:44:34.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.142+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f127c02d5c0 con 0x7f127c1031d0 2024-04-22T06:44:34.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.142+0000 7f1280a3d640 1 -- 172.21.15.79:0/925175296 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f126c09cf40 con 0x7f127c1031d0 2024-04-22T06:44:34.141 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f12640a3e40 msgr2=0x7f12640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f12640a3e40 0x7f12640a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f127c1008a0 tx=0x7f1270005b60 comp rx=0 tx=0).stop 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 msgr2=0x7f127c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 0x7f127c1986e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f126c00ebc0 tx=0x7f126c00c6a0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 shutdown_connections 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f12640a3e40 0x7f12640a6300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f127c10c510 0x7f127c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f127c103af0 0x7f127c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 --2- 172.21.15.79:0/925175296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f127c1031d0 0x7f127c1986e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 >> 172.21.15.79:0/925175296 conn(0x7f127c077f50 msgr2=0x7f127c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 shutdown_connections 2024-04-22T06:44:34.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.144+0000 7f1281a3f640 1 -- 172.21.15.79:0/925175296 wait complete. 2024-04-22T06:44:34.153 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:33 smithi079 ceph-mon[27946]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-22T06:44:34.153 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:33 smithi079 ceph-mon[27946]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:34.153 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:33 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:44:34.153 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:33 smithi079 ceph-mon[27946]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:34.153 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:33 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2171461492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:34.154 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 -- 172.21.15.79:0/3364654626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8101a40 msgr2=0x7f45b8105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/3364654626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8101a40 0x7f45b8105a80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009900 tx=0x7f45ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 -- 172.21.15.79:0/3364654626 shutdown_connections 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/3364654626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45b8106160 0x7f45b810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/3364654626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8101a40 0x7f45b8105a80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/3364654626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8101090 0x7f45b8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.256+0000 7f45bf4b3640 1 -- 172.21.15.79:0/3364654626 >> 172.21.15.79:0/3364654626 conn(0x7f45b8077f50 msgr2=0x7f45b8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 -- 172.21.15.79:0/3364654626 shutdown_connections 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 -- 172.21.15.79:0/3364654626 wait complete. 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 Processor -- start 2024-04-22T06:44:34.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 -- start start 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8101090 0x7f45b81976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 0x7f45b8197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45b819bf60 0x7f45b819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81110a0 con 0x7f45b8106160 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f45b8110f20 con 0x7f45b819bf60 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bf4b3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45b8111220 con 0x7f45b8101090 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bca4e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 0x7f45b8197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bca4e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 0x7f45b8197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53622/0 (socket says 172.21.15.79:53622) 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.257+0000 7f45bca4e640 1 -- 172.21.15.79:0/171059583 learned_addr learned my addr 172.21.15.79:0/171059583 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bca4e640 1 -- 172.21.15.79:0/171059583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8101090 msgr2=0x7f45b81976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bd24f640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8101090 0x7f45b81976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bda50640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45b819bf60 0x7f45b819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bca4e640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8101090 0x7f45b81976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bca4e640 1 -- 172.21.15.79:0/171059583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45b819bf60 msgr2=0x7f45b819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bca4e640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45b819bf60 0x7f45b819c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bca4e640 1 -- 172.21.15.79:0/171059583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b819ca50 con 0x7f45b8106160 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bca4e640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 0x7f45b8197be0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f45ac036040 tx=0x7f45ac02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac043020 con 0x7f45b8106160 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ac02cbe0 con 0x7f45b8106160 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b819cce0 con 0x7f45b8106160 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac003bf0 con 0x7f45b8106160 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.258+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b81a4530 con 0x7f45b8106160 2024-04-22T06:44:34.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.259+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b8068d70 con 0x7f45b8106160 2024-04-22T06:44:34.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.260+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f45ac046020 con 0x7f45b8106160 2024-04-22T06:44:34.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.261+0000 7f45b67fc640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f45a00a3ec0 0x7f45a00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.261+0000 7f45bd24f640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f45a00a3ec0 0x7f45a00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.261+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f45ac0f2bf0 con 0x7f45b8106160 2024-04-22T06:44:34.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.261+0000 7f45bd24f640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f45a00a3ec0 0x7f45a00a6380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f45b00058c0 tx=0x7f45b0005c60 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.266+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45ac04d020 con 0x7f45b8106160 2024-04-22T06:44:34.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:33 smithi193 ceph-mon[31815]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-22T06:44:34.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:33 smithi193 ceph-mon[31815]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:34.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:33 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:44:34.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:33 smithi193 ceph-mon[31815]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:34.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:33 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2171461492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:34.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.443+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45b8101a40 con 0x7f45b8106160 2024-04-22T06:44:34.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.443+0000 7f45b67fc640 1 -- 172.21.15.79:0/171059583 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f45ac0bf180 con 0x7f45b8106160 2024-04-22T06:44:34.442 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f45a00a3ec0 msgr2=0x7f45a00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f45a00a3ec0 0x7f45a00a6380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f45b00058c0 tx=0x7f45b0005c60 comp rx=0 tx=0).stop 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 msgr2=0x7f45b8197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 0x7f45b8197be0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f45ac036040 tx=0x7f45ac02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 shutdown_connections 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f45a00a3ec0 0x7f45a00a6380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45b819bf60 0x7f45b819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45b8106160 0x7f45b8197be0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 --2- 172.21.15.79:0/171059583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8101090 0x7f45b81976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 >> 172.21.15.79:0/171059583 conn(0x7f45b8077f50 msgr2=0x7f45b80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 shutdown_connections 2024-04-22T06:44:34.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.445+0000 7f45bf4b3640 1 -- 172.21.15.79:0/171059583 wait complete. 2024-04-22T06:44:34.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:33 smithi190 ceph-mon[31421]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-22T06:44:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:33 smithi190 ceph-mon[31421]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:33 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:44:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:33 smithi190 ceph-mon[31421]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:33 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2171461492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:34.455 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 -- 172.21.15.79:0/3627426861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591806a880 msgr2=0x7f591806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 --2- 172.21.15.79:0/3627426861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591806a880 0x7f591806ac60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5908009900 tx=0x7f590802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 -- 172.21.15.79:0/3627426861 shutdown_connections 2024-04-22T06:44:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 --2- 172.21.15.79:0/3627426861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591806b230 0x7f5918106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 --2- 172.21.15.79:0/3627426861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 0x7f591810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 --2- 172.21.15.79:0/3627426861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591806a880 0x7f591806ac60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 -- 172.21.15.79:0/3627426861 >> 172.21.15.79:0/3627426861 conn(0x7f5918077eb0 msgr2=0x7f59180782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 -- 172.21.15.79:0/3627426861 shutdown_connections 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.557+0000 7f591f315640 1 -- 172.21.15.79:0/3627426861 wait complete. 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 Processor -- start 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 -- start start 2024-04-22T06:44:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591806a880 0x7f5918198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 0x7f5918198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 0x7f591819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f59181121e0 con 0x7f591819d020 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5918112060 con 0x7f591806a880 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591f315640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5918112360 con 0x7f5918107270 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 0x7f591819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 0x7f591819d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53638/0 (socket says 172.21.15.79:53638) 2024-04-22T06:44:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 -- 172.21.15.79:0/1574458714 learned_addr learned my addr 172.21.15.79:0/1574458714 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d0b1640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591806a880 0x7f5918198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591c8b0640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 0x7f5918198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 -- 172.21.15.79:0/1574458714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 msgr2=0x7f5918198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 0x7f5918198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 -- 172.21.15.79:0/1574458714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591806a880 msgr2=0x7f5918198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591806a880 0x7f5918198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 -- 172.21.15.79:0/1574458714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591819db10 con 0x7f591819d020 2024-04-22T06:44:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591c8b0640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 0x7f5918198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.558+0000 7f591d8b2640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 0x7f591819d4d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f591000cad0 tx=0x7f591000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.559+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591000ae20 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.559+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5910005020 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.559+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591819de00 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.559+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591000f6b0 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.559+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59181a55f0 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.560+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59180691c0 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.561+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f59100051d0 con 0x7f591819d020 2024-04-22T06:44:34.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.561+0000 7f590e7fc640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f59040a3e40 0x7f59040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.561+0000 7f591d0b1640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f59040a3e40 0x7f59040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.561+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f59100d0b10 con 0x7f591819d020 2024-04-22T06:44:34.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.562+0000 7f591d0b1640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f59040a3e40 0x7f59040a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5908000c00 tx=0x7f5908004040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.566+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5910014030 con 0x7f591819d020 2024-04-22T06:44:34.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.743+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f591806b230 con 0x7f591819d020 2024-04-22T06:44:34.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.743+0000 7f590e7fc640 1 -- 172.21.15.79:0/1574458714 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f591009cf90 con 0x7f591819d020 2024-04-22T06:44:34.742 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:34.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f59040a3e40 msgr2=0x7f59040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f59040a3e40 0x7f59040a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5908000c00 tx=0x7f5908004040 comp rx=0 tx=0).stop 2024-04-22T06:44:34.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 msgr2=0x7f591819d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 0x7f591819d4d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f591000cad0 tx=0x7f591000cf90 comp rx=0 tx=0).stop 2024-04-22T06:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 shutdown_connections 2024-04-22T06:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f591819d020 0x7f591819d4d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5918107270 0x7f5918198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f59040a3e40 0x7f59040a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 --2- 172.21.15.79:0/1574458714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f591806a880 0x7f5918198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 >> 172.21.15.79:0/1574458714 conn(0x7f5918077eb0 msgr2=0x7f59180ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 shutdown_connections 2024-04-22T06:44:34.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.745+0000 7f591f315640 1 -- 172.21.15.79:0/1574458714 wait complete. 2024-04-22T06:44:34.754 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:34.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 -- 172.21.15.79:0/2048090612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 msgr2=0x7f5a98101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2048090612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a98101620 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009930 tx=0x7f5a8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 -- 172.21.15.79:0/2048090612 shutdown_connections 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2048090612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a98101b60 0x7f5a9810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2048090612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a98101620 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2048090612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a98105e30 0x7f5a98106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 -- 172.21.15.79:0/2048090612 >> 172.21.15.79:0/2048090612 conn(0x7f5a98077e30 msgr2=0x7f5a98078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 -- 172.21.15.79:0/2048090612 shutdown_connections 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.853+0000 7f5a9f167640 1 -- 172.21.15.79:0/2048090612 wait complete. 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 Processor -- start 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 -- start start 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a9819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a98101b60 0x7f5a9819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a98105e30 0x7f5a981985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5a98112240 con 0x7f5a981011a0 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a981120c0 con 0x7f5a98101b60 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9f167640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5a981123c0 con 0x7f5a98105e30 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a9819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a9819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53658/0 (socket says 172.21.15.79:53658) 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 -- 172.21.15.79:0/2692240426 learned_addr learned my addr 172.21.15.79:0/2692240426 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 -- 172.21.15.79:0/2692240426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a98105e30 msgr2=0x7f5a981985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a98105e30 0x7f5a981985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 -- 172.21.15.79:0/2692240426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a98101b60 msgr2=0x7f5a9819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a98101b60 0x7f5a9819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.854+0000 7f5a9cf03640 1 -- 172.21.15.79:0/2692240426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a98198d20 con 0x7f5a981011a0 2024-04-22T06:44:34.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.855+0000 7f5a9cf03640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a9819e3f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5a8800d950 tx=0x7f5a8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.855+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a88017030 con 0x7f5a981011a0 2024-04-22T06:44:34.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.855+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a88004020 con 0x7f5a981011a0 2024-04-22T06:44:34.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.855+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a98199010 con 0x7f5a981011a0 2024-04-22T06:44:34.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.855+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880047a0 con 0x7f5a981011a0 2024-04-22T06:44:34.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.855+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a981a52b0 con 0x7f5a981011a0 2024-04-22T06:44:34.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.856+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a98068de0 con 0x7f5a981011a0 2024-04-22T06:44:34.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.857+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5a88009a30 con 0x7f5a981011a0 2024-04-22T06:44:34.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.858+0000 7f5a95ffb640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5a800a3ec0 0x7f5a800a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.858+0000 7f5a97fff640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5a800a3ec0 0x7f5a800a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.858+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f5a880d0570 con 0x7f5a981011a0 2024-04-22T06:44:34.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.858+0000 7f5a97fff640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5a800a3ec0 0x7f5a800a6380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c005770 tx=0x7f5a8c000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:34.862+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a8809c9f0 con 0x7f5a981011a0 2024-04-22T06:44:35.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.045+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a98199ad0 con 0x7f5a981011a0 2024-04-22T06:44:35.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.047+0000 7f5a95ffb640 1 -- 172.21.15.79:0/2692240426 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7f5a8809c2f0 con 0x7f5a981011a0 2024-04-22T06:44:35.045 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:35.046 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5a800a3ec0 msgr2=0x7f5a800a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5a800a3ec0 0x7f5a800a6380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c005770 tx=0x7f5a8c000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 msgr2=0x7f5a9819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a9819e3f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5a8800d950 tx=0x7f5a8800de10 comp rx=0 tx=0).stop 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 shutdown_connections 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5a800a3ec0 0x7f5a800a6380 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a98101b60 0x7f5a9819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a98105e30 0x7f5a981985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 --2- 172.21.15.79:0/2692240426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a981011a0 0x7f5a9819e3f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 >> 172.21.15.79:0/2692240426 conn(0x7f5a98077e30 msgr2=0x7f5a980fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 shutdown_connections 2024-04-22T06:44:35.048 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.049+0000 7f5a9f167640 1 -- 172.21.15.79:0/2692240426 wait complete. 2024-04-22T06:44:35.057 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/925175296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.058 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/171059583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.058 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:34 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1574458714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.059 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:35.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:35.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 -- 172.21.15.79:0/1690841576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c102350 msgr2=0x7fda8c10dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 --2- 172.21.15.79:0/1690841576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c102350 0x7fda8c10dd00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fda84009cd0 tx=0x7fda8402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 -- 172.21.15.79:0/1690841576 shutdown_connections 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 --2- 172.21.15.79:0/1690841576 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c102350 0x7fda8c10dd00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 --2- 172.21.15.79:0/1690841576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda8c101990 0x7fda8c101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 --2- 172.21.15.79:0/1690841576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 0x7fda8c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 -- 172.21.15.79:0/1690841576 >> 172.21.15.79:0/1690841576 conn(0x7fda8c0760e0 msgr2=0x7fda8c0764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 -- 172.21.15.79:0/1690841576 shutdown_connections 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.163+0000 7fda936f1640 1 -- 172.21.15.79:0/1690841576 wait complete. 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 Processor -- start 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 -- start start 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 0x7fda8c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda8c102350 0x7fda8c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 0x7fda8c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c112200 con 0x7fda8c101990 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c112080 con 0x7fda8c105e20 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda936f1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c112380 con 0x7fda8c102350 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 0x7fda8c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 0x7fda8c198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53664/0 (socket says 172.21.15.79:53664) 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 -- 172.21.15.79:0/1127430008 learned_addr learned my addr 172.21.15.79:0/1127430008 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda91c8e640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 0x7fda8c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda90c8c640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda8c102350 0x7fda8c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 -- 172.21.15.79:0/1127430008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda8c102350 msgr2=0x7fda8c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda8c102350 0x7fda8c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 -- 172.21.15.79:0/1127430008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 msgr2=0x7fda8c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 0x7fda8c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 -- 172.21.15.79:0/1127430008 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c19d620 con 0x7fda8c101990 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda91c8e640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 0x7fda8c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:35.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.164+0000 7fda9148d640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 0x7fda8c198680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fda8800d770 tx=0x7fda8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.165+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88017070 con 0x7fda8c101990 2024-04-22T06:44:35.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.165+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda88009c30 con 0x7fda8c101990 2024-04-22T06:44:35.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.165+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c19d910 con 0x7fda8c101990 2024-04-22T06:44:35.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.165+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88003cc0 con 0x7fda8c101990 2024-04-22T06:44:35.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.165+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c1a51a0 con 0x7fda8c101990 2024-04-22T06:44:35.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.166+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8c0691c0 con 0x7fda8c101990 2024-04-22T06:44:35.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.167+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fda88003e60 con 0x7fda8c101990 2024-04-22T06:44:35.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.167+0000 7fda827fc640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fda780a3e40 0x7fda780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.167+0000 7fda90c8c640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fda780a3e40 0x7fda780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.167+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fda880d04f0 con 0x7fda8c101990 2024-04-22T06:44:35.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.168+0000 7fda90c8c640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fda780a3e40 0x7fda780a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda8c076730 tx=0x7fda7c00a4c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.172+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda8809c970 con 0x7fda8c101990 2024-04-22T06:44:35.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.347+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda8c02d5c0 con 0x7fda8c101990 2024-04-22T06:44:35.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.348+0000 7fda827fc640 1 -- 172.21.15.79:0/1127430008 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fda88009de0 con 0x7fda8c101990 2024-04-22T06:44:35.346 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:35.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.349+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fda780a3e40 msgr2=0x7fda780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.349+0000 7fda936f1640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fda780a3e40 0x7fda780a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda8c076730 tx=0x7fda7c00a4c0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 msgr2=0x7fda8c198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 0x7fda8c198680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fda8800d770 tx=0x7fda8800dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 shutdown_connections 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fda780a3e40 0x7fda780a6300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda8c105e20 0x7fda8c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda8c102350 0x7fda8c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 --2- 172.21.15.79:0/1127430008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fda8c101990 0x7fda8c198680 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 >> 172.21.15.79:0/1127430008 conn(0x7fda8c0760e0 msgr2=0x7fda8c0ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 shutdown_connections 2024-04-22T06:44:35.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.350+0000 7fda936f1640 1 -- 172.21.15.79:0/1127430008 wait complete. 2024-04-22T06:44:35.359 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:35.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/925175296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/171059583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:34 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1574458714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/925175296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/171059583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:34 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1574458714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/2096759992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd44106160 msgr2=0x7fdd4410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/2096759992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd44106160 0x7fdd4410dd30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009900 tx=0x7fdd3c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/2096759992 shutdown_connections 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/2096759992 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd44106160 0x7fdd4410dd30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/2096759992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd44101a40 0x7fdd44105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/2096759992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd44101090 0x7fdd44101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.461+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/2096759992 >> 172.21.15.79:0/2096759992 conn(0x7fdd44077f50 msgr2=0x7fdd44078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/2096759992 shutdown_connections 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/2096759992 wait complete. 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 Processor -- start 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 -- start start 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd44101090 0x7fdd44197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd44106160 0x7fdd44197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 0x7fdd4419c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441110b0 con 0x7fdd4419be30 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd44110f30 con 0x7fdd44106160 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4bcc1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdd44111230 con 0x7fdd44101090 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4a25e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 0x7fdd4419c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4a25e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 0x7fdd4419c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53672/0 (socket says 172.21.15.79:53672) 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4a25e640 1 -- 172.21.15.79:0/3789797690 learned_addr learned my addr 172.21.15.79:0/3789797690 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd49a5d640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd44101090 0x7fdd44197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.462+0000 7fdd4925c640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd44106160 0x7fdd44197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4a25e640 1 -- 172.21.15.79:0/3789797690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd44101090 msgr2=0x7fdd44197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4a25e640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd44101090 0x7fdd44197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4a25e640 1 -- 172.21.15.79:0/3789797690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd44106160 msgr2=0x7fdd44197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4a25e640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd44106160 0x7fdd44197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4a25e640 1 -- 172.21.15.79:0/3789797690 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4419c920 con 0x7fdd4419be30 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4a25e640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 0x7fdd4419c2e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009230 tx=0x7fdd3c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c043020 con 0x7fdd4419be30 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c02cb70 con 0x7fdd4419be30 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4419cbb0 con 0x7fdd4419be30 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c003ac0 con 0x7fdd4419be30 2024-04-22T06:44:35.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.463+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd441a43d0 con 0x7fdd4419be30 2024-04-22T06:44:35.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.464+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd44068d70 con 0x7fdd4419be30 2024-04-22T06:44:35.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.465+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fdd3c046020 con 0x7fdd4419be30 2024-04-22T06:44:35.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.466+0000 7fdd3affd640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdd300a3e40 0x7fdd300a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.466+0000 7fdd49a5d640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdd300a3e40 0x7fdd300a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.466+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fdd3c0f2b50 con 0x7fdd4419be30 2024-04-22T06:44:35.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.466+0000 7fdd49a5d640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdd300a3e40 0x7fdd300a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd40007ec0 tx=0x7fdd400075b0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.470+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd3c04d020 con 0x7fdd4419be30 2024-04-22T06:44:35.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.645+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdd44101a40 con 0x7fdd4419be30 2024-04-22T06:44:35.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.646+0000 7fdd3affd640 1 -- 172.21.15.79:0/3789797690 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fdd3c0bf0e0 con 0x7fdd4419be30 2024-04-22T06:44:35.644 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:35.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdd300a3e40 msgr2=0x7fdd300a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdd300a3e40 0x7fdd300a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd40007ec0 tx=0x7fdd400075b0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 msgr2=0x7fdd4419c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 0x7fdd4419c2e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009230 tx=0x7fdd3c0045e0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 shutdown_connections 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdd300a3e40 0x7fdd300a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd44106160 0x7fdd44197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd4419be30 0x7fdd4419c2e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 --2- 172.21.15.79:0/3789797690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd44101090 0x7fdd44197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 >> 172.21.15.79:0/3789797690 conn(0x7fdd44077f50 msgr2=0x7fdd440fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 shutdown_connections 2024-04-22T06:44:35.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.648+0000 7fdd4bcc1640 1 -- 172.21.15.79:0/3789797690 wait complete. 2024-04-22T06:44:35.657 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.760+0000 7ff4c065c640 1 -- 172.21.15.79:0/2858807191 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 msgr2=0x7ff4b810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.760+0000 7ff4c065c640 1 --2- 172.21.15.79:0/2858807191 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b810ee30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009ca0 tx=0x7ff4b002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- 172.21.15.79:0/2858807191 shutdown_connections 2024-04-22T06:44:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 --2- 172.21.15.79:0/2858807191 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b810ee30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 --2- 172.21.15.79:0/2858807191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4b8102b40 0x7ff4b8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 --2- 172.21.15.79:0/2858807191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4b8102190 0x7ff4b8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- 172.21.15.79:0/2858807191 >> 172.21.15.79:0/2858807191 conn(0x7ff4b80793a0 msgr2=0x7ff4b80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- 172.21.15.79:0/2858807191 shutdown_connections 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- 172.21.15.79:0/2858807191 wait complete. 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 Processor -- start 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- start start 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4b8102190 0x7ff4b819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4b81985c0 0x7ff4b8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8112240 con 0x7ff4b8107260 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81120c0 con 0x7ff4b81985c0 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.761+0000 7ff4c065c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81123c0 con 0x7ff4b8102190 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53698/0 (socket says 172.21.15.79:53698) 2024-04-22T06:44:35.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 -- 172.21.15.79:0/247821679 learned_addr learned my addr 172.21.15.79:0/247821679 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4be3f8640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4b8102190 0x7ff4b819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 -- 172.21.15.79:0/247821679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4b8102190 msgr2=0x7ff4b819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4b8102190 0x7ff4b819e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 -- 172.21.15.79:0/247821679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4b81985c0 msgr2=0x7ff4b8198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bebf9640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4b81985c0 0x7ff4b8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4b81985c0 0x7ff4b8198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 -- 172.21.15.79:0/247821679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b81990e0 con 0x7ff4b8107260 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4bdbf7640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b819e950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800d740 tx=0x7ff4a800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8017070 con 0x7ff4b8107260 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4a800de50 con 0x7ff4b8107260 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8004ce0 con 0x7ff4b8107260 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b81993d0 con 0x7ff4b8107260 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.762+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b81a55d0 con 0x7ff4b8107260 2024-04-22T06:44:35.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.763+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b80691c0 con 0x7ff4b8107260 2024-04-22T06:44:35.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.764+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ff4a8004e80 con 0x7ff4b8107260 2024-04-22T06:44:35.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.765+0000 7ff4af7fe640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4a40a3e40 0x7ff4a40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.765+0000 7ff4be3f8640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4a40a3e40 0x7ff4a40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.765+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7ff4a80d04e0 con 0x7ff4b8107260 2024-04-22T06:44:35.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.765+0000 7ff4be3f8640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4a40a3e40 0x7ff4a40a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400a810 tx=0x7ff4b4008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.770+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff4a809c960 con 0x7ff4b8107260 2024-04-22T06:44:35.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.945+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff4b8102b40 con 0x7ff4b8107260 2024-04-22T06:44:35.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.946+0000 7ff4af7fe640 1 -- 172.21.15.79:0/247821679 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7ff4a809c260 con 0x7ff4b8107260 2024-04-22T06:44:35.944 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4a40a3e40 msgr2=0x7ff4a40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4a40a3e40 0x7ff4a40a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400a810 tx=0x7ff4b4008040 comp rx=0 tx=0).stop 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 msgr2=0x7ff4b819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b819e950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800d740 tx=0x7ff4a800dc00 comp rx=0 tx=0).stop 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 shutdown_connections 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4a40a3e40 0x7ff4a40a6300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4b81985c0 0x7ff4b8198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4b8107260 0x7ff4b819e950 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 --2- 172.21.15.79:0/247821679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4b8102190 0x7ff4b819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 >> 172.21.15.79:0/247821679 conn(0x7ff4b80793a0 msgr2=0x7ff4b80ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 shutdown_connections 2024-04-22T06:44:35.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:35.948+0000 7ff4c065c640 1 -- 172.21.15.79:0/247821679 wait complete. 2024-04-22T06:44:35.957 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-22T06:44:35.958 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:36.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 -- 172.21.15.79:0/1790673737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10101010f0 msgr2=0x7f10101014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 --2- 172.21.15.79:0/1790673737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10101010f0 0x7f10101014d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1004009900 tx=0x7f100402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.058 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 -- 172.21.15.79:0/1790673737 shutdown_connections 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 --2- 172.21.15.79:0/1790673737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1010101aa0 0x7f1010105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 --2- 172.21.15.79:0/1790673737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1010106170 0x7f101010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 --2- 172.21.15.79:0/1790673737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10101010f0 0x7f10101014d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 -- 172.21.15.79:0/1790673737 >> 172.21.15.79:0/1790673737 conn(0x7f1010077fe0 msgr2=0x7f10100783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 -- 172.21.15.79:0/1790673737 shutdown_connections 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 -- 172.21.15.79:0/1790673737 wait complete. 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 Processor -- start 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.057+0000 7f1015fb2640 1 -- start start 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10101010f0 0x7f1010199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1010106170 0x7f1010199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 0x7f10101a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f10101110e0 con 0x7f101019a050 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1010110f60 con 0x7f1010106170 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1010111260 con 0x7f10101010f0 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 0x7f10101a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 0x7f10101a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53710/0 (socket says 172.21.15.79:53710) 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 -- 172.21.15.79:0/888187817 learned_addr learned my addr 172.21.15.79:0/888187817 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100effd640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1010106170 0x7f1010199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100f7fe640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10101010f0 0x7f1010199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 -- 172.21.15.79:0/888187817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10101010f0 msgr2=0x7f1010199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10101010f0 0x7f1010199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 -- 172.21.15.79:0/888187817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1010106170 msgr2=0x7f1010199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1010106170 0x7f1010199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 -- 172.21.15.79:0/888187817 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10101a4e00 con 0x7f101019a050 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100ffff640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 0x7f10101a4600 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f100800e960 tx=0x7f100800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008004d20 con 0x7f101019a050 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10101a50f0 con 0x7f101019a050 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1008005020 con 0x7f101019a050 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080116d0 con 0x7f101019a050 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.058+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10101a5690 con 0x7f101019a050 2024-04-22T06:44:36.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.059+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1010069230 con 0x7f101019a050 2024-04-22T06:44:36.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.061+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f10080054a0 con 0x7f101019a050 2024-04-22T06:44:36.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.061+0000 7f100cff9640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0ff80a3e40 0x7f0ff80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.061+0000 7f100f7fe640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0ff80a3e40 0x7f0ff80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.061+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f10080d0f30 con 0x7f101019a050 2024-04-22T06:44:36.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.062+0000 7f100f7fe640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0ff80a3e40 0x7f0ff80a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1004000c00 tx=0x7f1004004040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.065+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1008016030 con 0x7f101019a050 2024-04-22T06:44:36.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.240+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1010109410 con 0x7f0ff80a3e40 2024-04-22T06:44:36.239 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:35 smithi079 ceph-mon[27946]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-22T06:44:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2692240426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1127430008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3789797690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:35 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/247821679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.247+0000 7f100cff9640 1 -- 172.21.15.79:0/888187817 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258026 (secure 0 0 0) 0x7f1004004200 con 0x7f0ff80a3e40 2024-04-22T06:44:36.248 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:36.249 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0ff80a3e40 msgr2=0x7f0ff80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0ff80a3e40 0x7f0ff80a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1004000c00 tx=0x7f1004004040 comp rx=0 tx=0).stop 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 msgr2=0x7f10101a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 0x7f10101a4600 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f100800e960 tx=0x7f100800ee20 comp rx=0 tx=0).stop 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 shutdown_connections 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0ff80a3e40 0x7f0ff80a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1010106170 0x7f1010199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f101019a050 0x7f10101a4600 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 --2- 172.21.15.79:0/888187817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10101010f0 0x7f1010199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 >> 172.21.15.79:0/888187817 conn(0x7f1010077fe0 msgr2=0x7f10100ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 shutdown_connections 2024-04-22T06:44:36.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.250+0000 7f1015fb2640 1 -- 172.21.15.79:0/888187817 wait complete. 2024-04-22T06:44:36.260 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:44:34.966027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190511,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1103,"num_read_kb":874,"num_write":499,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":883,"ondisk_log_size":883,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381832,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559992,"statfs":{"total":1151940427776,"available":1151549431808,"internally_reserved":0,"allocated":12140544,"data_stored":9146037,"data_compressed":223461,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":232,"num_write":28,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991997"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953258+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:44:24.953258+0000","last_peered":"2024-04-22T06:44:24.953258+0000","last_clean":"2024-04-22T06:44:24.953258+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:44:24.953258+0000","last_undegraded":"2024-04-22T06:44:24.953258+0000","last_fullsized":"2024-04-22T06:44:24.953258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194515+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:44:25.194515+0000","last_peered":"2024-04-22T06:44:25.194515+0000","last_clean":"2024-04-22T06:44:25.194515+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:44:25.194515+0000","last_undegraded":"2024-04-22T06:44:25.194515+0000","last_fullsized":"2024-04-22T06:44:25.194515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194358+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:44:25.194358+0000","last_peered":"2024-04-22T06:44:25.194358+0000","last_clean":"2024-04-22T06:44:25.194358+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.194358+0000","last_undegraded":"2024-04-22T06:44:25.194358+0000","last_fullsized":"2024-04-22T06:44:25.194358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193481+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:44:25.193481+0000","last_peered":"2024-04-22T06:44:25.193481+0000","last_clean":"2024-04-22T06:44:25.193481+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:44:25.193481+0000","last_undegraded":"2024-04-22T06:44:25.193481+0000","last_fullsized":"2024-04-22T06:44:25.193481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044020+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:44:25.044020+0000","last_peered":"2024-04-22T06:44:25.044020+0000","last_clean":"2024-04-22T06:44:25.044020+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:44:25.044020+0000","last_undegraded":"2024-04-22T06:44:25.044020+0000","last_fullsized":"2024-04-22T06:44:25.044020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951989+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:44:24.951989+0000","last_peered":"2024-04-22T06:44:24.951989+0000","last_clean":"2024-04-22T06:44:24.951989+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:44:24.951989+0000","last_undegraded":"2024-04-22T06:44:24.951989+0000","last_fullsized":"2024-04-22T06:44:24.951989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194834+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:44:25.194834+0000","last_peered":"2024-04-22T06:44:25.194834+0000","last_clean":"2024-04-22T06:44:25.194834+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:44:25.194834+0000","last_undegraded":"2024-04-22T06:44:25.194834+0000","last_fullsized":"2024-04-22T06:44:25.194834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044085+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:44:25.044085+0000","last_peered":"2024-04-22T06:44:25.044085+0000","last_clean":"2024-04-22T06:44:25.044085+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:44:25.044085+0000","last_undegraded":"2024-04-22T06:44:25.044085+0000","last_fullsized":"2024-04-22T06:44:25.044085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023622+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:44:25.023622+0000","last_peered":"2024-04-22T06:44:25.023622+0000","last_clean":"2024-04-22T06:44:25.023622+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:44:25.023622+0000","last_undegraded":"2024-04-22T06:44:25.023622+0000","last_fullsized":"2024-04-22T06:44:25.023622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952422+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:44:24.952422+0000","last_peered":"2024-04-22T06:44:24.952422+0000","last_clean":"2024-04-22T06:44:24.952422+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:44:24.952422+0000","last_undegraded":"2024-04-22T06:44:24.952422+0000","last_fullsized":"2024-04-22T06:44:24.952422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195320+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:44:25.195320+0000","last_peered":"2024-04-22T06:44:25.195320+0000","last_clean":"2024-04-22T06:44:25.195320+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:44:25.195320+0000","last_undegraded":"2024-04-22T06:44:25.195320+0000","last_fullsized":"2024-04-22T06:44:25.195320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193429+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:44:25.193429+0000","last_peered":"2024-04-22T06:44:25.193429+0000","last_clean":"2024-04-22T06:44:25.193429+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:44:25.193429+0000","last_undegraded":"2024-04-22T06:44:25.193429+0000","last_fullsized":"2024-04-22T06:44:25.193429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046192+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:44:25.046192+0000","last_peered":"2024-04-22T06:44:25.046192+0000","last_clean":"2024-04-22T06:44:25.046192+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:44:25.046192+0000","last_undegraded":"2024-04-22T06:44:25.046192+0000","last_fullsized":"2024-04-22T06:44:25.046192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194092+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:44:25.194092+0000","last_peered":"2024-04-22T06:44:25.194092+0000","last_clean":"2024-04-22T06:44:25.194092+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:44:25.194092+0000","last_undegraded":"2024-04-22T06:44:25.194092+0000","last_fullsized":"2024-04-22T06:44:25.194092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193990+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:44:25.193990+0000","last_peered":"2024-04-22T06:44:25.193990+0000","last_clean":"2024-04-22T06:44:25.193990+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:44:25.193990+0000","last_undegraded":"2024-04-22T06:44:25.193990+0000","last_fullsized":"2024-04-22T06:44:25.193990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023187+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:44:25.023187+0000","last_peered":"2024-04-22T06:44:25.023187+0000","last_clean":"2024-04-22T06:44:25.023187+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:44:25.023187+0000","last_undegraded":"2024-04-22T06:44:25.023187+0000","last_fullsized":"2024-04-22T06:44:25.023187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953321+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:44:24.953321+0000","last_peered":"2024-04-22T06:44:24.953321+0000","last_clean":"2024-04-22T06:44:24.953321+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:44:24.953321+0000","last_undegraded":"2024-04-22T06:44:24.953321+0000","last_fullsized":"2024-04-22T06:44:24.953321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948506+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:44:24.948506+0000","last_peered":"2024-04-22T06:44:24.948506+0000","last_clean":"2024-04-22T06:44:24.948506+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:44:24.948506+0000","last_undegraded":"2024-04-22T06:44:24.948506+0000","last_fullsized":"2024-04-22T06:44:24.948506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195467+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:44:25.195467+0000","last_peered":"2024-04-22T06:44:25.195467+0000","last_clean":"2024-04-22T06:44:25.195467+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:44:25.195467+0000","last_undegraded":"2024-04-22T06:44:25.195467+0000","last_fullsized":"2024-04-22T06:44:25.195467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948609+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:44:24.948609+0000","last_peered":"2024-04-22T06:44:24.948609+0000","last_clean":"2024-04-22T06:44:24.948609+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:44:24.948609+0000","last_undegraded":"2024-04-22T06:44:24.948609+0000","last_fullsized":"2024-04-22T06:44:24.948609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044653+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:44:25.044653+0000","last_peered":"2024-04-22T06:44:25.044653+0000","last_clean":"2024-04-22T06:44:25.044653+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:44:25.044653+0000","last_undegraded":"2024-04-22T06:44:25.044653+0000","last_fullsized":"2024-04-22T06:44:25.044653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044718+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:44:25.044718+0000","last_peered":"2024-04-22T06:44:25.044718+0000","last_clean":"2024-04-22T06:44:25.044718+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:44:25.044718+0000","last_undegraded":"2024-04-22T06:44:25.044718+0000","last_fullsized":"2024-04-22T06:44:25.044718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194508+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:44:25.194508+0000","last_peered":"2024-04-22T06:44:25.194508+0000","last_clean":"2024-04-22T06:44:25.194508+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:44:25.194508+0000","last_undegraded":"2024-04-22T06:44:25.194508+0000","last_fullsized":"2024-04-22T06:44:25.194508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044333+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:44:25.044333+0000","last_peered":"2024-04-22T06:44:25.044333+0000","last_clean":"2024-04-22T06:44:25.044333+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:44:25.044333+0000","last_undegraded":"2024-04-22T06:44:25.044333+0000","last_fullsized":"2024-04-22T06:44:25.044333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023387+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:44:25.023387+0000","last_peered":"2024-04-22T06:44:25.023387+0000","last_clean":"2024-04-22T06:44:25.023387+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:44:25.023387+0000","last_undegraded":"2024-04-22T06:44:25.023387+0000","last_fullsized":"2024-04-22T06:44:25.023387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948348+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:44:24.948348+0000","last_peered":"2024-04-22T06:44:24.948348+0000","last_clean":"2024-04-22T06:44:24.948348+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:44:24.948348+0000","last_undegraded":"2024-04-22T06:44:24.948348+0000","last_fullsized":"2024-04-22T06:44:24.948348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'10","reported_seq":160,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.895067+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:44:29.895067+0000","last_peered":"2024-04-22T06:44:29.895067+0000","last_clean":"2024-04-22T06:44:29.895067+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:44:29.895067+0000","last_undegraded":"2024-04-22T06:44:29.895067+0000","last_fullsized":"2024-04-22T06:44:29.895067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2327,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043961+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:44:25.043961+0000","last_peered":"2024-04-22T06:44:25.043961+0000","last_clean":"2024-04-22T06:44:25.043961+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:44:25.043961+0000","last_undegraded":"2024-04-22T06:44:25.043961+0000","last_fullsized":"2024-04-22T06:44:25.043961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951916+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:44:24.951916+0000","last_peered":"2024-04-22T06:44:24.951916+0000","last_clean":"2024-04-22T06:44:24.951916+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:44:24.951916+0000","last_undegraded":"2024-04-22T06:44:24.951916+0000","last_fullsized":"2024-04-22T06:44:24.951916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050784+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:44:25.050784+0000","last_peered":"2024-04-22T06:44:25.050784+0000","last_clean":"2024-04-22T06:44:25.050784+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:44:25.050784+0000","last_undegraded":"2024-04-22T06:44:25.050784+0000","last_fullsized":"2024-04-22T06:44:25.050784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045482+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:44:25.045482+0000","last_peered":"2024-04-22T06:44:25.045482+0000","last_clean":"2024-04-22T06:44:25.045482+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:44:25.045482+0000","last_undegraded":"2024-04-22T06:44:25.045482+0000","last_fullsized":"2024-04-22T06:44:25.045482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044260+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:44:25.044260+0000","last_peered":"2024-04-22T06:44:25.044260+0000","last_clean":"2024-04-22T06:44:25.044260+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:44:25.044260+0000","last_undegraded":"2024-04-22T06:44:25.044260+0000","last_fullsized":"2024-04-22T06:44:25.044260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050589+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:44:25.050589+0000","last_peered":"2024-04-22T06:44:25.050589+0000","last_clean":"2024-04-22T06:44:25.050589+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:44:25.050589+0000","last_undegraded":"2024-04-22T06:44:25.050589+0000","last_fullsized":"2024-04-22T06:44:25.050589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948311+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:44:24.948311+0000","last_peered":"2024-04-22T06:44:24.948311+0000","last_clean":"2024-04-22T06:44:24.948311+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:44:24.948311+0000","last_undegraded":"2024-04-22T06:44:24.948311+0000","last_fullsized":"2024-04-22T06:44:24.948311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.894612+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:44:29.894612+0000","last_peered":"2024-04-22T06:44:29.894612+0000","last_clean":"2024-04-22T06:44:29.894612+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:44:29.894612+0000","last_undegraded":"2024-04-22T06:44:29.894612+0000","last_fullsized":"2024-04-22T06:44:29.894612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023518+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:44:25.023518+0000","last_peered":"2024-04-22T06:44:25.023518+0000","last_clean":"2024-04-22T06:44:25.023518+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:44:25.023518+0000","last_undegraded":"2024-04-22T06:44:25.023518+0000","last_fullsized":"2024-04-22T06:44:25.023518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195175+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:44:25.195175+0000","last_peered":"2024-04-22T06:44:25.195175+0000","last_clean":"2024-04-22T06:44:25.195175+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:44:25.195175+0000","last_undegraded":"2024-04-22T06:44:25.195175+0000","last_fullsized":"2024-04-22T06:44:25.195175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194428+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:44:25.194428+0000","last_peered":"2024-04-22T06:44:25.194428+0000","last_clean":"2024-04-22T06:44:25.194428+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:44:25.194428+0000","last_undegraded":"2024-04-22T06:44:25.194428+0000","last_fullsized":"2024-04-22T06:44:25.194428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050921+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:44:25.050921+0000","last_peered":"2024-04-22T06:44:25.050921+0000","last_clean":"2024-04-22T06:44:25.050921+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:44:25.050921+0000","last_undegraded":"2024-04-22T06:44:25.050921+0000","last_fullsized":"2024-04-22T06:44:25.050921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023508+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:44:25.023508+0000","last_peered":"2024-04-22T06:44:25.023508+0000","last_clean":"2024-04-22T06:44:25.023508+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:44:25.023508+0000","last_undegraded":"2024-04-22T06:44:25.023508+0000","last_fullsized":"2024-04-22T06:44:25.023508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":315,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.895869+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:44:30.895869+0000","last_peered":"2024-04-22T06:44:30.895869+0000","last_clean":"2024-04-22T06:44:30.895869+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:44:30.895869+0000","last_undegraded":"2024-04-22T06:44:30.895869+0000","last_fullsized":"2024-04-22T06:44:30.895869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68464,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":195,"num_write":37,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044125+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:44:25.044125+0000","last_peered":"2024-04-22T06:44:25.044125+0000","last_clean":"2024-04-22T06:44:25.044125+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:44:25.044125+0000","last_undegraded":"2024-04-22T06:44:25.044125+0000","last_fullsized":"2024-04-22T06:44:25.044125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195568+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:44:25.195568+0000","last_peered":"2024-04-22T06:44:25.195568+0000","last_clean":"2024-04-22T06:44:25.195568+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:44:25.195568+0000","last_undegraded":"2024-04-22T06:44:25.195568+0000","last_fullsized":"2024-04-22T06:44:25.195568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045677+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:44:25.045677+0000","last_peered":"2024-04-22T06:44:25.045677+0000","last_clean":"2024-04-22T06:44:25.045677+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:44:25.045677+0000","last_undegraded":"2024-04-22T06:44:25.045677+0000","last_fullsized":"2024-04-22T06:44:25.045677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050734+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:44:25.050734+0000","last_peered":"2024-04-22T06:44:25.050734+0000","last_clean":"2024-04-22T06:44:25.050734+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:44:25.050734+0000","last_undegraded":"2024-04-22T06:44:25.050734+0000","last_fullsized":"2024-04-22T06:44:25.050734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195155+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:44:25.195155+0000","last_peered":"2024-04-22T06:44:25.195155+0000","last_clean":"2024-04-22T06:44:25.195155+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:44:25.195155+0000","last_undegraded":"2024-04-22T06:44:25.195155+0000","last_fullsized":"2024-04-22T06:44:25.195155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'7","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.894859+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:44:29.894859+0000","last_peered":"2024-04-22T06:44:29.894859+0000","last_clean":"2024-04-22T06:44:29.894859+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:44:29.894859+0000","last_undegraded":"2024-04-22T06:44:29.894859+0000","last_fullsized":"2024-04-22T06:44:29.894859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":7,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194722+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:44:25.194722+0000","last_peered":"2024-04-22T06:44:25.194722+0000","last_clean":"2024-04-22T06:44:25.194722+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:44:25.194722+0000","last_undegraded":"2024-04-22T06:44:25.194722+0000","last_fullsized":"2024-04-22T06:44:25.194722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043891+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:44:25.043891+0000","last_peered":"2024-04-22T06:44:25.043891+0000","last_clean":"2024-04-22T06:44:25.043891+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:44:25.043891+0000","last_undegraded":"2024-04-22T06:44:25.043891+0000","last_fullsized":"2024-04-22T06:44:25.043891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050670+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:44:25.050670+0000","last_peered":"2024-04-22T06:44:25.050670+0000","last_clean":"2024-04-22T06:44:25.050670+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:44:25.050670+0000","last_undegraded":"2024-04-22T06:44:25.050670+0000","last_fullsized":"2024-04-22T06:44:25.050670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044274+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:44:25.044274+0000","last_peered":"2024-04-22T06:44:25.044274+0000","last_clean":"2024-04-22T06:44:25.044274+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:44:25.044274+0000","last_undegraded":"2024-04-22T06:44:25.044274+0000","last_fullsized":"2024-04-22T06:44:25.044274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953070+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:44:24.953070+0000","last_peered":"2024-04-22T06:44:24.953070+0000","last_clean":"2024-04-22T06:44:24.953070+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:44:24.953070+0000","last_undegraded":"2024-04-22T06:44:24.953070+0000","last_fullsized":"2024-04-22T06:44:24.953070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948485+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:44:24.948485+0000","last_peered":"2024-04-22T06:44:24.948485+0000","last_clean":"2024-04-22T06:44:24.948485+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:44:24.948485+0000","last_undegraded":"2024-04-22T06:44:24.948485+0000","last_fullsized":"2024-04-22T06:44:24.948485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952238+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:44:24.952238+0000","last_peered":"2024-04-22T06:44:24.952238+0000","last_clean":"2024-04-22T06:44:24.952238+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:44:24.952238+0000","last_undegraded":"2024-04-22T06:44:24.952238+0000","last_fullsized":"2024-04-22T06:44:24.952238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195062+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:44:25.195062+0000","last_peered":"2024-04-22T06:44:25.195062+0000","last_clean":"2024-04-22T06:44:25.195062+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:44:25.195062+0000","last_undegraded":"2024-04-22T06:44:25.195062+0000","last_fullsized":"2024-04-22T06:44:25.195062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952360+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:44:24.952360+0000","last_peered":"2024-04-22T06:44:24.952360+0000","last_clean":"2024-04-22T06:44:24.952360+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:44:24.952360+0000","last_undegraded":"2024-04-22T06:44:24.952360+0000","last_fullsized":"2024-04-22T06:44:24.952360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050823+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:44:25.050823+0000","last_peered":"2024-04-22T06:44:25.050823+0000","last_clean":"2024-04-22T06:44:25.050823+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:44:25.050823+0000","last_undegraded":"2024-04-22T06:44:25.050823+0000","last_fullsized":"2024-04-22T06:44:25.050823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":151,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.895537+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:44:30.895537+0000","last_peered":"2024-04-22T06:44:30.895537+0000","last_clean":"2024-04-22T06:44:30.895537+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:44:30.895537+0000","last_undegraded":"2024-04-22T06:44:30.895537+0000","last_fullsized":"2024-04-22T06:44:30.895537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045472+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:44:25.045472+0000","last_peered":"2024-04-22T06:44:25.045472+0000","last_clean":"2024-04-22T06:44:25.045472+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:44:25.045472+0000","last_undegraded":"2024-04-22T06:44:25.045472+0000","last_fullsized":"2024-04-22T06:44:25.045472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193427+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:44:25.193427+0000","last_peered":"2024-04-22T06:44:25.193427+0000","last_clean":"2024-04-22T06:44:25.193427+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:44:25.193427+0000","last_undegraded":"2024-04-22T06:44:25.193427+0000","last_fullsized":"2024-04-22T06:44:25.193427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:27.055913+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:44:27.055913+0000","last_peered":"2024-04-22T06:44:27.055913+0000","last_clean":"2024-04-22T06:44:27.055913+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:44:27.055913+0000","last_undegraded":"2024-04-22T06:44:27.055913+0000","last_fullsized":"2024-04-22T06:44:27.055913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045340+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:44:25.045340+0000","last_peered":"2024-04-22T06:44:25.045340+0000","last_clean":"2024-04-22T06:44:25.045340+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:44:25.045340+0000","last_undegraded":"2024-04-22T06:44:25.045340+0000","last_fullsized":"2024-04-22T06:44:25.045340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194994+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:44:25.194994+0000","last_peered":"2024-04-22T06:44:25.194994+0000","last_clean":"2024-04-22T06:44:25.194994+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:44:25.194994+0000","last_undegraded":"2024-04-22T06:44:25.194994+0000","last_fullsized":"2024-04-22T06:44:25.194994+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046002+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:44:25.046002+0000","last_peered":"2024-04-22T06:44:25.046002+0000","last_clean":"2024-04-22T06:44:25.046002+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:44:25.046002+0000","last_undegraded":"2024-04-22T06:44:25.046002+0000","last_fullsized":"2024-04-22T06:44:25.046002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'46","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:26.693435+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:44:26.693435+0000","last_peered":"2024-04-22T06:44:26.693435+0000","last_clean":"2024-04-22T06:44:26.693435+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:44:26.693435+0000","last_undegraded":"2024-04-22T06:44:26.693435+0000","last_fullsized":"2024-04-22T06:44:26.693435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044531+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:44:25.044531+0000","last_peered":"2024-04-22T06:44:25.044531+0000","last_clean":"2024-04-22T06:44:25.044531+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:44:25.044531+0000","last_undegraded":"2024-04-22T06:44:25.044531+0000","last_fullsized":"2024-04-22T06:44:25.044531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193293+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:44:25.193293+0000","last_peered":"2024-04-22T06:44:25.193293+0000","last_clean":"2024-04-22T06:44:25.193293+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:44:25.193293+0000","last_undegraded":"2024-04-22T06:44:25.193293+0000","last_fullsized":"2024-04-22T06:44:25.193293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194372+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:44:25.194372+0000","last_peered":"2024-04-22T06:44:25.194372+0000","last_clean":"2024-04-22T06:44:25.194372+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:44:25.194372+0000","last_undegraded":"2024-04-22T06:44:25.194372+0000","last_fullsized":"2024-04-22T06:44:25.194372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023295+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:44:25.023295+0000","last_peered":"2024-04-22T06:44:25.023295+0000","last_clean":"2024-04-22T06:44:25.023295+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:44:25.023295+0000","last_undegraded":"2024-04-22T06:44:25.023295+0000","last_fullsized":"2024-04-22T06:44:25.023295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043912+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:44:25.043912+0000","last_peered":"2024-04-22T06:44:25.043912+0000","last_clean":"2024-04-22T06:44:25.043912+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:44:25.043912+0000","last_undegraded":"2024-04-22T06:44:25.043912+0000","last_fullsized":"2024-04-22T06:44:25.043912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044235+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:44:25.044235+0000","last_peered":"2024-04-22T06:44:25.044235+0000","last_clean":"2024-04-22T06:44:25.044235+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:44:25.044235+0000","last_undegraded":"2024-04-22T06:44:25.044235+0000","last_fullsized":"2024-04-22T06:44:25.044235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952874+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:44:24.952874+0000","last_peered":"2024-04-22T06:44:24.952874+0000","last_clean":"2024-04-22T06:44:24.952874+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:44:24.952874+0000","last_undegraded":"2024-04-22T06:44:24.952874+0000","last_fullsized":"2024-04-22T06:44:24.952874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953015+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:44:24.953015+0000","last_peered":"2024-04-22T06:44:24.953015+0000","last_clean":"2024-04-22T06:44:24.953015+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:44:24.953015+0000","last_undegraded":"2024-04-22T06:44:24.953015+0000","last_fullsized":"2024-04-22T06:44:24.953015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044498+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:44:25.044498+0000","last_peered":"2024-04-22T06:44:25.044498+0000","last_clean":"2024-04-22T06:44:25.044498+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:44:25.044498+0000","last_undegraded":"2024-04-22T06:44:25.044498+0000","last_fullsized":"2024-04-22T06:44:25.044498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952952+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:44:24.952952+0000","last_peered":"2024-04-22T06:44:24.952952+0000","last_clean":"2024-04-22T06:44:24.952952+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:44:24.952952+0000","last_undegraded":"2024-04-22T06:44:24.952952+0000","last_fullsized":"2024-04-22T06:44:24.952952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195201+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:44:25.195201+0000","last_peered":"2024-04-22T06:44:25.195201+0000","last_clean":"2024-04-22T06:44:25.195201+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:44:25.195201+0000","last_undegraded":"2024-04-22T06:44:25.195201+0000","last_fullsized":"2024-04-22T06:44:25.195201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195645+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:44:25.195645+0000","last_peered":"2024-04-22T06:44:25.195645+0000","last_clean":"2024-04-22T06:44:25.195645+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:44:25.195645+0000","last_undegraded":"2024-04-22T06:44:25.195645+0000","last_fullsized":"2024-04-22T06:44:25.195645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195419+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:44:25.195419+0000","last_peered":"2024-04-22T06:44:25.195419+0000","last_clean":"2024-04-22T06:44:25.195419+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:44:25.195419+0000","last_undegraded":"2024-04-22T06:44:25.195419+0000","last_fullsized":"2024-04-22T06:44:25.195419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.150698+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:44:25.150698+0000","last_peered":"2024-04-22T06:44:25.150698+0000","last_clean":"2024-04-22T06:44:25.150698+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:44:25.150698+0000","last_undegraded":"2024-04-22T06:44:25.150698+0000","last_fullsized":"2024-04-22T06:44:25.150698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195126+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:44:25.195126+0000","last_peered":"2024-04-22T06:44:25.195126+0000","last_clean":"2024-04-22T06:44:25.195126+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:44:25.195126+0000","last_undegraded":"2024-04-22T06:44:25.195126+0000","last_fullsized":"2024-04-22T06:44:25.195126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046237+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:44:25.046237+0000","last_peered":"2024-04-22T06:44:25.046237+0000","last_clean":"2024-04-22T06:44:25.046237+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:44:25.046237+0000","last_undegraded":"2024-04-22T06:44:25.046237+0000","last_fullsized":"2024-04-22T06:44:25.046237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195405+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:44:25.195405+0000","last_peered":"2024-04-22T06:44:25.195405+0000","last_clean":"2024-04-22T06:44:25.195405+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:44:25.195405+0000","last_undegraded":"2024-04-22T06:44:25.195405+0000","last_fullsized":"2024-04-22T06:44:25.195405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953214+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:44:24.953214+0000","last_peered":"2024-04-22T06:44:24.953214+0000","last_clean":"2024-04-22T06:44:24.953214+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:44:24.953214+0000","last_undegraded":"2024-04-22T06:44:24.953214+0000","last_fullsized":"2024-04-22T06:44:24.953214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195606+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:44:25.195606+0000","last_peered":"2024-04-22T06:44:25.195606+0000","last_clean":"2024-04-22T06:44:25.195606+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:44:25.195606+0000","last_undegraded":"2024-04-22T06:44:25.195606+0000","last_fullsized":"2024-04-22T06:44:25.195606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195452+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:44:25.195452+0000","last_peered":"2024-04-22T06:44:25.195452+0000","last_clean":"2024-04-22T06:44:25.195452+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:44:25.195452+0000","last_undegraded":"2024-04-22T06:44:25.195452+0000","last_fullsized":"2024-04-22T06:44:25.195452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194286+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:44:25.194286+0000","last_peered":"2024-04-22T06:44:25.194286+0000","last_clean":"2024-04-22T06:44:25.194286+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:44:25.194286+0000","last_undegraded":"2024-04-22T06:44:25.194286+0000","last_fullsized":"2024-04-22T06:44:25.194286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045832+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:44:25.045832+0000","last_peered":"2024-04-22T06:44:25.045832+0000","last_clean":"2024-04-22T06:44:25.045832+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:44:25.045832+0000","last_undegraded":"2024-04-22T06:44:25.045832+0000","last_fullsized":"2024-04-22T06:44:25.045832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953143+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:44:24.953143+0000","last_peered":"2024-04-22T06:44:24.953143+0000","last_clean":"2024-04-22T06:44:24.953143+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:44:24.953143+0000","last_undegraded":"2024-04-22T06:44:24.953143+0000","last_fullsized":"2024-04-22T06:44:24.953143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195715+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:44:25.195715+0000","last_peered":"2024-04-22T06:44:25.195715+0000","last_clean":"2024-04-22T06:44:25.195715+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:44:25.195715+0000","last_undegraded":"2024-04-22T06:44:25.195715+0000","last_fullsized":"2024-04-22T06:44:25.195715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044467+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:44:25.044467+0000","last_peered":"2024-04-22T06:44:25.044467+0000","last_clean":"2024-04-22T06:44:25.044467+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:44:25.044467+0000","last_undegraded":"2024-04-22T06:44:25.044467+0000","last_fullsized":"2024-04-22T06:44:25.044467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050646+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:44:25.050646+0000","last_peered":"2024-04-22T06:44:25.050646+0000","last_clean":"2024-04-22T06:44:25.050646+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:44:25.050646+0000","last_undegraded":"2024-04-22T06:44:25.050646+0000","last_fullsized":"2024-04-22T06:44:25.050646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044351+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:44:25.044351+0000","last_peered":"2024-04-22T06:44:25.044351+0000","last_clean":"2024-04-22T06:44:25.044351+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:44:25.044351+0000","last_undegraded":"2024-04-22T06:44:25.044351+0000","last_fullsized":"2024-04-22T06:44:25.044351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023389+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:44:25.023389+0000","last_peered":"2024-04-22T06:44:25.023389+0000","last_clean":"2024-04-22T06:44:25.023389+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:44:25.023389+0000","last_undegraded":"2024-04-22T06:44:25.023389+0000","last_fullsized":"2024-04-22T06:44:25.023389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045996+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:44:25.045996+0000","last_peered":"2024-04-22T06:44:25.045996+0000","last_clean":"2024-04-22T06:44:25.045996+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.045996+0000","last_undegraded":"2024-04-22T06:44:25.045996+0000","last_fullsized":"2024-04-22T06:44:25.045996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953024+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:44:24.953024+0000","last_peered":"2024-04-22T06:44:24.953024+0000","last_clean":"2024-04-22T06:44:24.953024+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:44:24.953024+0000","last_undegraded":"2024-04-22T06:44:24.953024+0000","last_fullsized":"2024-04-22T06:44:24.953024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599909,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":866,"num_read_kb":544,"num_write":296,"num_write_kb":1445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1799727,"data_compressed":81528,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":465124,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257983,"data_compressed":46638,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858576,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858496,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1059838,"data_compressed":38608,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":464749,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":533006,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":466020,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":536061,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989876,"data_compressed":28203,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466500,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1189808,"data_compressed":44764,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68464,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71519,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134781,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202956,"data_compressed":8030,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:36.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 -- 172.21.15.79:0/660220861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 msgr2=0x7f394806acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 --2- 172.21.15.79:0/660220861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394806acd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3944009900 tx=0x7f394402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 -- 172.21.15.79:0/660220861 shutdown_connections 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 --2- 172.21.15.79:0/660220861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f394806b2a0 0x7f3948106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 --2- 172.21.15.79:0/660220861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39481072f0 0x7f394810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 --2- 172.21.15.79:0/660220861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394806acd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 -- 172.21.15.79:0/660220861 >> 172.21.15.79:0/660220861 conn(0x7f3948077e30 msgr2=0x7f3948078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 -- 172.21.15.79:0/660220861 shutdown_connections 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.400+0000 7f394f4e0640 1 -- 172.21.15.79:0/660220861 wait complete. 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 Processor -- start 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 -- start start 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39481072f0 0x7f394819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39481985b0 0x7f3948198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3948112210 con 0x7f394806a8f0 2024-04-22T06:44:36.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3948112090 con 0x7f39481072f0 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394f4e0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3948112390 con 0x7f39481985b0 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53732/0 (socket says 172.21.15.79:53732) 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 -- 172.21.15.79:0/1755165747 learned_addr learned my addr 172.21.15.79:0/1755165747 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394da7d640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39481985b0 0x7f3948198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394ca7b640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39481072f0 0x7f394819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 -- 172.21.15.79:0/1755165747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39481985b0 msgr2=0x7f3948198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39481985b0 0x7f3948198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 -- 172.21.15.79:0/1755165747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39481072f0 msgr2=0x7f394819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39481072f0 0x7f394819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 -- 172.21.15.79:0/1755165747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39481990d0 con 0x7f394806a8f0 2024-04-22T06:44:36.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394ca7b640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39481072f0 0x7f394819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.401+0000 7f394d27c640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394819e400 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3944005940 tx=0x7f394402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.402+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944043020 con 0x7f394806a8f0 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.402+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394402cbe0 con 0x7f394806a8f0 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.402+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3948199360 con 0x7f394806a8f0 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.402+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944003bf0 con 0x7f394806a8f0 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.402+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39481a55b0 con 0x7f394806a8f0 2024-04-22T06:44:36.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.403+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3948069230 con 0x7f394806a8f0 2024-04-22T06:44:36.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.404+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3944046020 con 0x7f394806a8f0 2024-04-22T06:44:36.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.404+0000 7f393e7fc640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f39340a3e40 0x7f39340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.404+0000 7f394ca7b640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f39340a3e40 0x7f39340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.404+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f39440f32b0 con 0x7f394806a8f0 2024-04-22T06:44:36.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.405+0000 7f394ca7b640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f39340a3e40 0x7f39340a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39380058c0 tx=0x7f3938005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.408+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f394404d020 con 0x7f394806a8f0 2024-04-22T06:44:36.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:35 smithi193 ceph-mon[31815]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-22T06:44:36.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:35 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2692240426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:35 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1127430008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:35 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3789797690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:35 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/247821679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:35 smithi190 ceph-mon[31421]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-22T06:44:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:35 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2692240426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:35 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1127430008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:35 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3789797690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:35 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/247821679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.581+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f394810a510 con 0x7f39340a3e40 2024-04-22T06:44:36.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.588+0000 7f393e7fc640 1 -- 172.21.15.79:0/1755165747 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258026 (secure 0 0 0) 0x7f394810a510 con 0x7f39340a3e40 2024-04-22T06:44:36.589 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:36.590 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:44:36.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f39340a3e40 msgr2=0x7f39340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f39340a3e40 0x7f39340a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39380058c0 tx=0x7f3938005850 comp rx=0 tx=0).stop 2024-04-22T06:44:36.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 msgr2=0x7f394819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394819e400 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3944005940 tx=0x7f394402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 shutdown_connections 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f39340a3e40 0x7f39340a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39481072f0 0x7f394819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39481985b0 0x7f3948198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 --2- 172.21.15.79:0/1755165747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f394806a8f0 0x7f394819e400 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.591+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 >> 172.21.15.79:0/1755165747 conn(0x7f3948077e30 msgr2=0x7f39480ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.592+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 shutdown_connections 2024-04-22T06:44:36.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.592+0000 7f394f4e0640 1 -- 172.21.15.79:0/1755165747 wait complete. 2024-04-22T06:44:36.601 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:44:34.966027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190511,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1103,"num_read_kb":874,"num_write":499,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":883,"ondisk_log_size":883,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381832,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559992,"statfs":{"total":1151940427776,"available":1151549431808,"internally_reserved":0,"allocated":12140544,"data_stored":9146037,"data_compressed":223461,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":232,"num_write":28,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991997"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953258+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:44:24.953258+0000","last_peered":"2024-04-22T06:44:24.953258+0000","last_clean":"2024-04-22T06:44:24.953258+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:44:24.953258+0000","last_undegraded":"2024-04-22T06:44:24.953258+0000","last_fullsized":"2024-04-22T06:44:24.953258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194515+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:44:25.194515+0000","last_peered":"2024-04-22T06:44:25.194515+0000","last_clean":"2024-04-22T06:44:25.194515+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:44:25.194515+0000","last_undegraded":"2024-04-22T06:44:25.194515+0000","last_fullsized":"2024-04-22T06:44:25.194515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194358+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:44:25.194358+0000","last_peered":"2024-04-22T06:44:25.194358+0000","last_clean":"2024-04-22T06:44:25.194358+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.194358+0000","last_undegraded":"2024-04-22T06:44:25.194358+0000","last_fullsized":"2024-04-22T06:44:25.194358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193481+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:44:25.193481+0000","last_peered":"2024-04-22T06:44:25.193481+0000","last_clean":"2024-04-22T06:44:25.193481+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:44:25.193481+0000","last_undegraded":"2024-04-22T06:44:25.193481+0000","last_fullsized":"2024-04-22T06:44:25.193481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044020+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:44:25.044020+0000","last_peered":"2024-04-22T06:44:25.044020+0000","last_clean":"2024-04-22T06:44:25.044020+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:44:25.044020+0000","last_undegraded":"2024-04-22T06:44:25.044020+0000","last_fullsized":"2024-04-22T06:44:25.044020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951989+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:44:24.951989+0000","last_peered":"2024-04-22T06:44:24.951989+0000","last_clean":"2024-04-22T06:44:24.951989+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:44:24.951989+0000","last_undegraded":"2024-04-22T06:44:24.951989+0000","last_fullsized":"2024-04-22T06:44:24.951989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194834+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:44:25.194834+0000","last_peered":"2024-04-22T06:44:25.194834+0000","last_clean":"2024-04-22T06:44:25.194834+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:44:25.194834+0000","last_undegraded":"2024-04-22T06:44:25.194834+0000","last_fullsized":"2024-04-22T06:44:25.194834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044085+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:44:25.044085+0000","last_peered":"2024-04-22T06:44:25.044085+0000","last_clean":"2024-04-22T06:44:25.044085+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:44:25.044085+0000","last_undegraded":"2024-04-22T06:44:25.044085+0000","last_fullsized":"2024-04-22T06:44:25.044085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023622+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:44:25.023622+0000","last_peered":"2024-04-22T06:44:25.023622+0000","last_clean":"2024-04-22T06:44:25.023622+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:44:25.023622+0000","last_undegraded":"2024-04-22T06:44:25.023622+0000","last_fullsized":"2024-04-22T06:44:25.023622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952422+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:44:24.952422+0000","last_peered":"2024-04-22T06:44:24.952422+0000","last_clean":"2024-04-22T06:44:24.952422+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:44:24.952422+0000","last_undegraded":"2024-04-22T06:44:24.952422+0000","last_fullsized":"2024-04-22T06:44:24.952422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195320+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:44:25.195320+0000","last_peered":"2024-04-22T06:44:25.195320+0000","last_clean":"2024-04-22T06:44:25.195320+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:44:25.195320+0000","last_undegraded":"2024-04-22T06:44:25.195320+0000","last_fullsized":"2024-04-22T06:44:25.195320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193429+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:44:25.193429+0000","last_peered":"2024-04-22T06:44:25.193429+0000","last_clean":"2024-04-22T06:44:25.193429+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:44:25.193429+0000","last_undegraded":"2024-04-22T06:44:25.193429+0000","last_fullsized":"2024-04-22T06:44:25.193429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046192+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:44:25.046192+0000","last_peered":"2024-04-22T06:44:25.046192+0000","last_clean":"2024-04-22T06:44:25.046192+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:44:25.046192+0000","last_undegraded":"2024-04-22T06:44:25.046192+0000","last_fullsized":"2024-04-22T06:44:25.046192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194092+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:44:25.194092+0000","last_peered":"2024-04-22T06:44:25.194092+0000","last_clean":"2024-04-22T06:44:25.194092+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:44:25.194092+0000","last_undegraded":"2024-04-22T06:44:25.194092+0000","last_fullsized":"2024-04-22T06:44:25.194092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193990+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:44:25.193990+0000","last_peered":"2024-04-22T06:44:25.193990+0000","last_clean":"2024-04-22T06:44:25.193990+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:44:25.193990+0000","last_undegraded":"2024-04-22T06:44:25.193990+0000","last_fullsized":"2024-04-22T06:44:25.193990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023187+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:44:25.023187+0000","last_peered":"2024-04-22T06:44:25.023187+0000","last_clean":"2024-04-22T06:44:25.023187+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:44:25.023187+0000","last_undegraded":"2024-04-22T06:44:25.023187+0000","last_fullsized":"2024-04-22T06:44:25.023187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953321+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:44:24.953321+0000","last_peered":"2024-04-22T06:44:24.953321+0000","last_clean":"2024-04-22T06:44:24.953321+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:44:24.953321+0000","last_undegraded":"2024-04-22T06:44:24.953321+0000","last_fullsized":"2024-04-22T06:44:24.953321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948506+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:44:24.948506+0000","last_peered":"2024-04-22T06:44:24.948506+0000","last_clean":"2024-04-22T06:44:24.948506+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:44:24.948506+0000","last_undegraded":"2024-04-22T06:44:24.948506+0000","last_fullsized":"2024-04-22T06:44:24.948506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195467+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:44:25.195467+0000","last_peered":"2024-04-22T06:44:25.195467+0000","last_clean":"2024-04-22T06:44:25.195467+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:44:25.195467+0000","last_undegraded":"2024-04-22T06:44:25.195467+0000","last_fullsized":"2024-04-22T06:44:25.195467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948609+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:44:24.948609+0000","last_peered":"2024-04-22T06:44:24.948609+0000","last_clean":"2024-04-22T06:44:24.948609+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:44:24.948609+0000","last_undegraded":"2024-04-22T06:44:24.948609+0000","last_fullsized":"2024-04-22T06:44:24.948609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044653+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:44:25.044653+0000","last_peered":"2024-04-22T06:44:25.044653+0000","last_clean":"2024-04-22T06:44:25.044653+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:44:25.044653+0000","last_undegraded":"2024-04-22T06:44:25.044653+0000","last_fullsized":"2024-04-22T06:44:25.044653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044718+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:44:25.044718+0000","last_peered":"2024-04-22T06:44:25.044718+0000","last_clean":"2024-04-22T06:44:25.044718+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:44:25.044718+0000","last_undegraded":"2024-04-22T06:44:25.044718+0000","last_fullsized":"2024-04-22T06:44:25.044718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194508+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:44:25.194508+0000","last_peered":"2024-04-22T06:44:25.194508+0000","last_clean":"2024-04-22T06:44:25.194508+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:44:25.194508+0000","last_undegraded":"2024-04-22T06:44:25.194508+0000","last_fullsized":"2024-04-22T06:44:25.194508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044333+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:44:25.044333+0000","last_peered":"2024-04-22T06:44:25.044333+0000","last_clean":"2024-04-22T06:44:25.044333+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:44:25.044333+0000","last_undegraded":"2024-04-22T06:44:25.044333+0000","last_fullsized":"2024-04-22T06:44:25.044333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023387+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:44:25.023387+0000","last_peered":"2024-04-22T06:44:25.023387+0000","last_clean":"2024-04-22T06:44:25.023387+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:44:25.023387+0000","last_undegraded":"2024-04-22T06:44:25.023387+0000","last_fullsized":"2024-04-22T06:44:25.023387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948348+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:44:24.948348+0000","last_peered":"2024-04-22T06:44:24.948348+0000","last_clean":"2024-04-22T06:44:24.948348+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:44:24.948348+0000","last_undegraded":"2024-04-22T06:44:24.948348+0000","last_fullsized":"2024-04-22T06:44:24.948348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'10","reported_seq":160,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.895067+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:44:29.895067+0000","last_peered":"2024-04-22T06:44:29.895067+0000","last_clean":"2024-04-22T06:44:29.895067+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:44:29.895067+0000","last_undegraded":"2024-04-22T06:44:29.895067+0000","last_fullsized":"2024-04-22T06:44:29.895067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2327,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043961+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:44:25.043961+0000","last_peered":"2024-04-22T06:44:25.043961+0000","last_clean":"2024-04-22T06:44:25.043961+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:44:25.043961+0000","last_undegraded":"2024-04-22T06:44:25.043961+0000","last_fullsized":"2024-04-22T06:44:25.043961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951916+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:44:24.951916+0000","last_peered":"2024-04-22T06:44:24.951916+0000","last_clean":"2024-04-22T06:44:24.951916+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:44:24.951916+0000","last_undegraded":"2024-04-22T06:44:24.951916+0000","last_fullsized":"2024-04-22T06:44:24.951916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050784+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:44:25.050784+0000","last_peered":"2024-04-22T06:44:25.050784+0000","last_clean":"2024-04-22T06:44:25.050784+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:44:25.050784+0000","last_undegraded":"2024-04-22T06:44:25.050784+0000","last_fullsized":"2024-04-22T06:44:25.050784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045482+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:44:25.045482+0000","last_peered":"2024-04-22T06:44:25.045482+0000","last_clean":"2024-04-22T06:44:25.045482+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:44:25.045482+0000","last_undegraded":"2024-04-22T06:44:25.045482+0000","last_fullsized":"2024-04-22T06:44:25.045482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044260+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:44:25.044260+0000","last_peered":"2024-04-22T06:44:25.044260+0000","last_clean":"2024-04-22T06:44:25.044260+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:44:25.044260+0000","last_undegraded":"2024-04-22T06:44:25.044260+0000","last_fullsized":"2024-04-22T06:44:25.044260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050589+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:44:25.050589+0000","last_peered":"2024-04-22T06:44:25.050589+0000","last_clean":"2024-04-22T06:44:25.050589+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:44:25.050589+0000","last_undegraded":"2024-04-22T06:44:25.050589+0000","last_fullsized":"2024-04-22T06:44:25.050589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948311+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:44:24.948311+0000","last_peered":"2024-04-22T06:44:24.948311+0000","last_clean":"2024-04-22T06:44:24.948311+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:44:24.948311+0000","last_undegraded":"2024-04-22T06:44:24.948311+0000","last_fullsized":"2024-04-22T06:44:24.948311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.894612+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:44:29.894612+0000","last_peered":"2024-04-22T06:44:29.894612+0000","last_clean":"2024-04-22T06:44:29.894612+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:44:29.894612+0000","last_undegraded":"2024-04-22T06:44:29.894612+0000","last_fullsized":"2024-04-22T06:44:29.894612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023518+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:44:25.023518+0000","last_peered":"2024-04-22T06:44:25.023518+0000","last_clean":"2024-04-22T06:44:25.023518+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:44:25.023518+0000","last_undegraded":"2024-04-22T06:44:25.023518+0000","last_fullsized":"2024-04-22T06:44:25.023518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195175+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:44:25.195175+0000","last_peered":"2024-04-22T06:44:25.195175+0000","last_clean":"2024-04-22T06:44:25.195175+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:44:25.195175+0000","last_undegraded":"2024-04-22T06:44:25.195175+0000","last_fullsized":"2024-04-22T06:44:25.195175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194428+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:44:25.194428+0000","last_peered":"2024-04-22T06:44:25.194428+0000","last_clean":"2024-04-22T06:44:25.194428+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:44:25.194428+0000","last_undegraded":"2024-04-22T06:44:25.194428+0000","last_fullsized":"2024-04-22T06:44:25.194428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050921+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:44:25.050921+0000","last_peered":"2024-04-22T06:44:25.050921+0000","last_clean":"2024-04-22T06:44:25.050921+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:44:25.050921+0000","last_undegraded":"2024-04-22T06:44:25.050921+0000","last_fullsized":"2024-04-22T06:44:25.050921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023508+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:44:25.023508+0000","last_peered":"2024-04-22T06:44:25.023508+0000","last_clean":"2024-04-22T06:44:25.023508+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:44:25.023508+0000","last_undegraded":"2024-04-22T06:44:25.023508+0000","last_fullsized":"2024-04-22T06:44:25.023508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":315,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.895869+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:44:30.895869+0000","last_peered":"2024-04-22T06:44:30.895869+0000","last_clean":"2024-04-22T06:44:30.895869+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:44:30.895869+0000","last_undegraded":"2024-04-22T06:44:30.895869+0000","last_fullsized":"2024-04-22T06:44:30.895869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68464,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":195,"num_write":37,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044125+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:44:25.044125+0000","last_peered":"2024-04-22T06:44:25.044125+0000","last_clean":"2024-04-22T06:44:25.044125+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:44:25.044125+0000","last_undegraded":"2024-04-22T06:44:25.044125+0000","last_fullsized":"2024-04-22T06:44:25.044125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195568+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:44:25.195568+0000","last_peered":"2024-04-22T06:44:25.195568+0000","last_clean":"2024-04-22T06:44:25.195568+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:44:25.195568+0000","last_undegraded":"2024-04-22T06:44:25.195568+0000","last_fullsized":"2024-04-22T06:44:25.195568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045677+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:44:25.045677+0000","last_peered":"2024-04-22T06:44:25.045677+0000","last_clean":"2024-04-22T06:44:25.045677+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:44:25.045677+0000","last_undegraded":"2024-04-22T06:44:25.045677+0000","last_fullsized":"2024-04-22T06:44:25.045677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050734+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:44:25.050734+0000","last_peered":"2024-04-22T06:44:25.050734+0000","last_clean":"2024-04-22T06:44:25.050734+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:44:25.050734+0000","last_undegraded":"2024-04-22T06:44:25.050734+0000","last_fullsized":"2024-04-22T06:44:25.050734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195155+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:44:25.195155+0000","last_peered":"2024-04-22T06:44:25.195155+0000","last_clean":"2024-04-22T06:44:25.195155+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:44:25.195155+0000","last_undegraded":"2024-04-22T06:44:25.195155+0000","last_fullsized":"2024-04-22T06:44:25.195155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'7","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.894859+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:44:29.894859+0000","last_peered":"2024-04-22T06:44:29.894859+0000","last_clean":"2024-04-22T06:44:29.894859+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:44:29.894859+0000","last_undegraded":"2024-04-22T06:44:29.894859+0000","last_fullsized":"2024-04-22T06:44:29.894859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":7,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194722+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:44:25.194722+0000","last_peered":"2024-04-22T06:44:25.194722+0000","last_clean":"2024-04-22T06:44:25.194722+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:44:25.194722+0000","last_undegraded":"2024-04-22T06:44:25.194722+0000","last_fullsized":"2024-04-22T06:44:25.194722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043891+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:44:25.043891+0000","last_peered":"2024-04-22T06:44:25.043891+0000","last_clean":"2024-04-22T06:44:25.043891+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:44:25.043891+0000","last_undegraded":"2024-04-22T06:44:25.043891+0000","last_fullsized":"2024-04-22T06:44:25.043891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050670+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:44:25.050670+0000","last_peered":"2024-04-22T06:44:25.050670+0000","last_clean":"2024-04-22T06:44:25.050670+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:44:25.050670+0000","last_undegraded":"2024-04-22T06:44:25.050670+0000","last_fullsized":"2024-04-22T06:44:25.050670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044274+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:44:25.044274+0000","last_peered":"2024-04-22T06:44:25.044274+0000","last_clean":"2024-04-22T06:44:25.044274+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:44:25.044274+0000","last_undegraded":"2024-04-22T06:44:25.044274+0000","last_fullsized":"2024-04-22T06:44:25.044274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953070+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:44:24.953070+0000","last_peered":"2024-04-22T06:44:24.953070+0000","last_clean":"2024-04-22T06:44:24.953070+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:44:24.953070+0000","last_undegraded":"2024-04-22T06:44:24.953070+0000","last_fullsized":"2024-04-22T06:44:24.953070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948485+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:44:24.948485+0000","last_peered":"2024-04-22T06:44:24.948485+0000","last_clean":"2024-04-22T06:44:24.948485+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:44:24.948485+0000","last_undegraded":"2024-04-22T06:44:24.948485+0000","last_fullsized":"2024-04-22T06:44:24.948485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952238+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:44:24.952238+0000","last_peered":"2024-04-22T06:44:24.952238+0000","last_clean":"2024-04-22T06:44:24.952238+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:44:24.952238+0000","last_undegraded":"2024-04-22T06:44:24.952238+0000","last_fullsized":"2024-04-22T06:44:24.952238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195062+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:44:25.195062+0000","last_peered":"2024-04-22T06:44:25.195062+0000","last_clean":"2024-04-22T06:44:25.195062+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:44:25.195062+0000","last_undegraded":"2024-04-22T06:44:25.195062+0000","last_fullsized":"2024-04-22T06:44:25.195062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952360+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:44:24.952360+0000","last_peered":"2024-04-22T06:44:24.952360+0000","last_clean":"2024-04-22T06:44:24.952360+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:44:24.952360+0000","last_undegraded":"2024-04-22T06:44:24.952360+0000","last_fullsized":"2024-04-22T06:44:24.952360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050823+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:44:25.050823+0000","last_peered":"2024-04-22T06:44:25.050823+0000","last_clean":"2024-04-22T06:44:25.050823+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:44:25.050823+0000","last_undegraded":"2024-04-22T06:44:25.050823+0000","last_fullsized":"2024-04-22T06:44:25.050823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":151,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.895537+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:44:30.895537+0000","last_peered":"2024-04-22T06:44:30.895537+0000","last_clean":"2024-04-22T06:44:30.895537+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:44:30.895537+0000","last_undegraded":"2024-04-22T06:44:30.895537+0000","last_fullsized":"2024-04-22T06:44:30.895537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045472+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:44:25.045472+0000","last_peered":"2024-04-22T06:44:25.045472+0000","last_clean":"2024-04-22T06:44:25.045472+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:44:25.045472+0000","last_undegraded":"2024-04-22T06:44:25.045472+0000","last_fullsized":"2024-04-22T06:44:25.045472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193427+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:44:25.193427+0000","last_peered":"2024-04-22T06:44:25.193427+0000","last_clean":"2024-04-22T06:44:25.193427+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:44:25.193427+0000","last_undegraded":"2024-04-22T06:44:25.193427+0000","last_fullsized":"2024-04-22T06:44:25.193427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:27.055913+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:44:27.055913+0000","last_peered":"2024-04-22T06:44:27.055913+0000","last_clean":"2024-04-22T06:44:27.055913+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:44:27.055913+0000","last_undegraded":"2024-04-22T06:44:27.055913+0000","last_fullsized":"2024-04-22T06:44:27.055913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045340+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:44:25.045340+0000","last_peered":"2024-04-22T06:44:25.045340+0000","last_clean":"2024-04-22T06:44:25.045340+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:44:25.045340+0000","last_undegraded":"2024-04-22T06:44:25.045340+0000","last_fullsized":"2024-04-22T06:44:25.045340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194994+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:44:25.194994+0000","last_peered":"2024-04-22T06:44:25.194994+0000","last_clean":"2024-04-22T06:44:25.194994+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:44:25.194994+0000","last_undegraded":"2024-04-22T06:44:25.194994+0000","last_fullsized":"2024-04-22T06:44:25.194994+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046002+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:44:25.046002+0000","last_peered":"2024-04-22T06:44:25.046002+0000","last_clean":"2024-04-22T06:44:25.046002+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:44:25.046002+0000","last_undegraded":"2024-04-22T06:44:25.046002+0000","last_fullsized":"2024-04-22T06:44:25.046002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'46","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:26.693435+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:44:26.693435+0000","last_peered":"2024-04-22T06:44:26.693435+0000","last_clean":"2024-04-22T06:44:26.693435+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:44:26.693435+0000","last_undegraded":"2024-04-22T06:44:26.693435+0000","last_fullsized":"2024-04-22T06:44:26.693435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044531+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:44:25.044531+0000","last_peered":"2024-04-22T06:44:25.044531+0000","last_clean":"2024-04-22T06:44:25.044531+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:44:25.044531+0000","last_undegraded":"2024-04-22T06:44:25.044531+0000","last_fullsized":"2024-04-22T06:44:25.044531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193293+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:44:25.193293+0000","last_peered":"2024-04-22T06:44:25.193293+0000","last_clean":"2024-04-22T06:44:25.193293+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:44:25.193293+0000","last_undegraded":"2024-04-22T06:44:25.193293+0000","last_fullsized":"2024-04-22T06:44:25.193293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194372+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:44:25.194372+0000","last_peered":"2024-04-22T06:44:25.194372+0000","last_clean":"2024-04-22T06:44:25.194372+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:44:25.194372+0000","last_undegraded":"2024-04-22T06:44:25.194372+0000","last_fullsized":"2024-04-22T06:44:25.194372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023295+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:44:25.023295+0000","last_peered":"2024-04-22T06:44:25.023295+0000","last_clean":"2024-04-22T06:44:25.023295+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:44:25.023295+0000","last_undegraded":"2024-04-22T06:44:25.023295+0000","last_fullsized":"2024-04-22T06:44:25.023295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043912+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:44:25.043912+0000","last_peered":"2024-04-22T06:44:25.043912+0000","last_clean":"2024-04-22T06:44:25.043912+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:44:25.043912+0000","last_undegraded":"2024-04-22T06:44:25.043912+0000","last_fullsized":"2024-04-22T06:44:25.043912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044235+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:44:25.044235+0000","last_peered":"2024-04-22T06:44:25.044235+0000","last_clean":"2024-04-22T06:44:25.044235+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:44:25.044235+0000","last_undegraded":"2024-04-22T06:44:25.044235+0000","last_fullsized":"2024-04-22T06:44:25.044235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952874+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:44:24.952874+0000","last_peered":"2024-04-22T06:44:24.952874+0000","last_clean":"2024-04-22T06:44:24.952874+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:44:24.952874+0000","last_undegraded":"2024-04-22T06:44:24.952874+0000","last_fullsized":"2024-04-22T06:44:24.952874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953015+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:44:24.953015+0000","last_peered":"2024-04-22T06:44:24.953015+0000","last_clean":"2024-04-22T06:44:24.953015+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:44:24.953015+0000","last_undegraded":"2024-04-22T06:44:24.953015+0000","last_fullsized":"2024-04-22T06:44:24.953015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044498+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:44:25.044498+0000","last_peered":"2024-04-22T06:44:25.044498+0000","last_clean":"2024-04-22T06:44:25.044498+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:44:25.044498+0000","last_undegraded":"2024-04-22T06:44:25.044498+0000","last_fullsized":"2024-04-22T06:44:25.044498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952952+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:44:24.952952+0000","last_peered":"2024-04-22T06:44:24.952952+0000","last_clean":"2024-04-22T06:44:24.952952+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:44:24.952952+0000","last_undegraded":"2024-04-22T06:44:24.952952+0000","last_fullsized":"2024-04-22T06:44:24.952952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195201+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:44:25.195201+0000","last_peered":"2024-04-22T06:44:25.195201+0000","last_clean":"2024-04-22T06:44:25.195201+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:44:25.195201+0000","last_undegraded":"2024-04-22T06:44:25.195201+0000","last_fullsized":"2024-04-22T06:44:25.195201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195645+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:44:25.195645+0000","last_peered":"2024-04-22T06:44:25.195645+0000","last_clean":"2024-04-22T06:44:25.195645+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:44:25.195645+0000","last_undegraded":"2024-04-22T06:44:25.195645+0000","last_fullsized":"2024-04-22T06:44:25.195645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195419+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:44:25.195419+0000","last_peered":"2024-04-22T06:44:25.195419+0000","last_clean":"2024-04-22T06:44:25.195419+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:44:25.195419+0000","last_undegraded":"2024-04-22T06:44:25.195419+0000","last_fullsized":"2024-04-22T06:44:25.195419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.150698+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:44:25.150698+0000","last_peered":"2024-04-22T06:44:25.150698+0000","last_clean":"2024-04-22T06:44:25.150698+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:44:25.150698+0000","last_undegraded":"2024-04-22T06:44:25.150698+0000","last_fullsized":"2024-04-22T06:44:25.150698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195126+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:44:25.195126+0000","last_peered":"2024-04-22T06:44:25.195126+0000","last_clean":"2024-04-22T06:44:25.195126+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:44:25.195126+0000","last_undegraded":"2024-04-22T06:44:25.195126+0000","last_fullsized":"2024-04-22T06:44:25.195126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046237+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:44:25.046237+0000","last_peered":"2024-04-22T06:44:25.046237+0000","last_clean":"2024-04-22T06:44:25.046237+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:44:25.046237+0000","last_undegraded":"2024-04-22T06:44:25.046237+0000","last_fullsized":"2024-04-22T06:44:25.046237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195405+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:44:25.195405+0000","last_peered":"2024-04-22T06:44:25.195405+0000","last_clean":"2024-04-22T06:44:25.195405+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:44:25.195405+0000","last_undegraded":"2024-04-22T06:44:25.195405+0000","last_fullsized":"2024-04-22T06:44:25.195405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953214+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:44:24.953214+0000","last_peered":"2024-04-22T06:44:24.953214+0000","last_clean":"2024-04-22T06:44:24.953214+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:44:24.953214+0000","last_undegraded":"2024-04-22T06:44:24.953214+0000","last_fullsized":"2024-04-22T06:44:24.953214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195606+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:44:25.195606+0000","last_peered":"2024-04-22T06:44:25.195606+0000","last_clean":"2024-04-22T06:44:25.195606+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:44:25.195606+0000","last_undegraded":"2024-04-22T06:44:25.195606+0000","last_fullsized":"2024-04-22T06:44:25.195606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195452+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:44:25.195452+0000","last_peered":"2024-04-22T06:44:25.195452+0000","last_clean":"2024-04-22T06:44:25.195452+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:44:25.195452+0000","last_undegraded":"2024-04-22T06:44:25.195452+0000","last_fullsized":"2024-04-22T06:44:25.195452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194286+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:44:25.194286+0000","last_peered":"2024-04-22T06:44:25.194286+0000","last_clean":"2024-04-22T06:44:25.194286+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:44:25.194286+0000","last_undegraded":"2024-04-22T06:44:25.194286+0000","last_fullsized":"2024-04-22T06:44:25.194286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045832+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:44:25.045832+0000","last_peered":"2024-04-22T06:44:25.045832+0000","last_clean":"2024-04-22T06:44:25.045832+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:44:25.045832+0000","last_undegraded":"2024-04-22T06:44:25.045832+0000","last_fullsized":"2024-04-22T06:44:25.045832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953143+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:44:24.953143+0000","last_peered":"2024-04-22T06:44:24.953143+0000","last_clean":"2024-04-22T06:44:24.953143+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:44:24.953143+0000","last_undegraded":"2024-04-22T06:44:24.953143+0000","last_fullsized":"2024-04-22T06:44:24.953143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195715+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:44:25.195715+0000","last_peered":"2024-04-22T06:44:25.195715+0000","last_clean":"2024-04-22T06:44:25.195715+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:44:25.195715+0000","last_undegraded":"2024-04-22T06:44:25.195715+0000","last_fullsized":"2024-04-22T06:44:25.195715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044467+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:44:25.044467+0000","last_peered":"2024-04-22T06:44:25.044467+0000","last_clean":"2024-04-22T06:44:25.044467+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:44:25.044467+0000","last_undegraded":"2024-04-22T06:44:25.044467+0000","last_fullsized":"2024-04-22T06:44:25.044467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050646+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:44:25.050646+0000","last_peered":"2024-04-22T06:44:25.050646+0000","last_clean":"2024-04-22T06:44:25.050646+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:44:25.050646+0000","last_undegraded":"2024-04-22T06:44:25.050646+0000","last_fullsized":"2024-04-22T06:44:25.050646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044351+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:44:25.044351+0000","last_peered":"2024-04-22T06:44:25.044351+0000","last_clean":"2024-04-22T06:44:25.044351+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:44:25.044351+0000","last_undegraded":"2024-04-22T06:44:25.044351+0000","last_fullsized":"2024-04-22T06:44:25.044351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023389+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:44:25.023389+0000","last_peered":"2024-04-22T06:44:25.023389+0000","last_clean":"2024-04-22T06:44:25.023389+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:44:25.023389+0000","last_undegraded":"2024-04-22T06:44:25.023389+0000","last_fullsized":"2024-04-22T06:44:25.023389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045996+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:44:25.045996+0000","last_peered":"2024-04-22T06:44:25.045996+0000","last_clean":"2024-04-22T06:44:25.045996+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.045996+0000","last_undegraded":"2024-04-22T06:44:25.045996+0000","last_fullsized":"2024-04-22T06:44:25.045996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953024+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:44:24.953024+0000","last_peered":"2024-04-22T06:44:24.953024+0000","last_clean":"2024-04-22T06:44:24.953024+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:44:24.953024+0000","last_undegraded":"2024-04-22T06:44:24.953024+0000","last_fullsized":"2024-04-22T06:44:24.953024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599909,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":866,"num_read_kb":544,"num_write":296,"num_write_kb":1445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1799727,"data_compressed":81528,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":465124,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257983,"data_compressed":46638,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858576,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858496,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1059838,"data_compressed":38608,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":464749,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":533006,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":466020,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":536061,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989876,"data_compressed":28203,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466500,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1189808,"data_compressed":44764,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68464,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71519,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134781,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202956,"data_compressed":8030,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:36.605 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-22T06:44:36.605 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:36.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 -- 172.21.15.79:0/2573976471 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c069aa0 msgr2=0x7f550c10edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.741 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 --2- 172.21.15.79:0/2573976471 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c069aa0 0x7f550c10edd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5504009cd0 tx=0x7f550402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 -- 172.21.15.79:0/2573976471 shutdown_connections 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 --2- 172.21.15.79:0/2573976471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c0690e0 0x7f550c069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 --2- 172.21.15.79:0/2573976471 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c069aa0 0x7f550c10edd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 --2- 172.21.15.79:0/2573976471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c108030 0x7f550c108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 -- 172.21.15.79:0/2573976471 >> 172.21.15.79:0/2573976471 conn(0x7f550c077e30 msgr2=0x7f550c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 -- 172.21.15.79:0/2573976471 shutdown_connections 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.741+0000 7f5511d6d640 1 -- 172.21.15.79:0/2573976471 wait complete. 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 Processor -- start 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 -- start start 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 0x7f550c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c069aa0 0x7f550c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c108030 0x7f550c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f550c1121c0 con 0x7f550c0690e0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f550c112040 con 0x7f550c069aa0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f5511d6d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f550c112340 con 0x7f550c108030 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 0x7f550c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 0x7f550c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53746/0 (socket says 172.21.15.79:53746) 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 -- 172.21.15.79:0/343282102 learned_addr learned my addr 172.21.15.79:0/343282102 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550affd640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c069aa0 0x7f550c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550bfff640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c108030 0x7f550c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 -- 172.21.15.79:0/343282102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c108030 msgr2=0x7f550c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c108030 0x7f550c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 -- 172.21.15.79:0/343282102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c069aa0 msgr2=0x7f550c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c069aa0 0x7f550c198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.742+0000 7f550b7fe640 1 -- 172.21.15.79:0/343282102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c19d6d0 con 0x7f550c0690e0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.743+0000 7f550b7fe640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 0x7f550c1986e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f550000b4a0 tx=0x7f550000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.743+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500014070 con 0x7f550c0690e0 2024-04-22T06:44:36.743 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.743+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5500010ac0 con 0x7f550c0690e0 2024-04-22T06:44:36.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.743+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550c19d9c0 con 0x7f550c0690e0 2024-04-22T06:44:36.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.743+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500017c30 con 0x7f550c0690e0 2024-04-22T06:44:36.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.743+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550c1a5250 con 0x7f550c0690e0 2024-04-22T06:44:36.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.744+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550c068de0 con 0x7f550c0690e0 2024-04-22T06:44:36.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.745+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5500017070 con 0x7f550c0690e0 2024-04-22T06:44:36.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.746+0000 7f5508ff9640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f54f40a3e40 0x7f54f40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.746+0000 7f550affd640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f54f40a3e40 0x7f54f40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.746+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f55000d0050 con 0x7f550c0690e0 2024-04-22T06:44:36.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.746+0000 7f550affd640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f54f40a3e40 0x7f54f40a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f550c1076e0 tx=0x7f54fc00a2d0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.750+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f550009c5e0 con 0x7f550c0690e0 2024-04-22T06:44:36.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.924+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f550c062390 con 0x7f54f40a3e40 2024-04-22T06:44:36.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.931+0000 7f5508ff9640 1 -- 172.21.15.79:0/343282102 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258026 (secure 0 0 0) 0x7f550c062390 con 0x7f54f40a3e40 2024-04-22T06:44:36.933 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:36.934 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-04-22T06:44:36.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f54f40a3e40 msgr2=0x7f54f40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f54f40a3e40 0x7f54f40a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f550c1076e0 tx=0x7f54fc00a2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 msgr2=0x7f550c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 0x7f550c1986e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f550000b4a0 tx=0x7f550000b960 comp rx=0 tx=0).stop 2024-04-22T06:44:36.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 shutdown_connections 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f54f40a3e40 0x7f54f40a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f550c069aa0 0x7f550c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f550c108030 0x7f550c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 --2- 172.21.15.79:0/343282102 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f550c0690e0 0x7f550c1986e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 >> 172.21.15.79:0/343282102 conn(0x7f550c077e30 msgr2=0x7f550c0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.934+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 shutdown_connections 2024-04-22T06:44:36.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:36.935+0000 7f5511d6d640 1 -- 172.21.15.79:0/343282102 wait complete. 2024-04-22T06:44:36.944 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:44:34.966027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190511,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1103,"num_read_kb":874,"num_write":499,"num_write_kb":4385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":883,"ondisk_log_size":883,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381832,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559992,"statfs":{"total":1151940427776,"available":1151549431808,"internally_reserved":0,"allocated":12140544,"data_stored":9146037,"data_compressed":223461,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3693,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":232,"num_write":28,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991997"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953258+0000","last_change":"2024-04-22T06:42:24.200032+0000","last_active":"2024-04-22T06:44:24.953258+0000","last_peered":"2024-04-22T06:44:24.953258+0000","last_clean":"2024-04-22T06:44:24.953258+0000","last_became_active":"2024-04-22T06:42:24.199669+0000","last_became_peered":"2024-04-22T06:42:24.199669+0000","last_unstale":"2024-04-22T06:44:24.953258+0000","last_undegraded":"2024-04-22T06:44:24.953258+0000","last_fullsized":"2024-04-22T06:44:24.953258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:45.975157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194515+0000","last_change":"2024-04-22T06:42:24.188905+0000","last_active":"2024-04-22T06:44:25.194515+0000","last_peered":"2024-04-22T06:44:25.194515+0000","last_clean":"2024-04-22T06:44:25.194515+0000","last_became_active":"2024-04-22T06:42:24.188618+0000","last_became_peered":"2024-04-22T06:42:24.188618+0000","last_unstale":"2024-04-22T06:44:25.194515+0000","last_undegraded":"2024-04-22T06:44:25.194515+0000","last_fullsized":"2024-04-22T06:44:25.194515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:38:22.996389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194358+0000","last_change":"2024-04-22T06:42:24.187633+0000","last_active":"2024-04-22T06:44:25.194358+0000","last_peered":"2024-04-22T06:44:25.194358+0000","last_clean":"2024-04-22T06:44:25.194358+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.194358+0000","last_undegraded":"2024-04-22T06:44:25.194358+0000","last_fullsized":"2024-04-22T06:44:25.194358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:51:01.338972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193481+0000","last_change":"2024-04-22T06:42:24.194819+0000","last_active":"2024-04-22T06:44:25.193481+0000","last_peered":"2024-04-22T06:44:25.193481+0000","last_clean":"2024-04-22T06:44:25.193481+0000","last_became_active":"2024-04-22T06:42:24.192704+0000","last_became_peered":"2024-04-22T06:42:24.192704+0000","last_unstale":"2024-04-22T06:44:25.193481+0000","last_undegraded":"2024-04-22T06:44:25.193481+0000","last_fullsized":"2024-04-22T06:44:25.193481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:06.113272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044020+0000","last_change":"2024-04-22T06:42:24.187764+0000","last_active":"2024-04-22T06:44:25.044020+0000","last_peered":"2024-04-22T06:44:25.044020+0000","last_clean":"2024-04-22T06:44:25.044020+0000","last_became_active":"2024-04-22T06:42:24.186171+0000","last_became_peered":"2024-04-22T06:42:24.186171+0000","last_unstale":"2024-04-22T06:44:25.044020+0000","last_undegraded":"2024-04-22T06:44:25.044020+0000","last_fullsized":"2024-04-22T06:44:25.044020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.407725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951989+0000","last_change":"2024-04-22T06:42:24.200665+0000","last_active":"2024-04-22T06:44:24.951989+0000","last_peered":"2024-04-22T06:44:24.951989+0000","last_clean":"2024-04-22T06:44:24.951989+0000","last_became_active":"2024-04-22T06:42:24.200248+0000","last_became_peered":"2024-04-22T06:42:24.200248+0000","last_unstale":"2024-04-22T06:44:24.951989+0000","last_undegraded":"2024-04-22T06:44:24.951989+0000","last_fullsized":"2024-04-22T06:44:24.951989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:22:45.121784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194834+0000","last_change":"2024-04-22T06:42:24.197270+0000","last_active":"2024-04-22T06:44:25.194834+0000","last_peered":"2024-04-22T06:44:25.194834+0000","last_clean":"2024-04-22T06:44:25.194834+0000","last_became_active":"2024-04-22T06:42:24.196655+0000","last_became_peered":"2024-04-22T06:42:24.196655+0000","last_unstale":"2024-04-22T06:44:25.194834+0000","last_undegraded":"2024-04-22T06:44:25.194834+0000","last_fullsized":"2024-04-22T06:44:25.194834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:34:42.904317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044085+0000","last_change":"2024-04-22T06:42:24.183033+0000","last_active":"2024-04-22T06:44:25.044085+0000","last_peered":"2024-04-22T06:44:25.044085+0000","last_clean":"2024-04-22T06:44:25.044085+0000","last_became_active":"2024-04-22T06:42:24.182676+0000","last_became_peered":"2024-04-22T06:42:24.182676+0000","last_unstale":"2024-04-22T06:44:25.044085+0000","last_undegraded":"2024-04-22T06:44:25.044085+0000","last_fullsized":"2024-04-22T06:44:25.044085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:47.343920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023622+0000","last_change":"2024-04-22T06:42:24.191453+0000","last_active":"2024-04-22T06:44:25.023622+0000","last_peered":"2024-04-22T06:44:25.023622+0000","last_clean":"2024-04-22T06:44:25.023622+0000","last_became_active":"2024-04-22T06:42:24.190795+0000","last_became_peered":"2024-04-22T06:42:24.190795+0000","last_unstale":"2024-04-22T06:44:25.023622+0000","last_undegraded":"2024-04-22T06:44:25.023622+0000","last_fullsized":"2024-04-22T06:44:25.023622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:50:24.028550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952422+0000","last_change":"2024-04-22T06:43:17.814042+0000","last_active":"2024-04-22T06:44:24.952422+0000","last_peered":"2024-04-22T06:44:24.952422+0000","last_clean":"2024-04-22T06:44:24.952422+0000","last_became_active":"2024-04-22T06:43:17.813731+0000","last_became_peered":"2024-04-22T06:43:17.813731+0000","last_unstale":"2024-04-22T06:44:24.952422+0000","last_undegraded":"2024-04-22T06:44:24.952422+0000","last_fullsized":"2024-04-22T06:44:24.952422+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:10:59.878138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195320+0000","last_change":"2024-04-22T06:42:24.192445+0000","last_active":"2024-04-22T06:44:25.195320+0000","last_peered":"2024-04-22T06:44:25.195320+0000","last_clean":"2024-04-22T06:44:25.195320+0000","last_became_active":"2024-04-22T06:42:24.192087+0000","last_became_peered":"2024-04-22T06:42:24.192087+0000","last_unstale":"2024-04-22T06:44:25.195320+0000","last_undegraded":"2024-04-22T06:44:25.195320+0000","last_fullsized":"2024-04-22T06:44:25.195320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:44:14.393773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193429+0000","last_change":"2024-04-22T06:42:24.194336+0000","last_active":"2024-04-22T06:44:25.193429+0000","last_peered":"2024-04-22T06:44:25.193429+0000","last_clean":"2024-04-22T06:44:25.193429+0000","last_became_active":"2024-04-22T06:42:24.193416+0000","last_became_peered":"2024-04-22T06:42:24.193416+0000","last_unstale":"2024-04-22T06:44:25.193429+0000","last_undegraded":"2024-04-22T06:44:25.193429+0000","last_fullsized":"2024-04-22T06:44:25.193429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:04.029606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046192+0000","last_change":"2024-04-22T06:42:24.179301+0000","last_active":"2024-04-22T06:44:25.046192+0000","last_peered":"2024-04-22T06:44:25.046192+0000","last_clean":"2024-04-22T06:44:25.046192+0000","last_became_active":"2024-04-22T06:42:24.177847+0000","last_became_peered":"2024-04-22T06:42:24.177847+0000","last_unstale":"2024-04-22T06:44:25.046192+0000","last_undegraded":"2024-04-22T06:44:25.046192+0000","last_fullsized":"2024-04-22T06:44:25.046192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:30.234019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194092+0000","last_change":"2024-04-22T06:42:24.185847+0000","last_active":"2024-04-22T06:44:25.194092+0000","last_peered":"2024-04-22T06:44:25.194092+0000","last_clean":"2024-04-22T06:44:25.194092+0000","last_became_active":"2024-04-22T06:42:24.185517+0000","last_became_peered":"2024-04-22T06:42:24.185517+0000","last_unstale":"2024-04-22T06:44:25.194092+0000","last_undegraded":"2024-04-22T06:44:25.194092+0000","last_fullsized":"2024-04-22T06:44:25.194092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:04:13.692012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193990+0000","last_change":"2024-04-22T06:42:24.196685+0000","last_active":"2024-04-22T06:44:25.193990+0000","last_peered":"2024-04-22T06:44:25.193990+0000","last_clean":"2024-04-22T06:44:25.193990+0000","last_became_active":"2024-04-22T06:42:24.196348+0000","last_became_peered":"2024-04-22T06:42:24.196348+0000","last_unstale":"2024-04-22T06:44:25.193990+0000","last_undegraded":"2024-04-22T06:44:25.193990+0000","last_fullsized":"2024-04-22T06:44:25.193990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:06.089635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023187+0000","last_change":"2024-04-22T06:42:24.190163+0000","last_active":"2024-04-22T06:44:25.023187+0000","last_peered":"2024-04-22T06:44:25.023187+0000","last_clean":"2024-04-22T06:44:25.023187+0000","last_became_active":"2024-04-22T06:42:24.189806+0000","last_became_peered":"2024-04-22T06:42:24.189806+0000","last_unstale":"2024-04-22T06:44:25.023187+0000","last_undegraded":"2024-04-22T06:44:25.023187+0000","last_fullsized":"2024-04-22T06:44:25.023187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:40:12.753847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953321+0000","last_change":"2024-04-22T06:42:24.198921+0000","last_active":"2024-04-22T06:44:24.953321+0000","last_peered":"2024-04-22T06:44:24.953321+0000","last_clean":"2024-04-22T06:44:24.953321+0000","last_became_active":"2024-04-22T06:42:24.195979+0000","last_became_peered":"2024-04-22T06:42:24.195979+0000","last_unstale":"2024-04-22T06:44:24.953321+0000","last_undegraded":"2024-04-22T06:44:24.953321+0000","last_fullsized":"2024-04-22T06:44:24.953321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:34:56.560112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948506+0000","last_change":"2024-04-22T06:42:24.185905+0000","last_active":"2024-04-22T06:44:24.948506+0000","last_peered":"2024-04-22T06:44:24.948506+0000","last_clean":"2024-04-22T06:44:24.948506+0000","last_became_active":"2024-04-22T06:42:24.185591+0000","last_became_peered":"2024-04-22T06:42:24.185591+0000","last_unstale":"2024-04-22T06:44:24.948506+0000","last_undegraded":"2024-04-22T06:44:24.948506+0000","last_fullsized":"2024-04-22T06:44:24.948506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:59:30.507819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195467+0000","last_change":"2024-04-22T06:42:24.192301+0000","last_active":"2024-04-22T06:44:25.195467+0000","last_peered":"2024-04-22T06:44:25.195467+0000","last_clean":"2024-04-22T06:44:25.195467+0000","last_became_active":"2024-04-22T06:42:24.191665+0000","last_became_peered":"2024-04-22T06:42:24.191665+0000","last_unstale":"2024-04-22T06:44:25.195467+0000","last_undegraded":"2024-04-22T06:44:25.195467+0000","last_fullsized":"2024-04-22T06:44:25.195467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:04:41.828770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948609+0000","last_change":"2024-04-22T06:42:23.673226+0000","last_active":"2024-04-22T06:44:24.948609+0000","last_peered":"2024-04-22T06:44:24.948609+0000","last_clean":"2024-04-22T06:44:24.948609+0000","last_became_active":"2024-04-22T06:42:23.672891+0000","last_became_peered":"2024-04-22T06:42:23.672891+0000","last_unstale":"2024-04-22T06:44:24.948609+0000","last_undegraded":"2024-04-22T06:44:24.948609+0000","last_fullsized":"2024-04-22T06:44:24.948609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:06:34.535637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044653+0000","last_change":"2024-04-22T06:42:24.191909+0000","last_active":"2024-04-22T06:44:25.044653+0000","last_peered":"2024-04-22T06:44:25.044653+0000","last_clean":"2024-04-22T06:44:25.044653+0000","last_became_active":"2024-04-22T06:42:24.191404+0000","last_became_peered":"2024-04-22T06:42:24.191404+0000","last_unstale":"2024-04-22T06:44:25.044653+0000","last_undegraded":"2024-04-22T06:44:25.044653+0000","last_fullsized":"2024-04-22T06:44:25.044653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:21:21.039694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044718+0000","last_change":"2024-04-22T06:42:23.677108+0000","last_active":"2024-04-22T06:44:25.044718+0000","last_peered":"2024-04-22T06:44:25.044718+0000","last_clean":"2024-04-22T06:44:25.044718+0000","last_became_active":"2024-04-22T06:42:23.676697+0000","last_became_peered":"2024-04-22T06:42:23.676697+0000","last_unstale":"2024-04-22T06:44:25.044718+0000","last_undegraded":"2024-04-22T06:44:25.044718+0000","last_fullsized":"2024-04-22T06:44:25.044718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:58.232663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194508+0000","last_change":"2024-04-22T06:42:24.186801+0000","last_active":"2024-04-22T06:44:25.194508+0000","last_peered":"2024-04-22T06:44:25.194508+0000","last_clean":"2024-04-22T06:44:25.194508+0000","last_became_active":"2024-04-22T06:42:24.186467+0000","last_became_peered":"2024-04-22T06:42:24.186467+0000","last_unstale":"2024-04-22T06:44:25.194508+0000","last_undegraded":"2024-04-22T06:44:25.194508+0000","last_fullsized":"2024-04-22T06:44:25.194508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:57.770241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044333+0000","last_change":"2024-04-22T06:42:23.673280+0000","last_active":"2024-04-22T06:44:25.044333+0000","last_peered":"2024-04-22T06:44:25.044333+0000","last_clean":"2024-04-22T06:44:25.044333+0000","last_became_active":"2024-04-22T06:42:23.672903+0000","last_became_peered":"2024-04-22T06:42:23.672903+0000","last_unstale":"2024-04-22T06:44:25.044333+0000","last_undegraded":"2024-04-22T06:44:25.044333+0000","last_fullsized":"2024-04-22T06:44:25.044333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:04:00.188820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023387+0000","last_change":"2024-04-22T06:42:24.183448+0000","last_active":"2024-04-22T06:44:25.023387+0000","last_peered":"2024-04-22T06:44:25.023387+0000","last_clean":"2024-04-22T06:44:25.023387+0000","last_became_active":"2024-04-22T06:42:24.182368+0000","last_became_peered":"2024-04-22T06:42:24.182368+0000","last_unstale":"2024-04-22T06:44:25.023387+0000","last_undegraded":"2024-04-22T06:44:25.023387+0000","last_fullsized":"2024-04-22T06:44:25.023387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:32.185321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.672864+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.672496+0000","last_became_peered":"2024-04-22T06:42:23.672496+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:08:31.991746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948348+0000","last_change":"2024-04-22T06:43:18.084301+0000","last_active":"2024-04-22T06:44:24.948348+0000","last_peered":"2024-04-22T06:44:24.948348+0000","last_clean":"2024-04-22T06:44:24.948348+0000","last_became_active":"2024-04-22T06:43:18.084037+0000","last_became_peered":"2024-04-22T06:43:18.084037+0000","last_unstale":"2024-04-22T06:44:24.948348+0000","last_undegraded":"2024-04-22T06:44:24.948348+0000","last_fullsized":"2024-04-22T06:44:24.948348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:51:13.199742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'10","reported_seq":160,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.895067+0000","last_change":"2024-04-22T06:42:23.673162+0000","last_active":"2024-04-22T06:44:29.895067+0000","last_peered":"2024-04-22T06:44:29.895067+0000","last_clean":"2024-04-22T06:44:29.895067+0000","last_became_active":"2024-04-22T06:42:23.672812+0000","last_became_peered":"2024-04-22T06:42:23.672812+0000","last_unstale":"2024-04-22T06:44:29.895067+0000","last_undegraded":"2024-04-22T06:44:29.895067+0000","last_fullsized":"2024-04-22T06:44:29.895067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:23:58.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2327,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043961+0000","last_change":"2024-04-22T06:42:24.190778+0000","last_active":"2024-04-22T06:44:25.043961+0000","last_peered":"2024-04-22T06:44:25.043961+0000","last_clean":"2024-04-22T06:44:25.043961+0000","last_became_active":"2024-04-22T06:42:24.188993+0000","last_became_peered":"2024-04-22T06:42:24.188993+0000","last_unstale":"2024-04-22T06:44:25.043961+0000","last_undegraded":"2024-04-22T06:44:25.043961+0000","last_fullsized":"2024-04-22T06:44:25.043961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:24.932410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.951916+0000","last_change":"2024-04-22T06:42:23.678591+0000","last_active":"2024-04-22T06:44:24.951916+0000","last_peered":"2024-04-22T06:44:24.951916+0000","last_clean":"2024-04-22T06:44:24.951916+0000","last_became_active":"2024-04-22T06:42:23.678195+0000","last_became_peered":"2024-04-22T06:42:23.678195+0000","last_unstale":"2024-04-22T06:44:24.951916+0000","last_undegraded":"2024-04-22T06:44:24.951916+0000","last_fullsized":"2024-04-22T06:44:24.951916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:54.205986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050784+0000","last_change":"2024-04-22T06:42:24.194730+0000","last_active":"2024-04-22T06:44:25.050784+0000","last_peered":"2024-04-22T06:44:25.050784+0000","last_clean":"2024-04-22T06:44:25.050784+0000","last_became_active":"2024-04-22T06:42:24.194365+0000","last_became_peered":"2024-04-22T06:42:24.194365+0000","last_unstale":"2024-04-22T06:44:25.050784+0000","last_undegraded":"2024-04-22T06:44:25.050784+0000","last_fullsized":"2024-04-22T06:44:25.050784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:38:40.860986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045482+0000","last_change":"2024-04-22T06:42:23.667238+0000","last_active":"2024-04-22T06:44:25.045482+0000","last_peered":"2024-04-22T06:44:25.045482+0000","last_clean":"2024-04-22T06:44:25.045482+0000","last_became_active":"2024-04-22T06:42:23.666836+0000","last_became_peered":"2024-04-22T06:42:23.666836+0000","last_unstale":"2024-04-22T06:44:25.045482+0000","last_undegraded":"2024-04-22T06:44:25.045482+0000","last_fullsized":"2024-04-22T06:44:25.045482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:36.549110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044260+0000","last_change":"2024-04-22T06:42:24.189762+0000","last_active":"2024-04-22T06:44:25.044260+0000","last_peered":"2024-04-22T06:44:25.044260+0000","last_clean":"2024-04-22T06:44:25.044260+0000","last_became_active":"2024-04-22T06:42:24.189448+0000","last_became_peered":"2024-04-22T06:42:24.189448+0000","last_unstale":"2024-04-22T06:44:25.044260+0000","last_undegraded":"2024-04-22T06:44:25.044260+0000","last_fullsized":"2024-04-22T06:44:25.044260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:44.555410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050589+0000","last_change":"2024-04-22T06:42:23.671038+0000","last_active":"2024-04-22T06:44:25.050589+0000","last_peered":"2024-04-22T06:44:25.050589+0000","last_clean":"2024-04-22T06:44:25.050589+0000","last_became_active":"2024-04-22T06:42:23.670740+0000","last_became_peered":"2024-04-22T06:42:23.670740+0000","last_unstale":"2024-04-22T06:44:25.050589+0000","last_undegraded":"2024-04-22T06:44:25.050589+0000","last_fullsized":"2024-04-22T06:44:25.050589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:42:47.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948311+0000","last_change":"2024-04-22T06:42:24.182670+0000","last_active":"2024-04-22T06:44:24.948311+0000","last_peered":"2024-04-22T06:44:24.948311+0000","last_clean":"2024-04-22T06:44:24.948311+0000","last_became_active":"2024-04-22T06:42:24.182330+0000","last_became_peered":"2024-04-22T06:42:24.182330+0000","last_unstale":"2024-04-22T06:44:24.948311+0000","last_undegraded":"2024-04-22T06:44:24.948311+0000","last_fullsized":"2024-04-22T06:44:24.948311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:28:28.027743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.894612+0000","last_change":"2024-04-22T06:42:23.675465+0000","last_active":"2024-04-22T06:44:29.894612+0000","last_peered":"2024-04-22T06:44:29.894612+0000","last_clean":"2024-04-22T06:44:29.894612+0000","last_became_active":"2024-04-22T06:42:23.674986+0000","last_became_peered":"2024-04-22T06:42:23.674986+0000","last_unstale":"2024-04-22T06:44:29.894612+0000","last_undegraded":"2024-04-22T06:44:29.894612+0000","last_fullsized":"2024-04-22T06:44:29.894612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:00:14.264093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023518+0000","last_change":"2024-04-22T06:42:24.194665+0000","last_active":"2024-04-22T06:44:25.023518+0000","last_peered":"2024-04-22T06:44:25.023518+0000","last_clean":"2024-04-22T06:44:25.023518+0000","last_became_active":"2024-04-22T06:42:24.193918+0000","last_became_peered":"2024-04-22T06:42:24.193918+0000","last_unstale":"2024-04-22T06:44:25.023518+0000","last_undegraded":"2024-04-22T06:44:25.023518+0000","last_fullsized":"2024-04-22T06:44:25.023518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:20.628016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195175+0000","last_change":"2024-04-22T06:42:23.673753+0000","last_active":"2024-04-22T06:44:25.195175+0000","last_peered":"2024-04-22T06:44:25.195175+0000","last_clean":"2024-04-22T06:44:25.195175+0000","last_became_active":"2024-04-22T06:42:23.673196+0000","last_became_peered":"2024-04-22T06:42:23.673196+0000","last_unstale":"2024-04-22T06:44:25.195175+0000","last_undegraded":"2024-04-22T06:44:25.195175+0000","last_fullsized":"2024-04-22T06:44:25.195175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:35:20.842559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194428+0000","last_change":"2024-04-22T06:42:24.199646+0000","last_active":"2024-04-22T06:44:25.194428+0000","last_peered":"2024-04-22T06:44:25.194428+0000","last_clean":"2024-04-22T06:44:25.194428+0000","last_became_active":"2024-04-22T06:42:24.199413+0000","last_became_peered":"2024-04-22T06:42:24.199413+0000","last_unstale":"2024-04-22T06:44:25.194428+0000","last_undegraded":"2024-04-22T06:44:25.194428+0000","last_fullsized":"2024-04-22T06:44:25.194428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:38.271325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050921+0000","last_change":"2024-04-22T06:42:23.667945+0000","last_active":"2024-04-22T06:44:25.050921+0000","last_peered":"2024-04-22T06:44:25.050921+0000","last_clean":"2024-04-22T06:44:25.050921+0000","last_became_active":"2024-04-22T06:42:23.667600+0000","last_became_peered":"2024-04-22T06:42:23.667600+0000","last_unstale":"2024-04-22T06:44:25.050921+0000","last_undegraded":"2024-04-22T06:44:25.050921+0000","last_fullsized":"2024-04-22T06:44:25.050921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:42:42.594240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023508+0000","last_change":"2024-04-22T06:42:24.193731+0000","last_active":"2024-04-22T06:44:25.023508+0000","last_peered":"2024-04-22T06:44:25.023508+0000","last_clean":"2024-04-22T06:44:25.023508+0000","last_became_active":"2024-04-22T06:42:24.193366+0000","last_became_peered":"2024-04-22T06:42:24.193366+0000","last_unstale":"2024-04-22T06:44:25.023508+0000","last_undegraded":"2024-04-22T06:44:25.023508+0000","last_fullsized":"2024-04-22T06:44:25.023508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:37:06.703379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":315,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.895869+0000","last_change":"2024-04-22T06:42:23.677991+0000","last_active":"2024-04-22T06:44:30.895869+0000","last_peered":"2024-04-22T06:44:30.895869+0000","last_clean":"2024-04-22T06:44:30.895869+0000","last_became_active":"2024-04-22T06:42:23.677552+0000","last_became_peered":"2024-04-22T06:42:23.677552+0000","last_unstale":"2024-04-22T06:44:30.895869+0000","last_undegraded":"2024-04-22T06:44:30.895869+0000","last_fullsized":"2024-04-22T06:44:30.895869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:39.422881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68464,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":195,"num_write":37,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044125+0000","last_change":"2024-04-22T06:42:24.190556+0000","last_active":"2024-04-22T06:44:25.044125+0000","last_peered":"2024-04-22T06:44:25.044125+0000","last_clean":"2024-04-22T06:44:25.044125+0000","last_became_active":"2024-04-22T06:42:24.190234+0000","last_became_peered":"2024-04-22T06:42:24.190234+0000","last_unstale":"2024-04-22T06:44:25.044125+0000","last_undegraded":"2024-04-22T06:44:25.044125+0000","last_fullsized":"2024-04-22T06:44:25.044125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:30.239748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195568+0000","last_change":"2024-04-22T06:42:23.681096+0000","last_active":"2024-04-22T06:44:25.195568+0000","last_peered":"2024-04-22T06:44:25.195568+0000","last_clean":"2024-04-22T06:44:25.195568+0000","last_became_active":"2024-04-22T06:42:23.680768+0000","last_became_peered":"2024-04-22T06:42:23.680768+0000","last_unstale":"2024-04-22T06:44:25.195568+0000","last_undegraded":"2024-04-22T06:44:25.195568+0000","last_fullsized":"2024-04-22T06:44:25.195568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:24:40.065210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045677+0000","last_change":"2024-04-22T06:42:24.187668+0000","last_active":"2024-04-22T06:44:25.045677+0000","last_peered":"2024-04-22T06:44:25.045677+0000","last_clean":"2024-04-22T06:44:25.045677+0000","last_became_active":"2024-04-22T06:42:24.187354+0000","last_became_peered":"2024-04-22T06:42:24.187354+0000","last_unstale":"2024-04-22T06:44:25.045677+0000","last_undegraded":"2024-04-22T06:44:25.045677+0000","last_fullsized":"2024-04-22T06:44:25.045677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:35:28.557306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050734+0000","last_change":"2024-04-22T06:42:23.671934+0000","last_active":"2024-04-22T06:44:25.050734+0000","last_peered":"2024-04-22T06:44:25.050734+0000","last_clean":"2024-04-22T06:44:25.050734+0000","last_became_active":"2024-04-22T06:42:23.671589+0000","last_became_peered":"2024-04-22T06:42:23.671589+0000","last_unstale":"2024-04-22T06:44:25.050734+0000","last_undegraded":"2024-04-22T06:44:25.050734+0000","last_fullsized":"2024-04-22T06:44:25.050734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:00.910027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195155+0000","last_change":"2024-04-22T06:42:24.202561+0000","last_active":"2024-04-22T06:44:25.195155+0000","last_peered":"2024-04-22T06:44:25.195155+0000","last_clean":"2024-04-22T06:44:25.195155+0000","last_became_active":"2024-04-22T06:42:24.202164+0000","last_became_peered":"2024-04-22T06:42:24.202164+0000","last_unstale":"2024-04-22T06:44:25.195155+0000","last_undegraded":"2024-04-22T06:44:25.195155+0000","last_fullsized":"2024-04-22T06:44:25.195155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:22:40.960729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'7","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:29.894859+0000","last_change":"2024-04-22T06:42:23.671536+0000","last_active":"2024-04-22T06:44:29.894859+0000","last_peered":"2024-04-22T06:44:29.894859+0000","last_clean":"2024-04-22T06:44:29.894859+0000","last_became_active":"2024-04-22T06:42:23.670448+0000","last_became_peered":"2024-04-22T06:42:23.670448+0000","last_unstale":"2024-04-22T06:44:29.894859+0000","last_undegraded":"2024-04-22T06:44:29.894859+0000","last_fullsized":"2024-04-22T06:44:29.894859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:43.596663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":7,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194722+0000","last_change":"2024-04-22T06:42:24.197258+0000","last_active":"2024-04-22T06:44:25.194722+0000","last_peered":"2024-04-22T06:44:25.194722+0000","last_clean":"2024-04-22T06:44:25.194722+0000","last_became_active":"2024-04-22T06:42:24.196802+0000","last_became_peered":"2024-04-22T06:42:24.196802+0000","last_unstale":"2024-04-22T06:44:25.194722+0000","last_undegraded":"2024-04-22T06:44:25.194722+0000","last_fullsized":"2024-04-22T06:44:25.194722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:35:36.086477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043891+0000","last_change":"2024-04-22T06:42:23.675207+0000","last_active":"2024-04-22T06:44:25.043891+0000","last_peered":"2024-04-22T06:44:25.043891+0000","last_clean":"2024-04-22T06:44:25.043891+0000","last_became_active":"2024-04-22T06:42:23.674902+0000","last_became_peered":"2024-04-22T06:42:23.674902+0000","last_unstale":"2024-04-22T06:44:25.043891+0000","last_undegraded":"2024-04-22T06:44:25.043891+0000","last_fullsized":"2024-04-22T06:44:25.043891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:41:04.240131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050670+0000","last_change":"2024-04-22T06:42:24.194438+0000","last_active":"2024-04-22T06:44:25.050670+0000","last_peered":"2024-04-22T06:44:25.050670+0000","last_clean":"2024-04-22T06:44:25.050670+0000","last_became_active":"2024-04-22T06:42:24.193419+0000","last_became_peered":"2024-04-22T06:42:24.193419+0000","last_unstale":"2024-04-22T06:44:25.050670+0000","last_undegraded":"2024-04-22T06:44:25.050670+0000","last_fullsized":"2024-04-22T06:44:25.050670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:22:10.719156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044274+0000","last_change":"2024-04-22T06:42:23.669419+0000","last_active":"2024-04-22T06:44:25.044274+0000","last_peered":"2024-04-22T06:44:25.044274+0000","last_clean":"2024-04-22T06:44:25.044274+0000","last_became_active":"2024-04-22T06:42:23.669073+0000","last_became_peered":"2024-04-22T06:42:23.669073+0000","last_unstale":"2024-04-22T06:44:25.044274+0000","last_undegraded":"2024-04-22T06:44:25.044274+0000","last_fullsized":"2024-04-22T06:44:25.044274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:16:05.308712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953070+0000","last_change":"2024-04-22T06:42:24.203679+0000","last_active":"2024-04-22T06:44:24.953070+0000","last_peered":"2024-04-22T06:44:24.953070+0000","last_clean":"2024-04-22T06:44:24.953070+0000","last_became_active":"2024-04-22T06:42:24.203367+0000","last_became_peered":"2024-04-22T06:42:24.203367+0000","last_unstale":"2024-04-22T06:44:24.953070+0000","last_undegraded":"2024-04-22T06:44:24.953070+0000","last_fullsized":"2024-04-22T06:44:24.953070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:32:06.159306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.948485+0000","last_change":"2024-04-22T06:42:23.669623+0000","last_active":"2024-04-22T06:44:24.948485+0000","last_peered":"2024-04-22T06:44:24.948485+0000","last_clean":"2024-04-22T06:44:24.948485+0000","last_became_active":"2024-04-22T06:42:23.669303+0000","last_became_peered":"2024-04-22T06:42:23.669303+0000","last_unstale":"2024-04-22T06:44:24.948485+0000","last_undegraded":"2024-04-22T06:44:24.948485+0000","last_fullsized":"2024-04-22T06:44:24.948485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:29.676819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952238+0000","last_change":"2024-04-22T06:42:24.183383+0000","last_active":"2024-04-22T06:44:24.952238+0000","last_peered":"2024-04-22T06:44:24.952238+0000","last_clean":"2024-04-22T06:44:24.952238+0000","last_became_active":"2024-04-22T06:42:24.182623+0000","last_became_peered":"2024-04-22T06:42:24.182623+0000","last_unstale":"2024-04-22T06:44:24.952238+0000","last_undegraded":"2024-04-22T06:44:24.952238+0000","last_fullsized":"2024-04-22T06:44:24.952238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:42:06.088529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195062+0000","last_change":"2024-04-22T06:42:23.679241+0000","last_active":"2024-04-22T06:44:25.195062+0000","last_peered":"2024-04-22T06:44:25.195062+0000","last_clean":"2024-04-22T06:44:25.195062+0000","last_became_active":"2024-04-22T06:42:23.678785+0000","last_became_peered":"2024-04-22T06:42:23.678785+0000","last_unstale":"2024-04-22T06:44:25.195062+0000","last_undegraded":"2024-04-22T06:44:25.195062+0000","last_fullsized":"2024-04-22T06:44:25.195062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:19.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952360+0000","last_change":"2024-04-22T06:42:24.202397+0000","last_active":"2024-04-22T06:44:24.952360+0000","last_peered":"2024-04-22T06:44:24.952360+0000","last_clean":"2024-04-22T06:44:24.952360+0000","last_became_active":"2024-04-22T06:42:24.201930+0000","last_became_peered":"2024-04-22T06:42:24.201930+0000","last_unstale":"2024-04-22T06:44:24.952360+0000","last_undegraded":"2024-04-22T06:44:24.952360+0000","last_fullsized":"2024-04-22T06:44:24.952360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:45.354152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050823+0000","last_change":"2024-04-22T06:43:17.897820+0000","last_active":"2024-04-22T06:44:25.050823+0000","last_peered":"2024-04-22T06:44:25.050823+0000","last_clean":"2024-04-22T06:44:25.050823+0000","last_became_active":"2024-04-22T06:43:17.897424+0000","last_became_peered":"2024-04-22T06:43:17.897424+0000","last_unstale":"2024-04-22T06:44:25.050823+0000","last_undegraded":"2024-04-22T06:44:25.050823+0000","last_fullsized":"2024-04-22T06:44:25.050823+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:56:55.068970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'6","reported_seq":151,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.895537+0000","last_change":"2024-04-22T06:42:23.679166+0000","last_active":"2024-04-22T06:44:30.895537+0000","last_peered":"2024-04-22T06:44:30.895537+0000","last_clean":"2024-04-22T06:44:30.895537+0000","last_became_active":"2024-04-22T06:42:23.678844+0000","last_became_peered":"2024-04-22T06:42:23.678844+0000","last_unstale":"2024-04-22T06:44:30.895537+0000","last_undegraded":"2024-04-22T06:44:30.895537+0000","last_fullsized":"2024-04-22T06:44:30.895537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:07.282558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045472+0000","last_change":"2024-04-22T06:42:24.182367+0000","last_active":"2024-04-22T06:44:25.045472+0000","last_peered":"2024-04-22T06:44:25.045472+0000","last_clean":"2024-04-22T06:44:25.045472+0000","last_became_active":"2024-04-22T06:42:24.182053+0000","last_became_peered":"2024-04-22T06:42:24.182053+0000","last_unstale":"2024-04-22T06:44:25.045472+0000","last_undegraded":"2024-04-22T06:44:25.045472+0000","last_fullsized":"2024-04-22T06:44:25.045472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:24:11.654861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193427+0000","last_change":"2024-04-22T06:42:23.671253+0000","last_active":"2024-04-22T06:44:25.193427+0000","last_peered":"2024-04-22T06:44:25.193427+0000","last_clean":"2024-04-22T06:44:25.193427+0000","last_became_active":"2024-04-22T06:42:23.670900+0000","last_became_peered":"2024-04-22T06:42:23.670900+0000","last_unstale":"2024-04-22T06:44:25.193427+0000","last_undegraded":"2024-04-22T06:44:25.193427+0000","last_fullsized":"2024-04-22T06:44:25.193427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:16.808601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:27.055913+0000","last_change":"2024-04-22T06:41:07.169017+0000","last_active":"2024-04-22T06:44:27.055913+0000","last_peered":"2024-04-22T06:44:27.055913+0000","last_clean":"2024-04-22T06:44:27.055913+0000","last_became_active":"2024-04-22T06:41:07.168579+0000","last_became_peered":"2024-04-22T06:41:07.168579+0000","last_unstale":"2024-04-22T06:44:27.055913+0000","last_undegraded":"2024-04-22T06:44:27.055913+0000","last_fullsized":"2024-04-22T06:44:27.055913+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:42.746068+0000","last_clean_scrub_stamp":"2024-04-22T06:37:42.746068+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:16:15.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045340+0000","last_change":"2024-04-22T06:42:24.196653+0000","last_active":"2024-04-22T06:44:25.045340+0000","last_peered":"2024-04-22T06:44:25.045340+0000","last_clean":"2024-04-22T06:44:25.045340+0000","last_became_active":"2024-04-22T06:42:24.196345+0000","last_became_peered":"2024-04-22T06:42:24.196345+0000","last_unstale":"2024-04-22T06:44:25.045340+0000","last_undegraded":"2024-04-22T06:44:25.045340+0000","last_fullsized":"2024-04-22T06:44:25.045340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:30:39.943664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194994+0000","last_change":"2024-04-22T06:42:23.675276+0000","last_active":"2024-04-22T06:44:25.194994+0000","last_peered":"2024-04-22T06:44:25.194994+0000","last_clean":"2024-04-22T06:44:25.194994+0000","last_became_active":"2024-04-22T06:42:23.674929+0000","last_became_peered":"2024-04-22T06:42:23.674929+0000","last_unstale":"2024-04-22T06:44:25.194994+0000","last_undegraded":"2024-04-22T06:44:25.194994+0000","last_fullsized":"2024-04-22T06:44:25.194994+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:12.139681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044294+0000","last_change":"2024-04-22T06:42:23.676058+0000","last_active":"2024-04-22T06:44:25.044294+0000","last_peered":"2024-04-22T06:44:25.044294+0000","last_clean":"2024-04-22T06:44:25.044294+0000","last_became_active":"2024-04-22T06:42:23.675747+0000","last_became_peered":"2024-04-22T06:42:23.675747+0000","last_unstale":"2024-04-22T06:44:25.044294+0000","last_undegraded":"2024-04-22T06:44:25.044294+0000","last_fullsized":"2024-04-22T06:44:25.044294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:36:24.534819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046002+0000","last_change":"2024-04-22T06:42:24.184728+0000","last_active":"2024-04-22T06:44:25.046002+0000","last_peered":"2024-04-22T06:44:25.046002+0000","last_clean":"2024-04-22T06:44:25.046002+0000","last_became_active":"2024-04-22T06:42:24.184426+0000","last_became_peered":"2024-04-22T06:42:24.184426+0000","last_unstale":"2024-04-22T06:44:25.046002+0000","last_undegraded":"2024-04-22T06:44:25.046002+0000","last_fullsized":"2024-04-22T06:44:25.046002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:12.575449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'46","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:26.693435+0000","last_change":"2024-04-22T06:42:23.671724+0000","last_active":"2024-04-22T06:44:26.693435+0000","last_peered":"2024-04-22T06:44:26.693435+0000","last_clean":"2024-04-22T06:44:26.693435+0000","last_became_active":"2024-04-22T06:42:23.671371+0000","last_became_peered":"2024-04-22T06:42:23.671371+0000","last_unstale":"2024-04-22T06:44:26.693435+0000","last_undegraded":"2024-04-22T06:44:26.693435+0000","last_fullsized":"2024-04-22T06:44:26.693435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:54:52.153449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044531+0000","last_change":"2024-04-22T06:42:24.191574+0000","last_active":"2024-04-22T06:44:25.044531+0000","last_peered":"2024-04-22T06:44:25.044531+0000","last_clean":"2024-04-22T06:44:25.044531+0000","last_became_active":"2024-04-22T06:42:24.191058+0000","last_became_peered":"2024-04-22T06:42:24.191058+0000","last_unstale":"2024-04-22T06:44:25.044531+0000","last_undegraded":"2024-04-22T06:44:25.044531+0000","last_fullsized":"2024-04-22T06:44:25.044531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:06:46.153654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.193293+0000","last_change":"2024-04-22T06:42:23.669731+0000","last_active":"2024-04-22T06:44:25.193293+0000","last_peered":"2024-04-22T06:44:25.193293+0000","last_clean":"2024-04-22T06:44:25.193293+0000","last_became_active":"2024-04-22T06:42:23.669292+0000","last_became_peered":"2024-04-22T06:42:23.669292+0000","last_unstale":"2024-04-22T06:44:25.193293+0000","last_undegraded":"2024-04-22T06:44:25.193293+0000","last_fullsized":"2024-04-22T06:44:25.193293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:37:51.450801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194372+0000","last_change":"2024-04-22T06:42:24.197183+0000","last_active":"2024-04-22T06:44:25.194372+0000","last_peered":"2024-04-22T06:44:25.194372+0000","last_clean":"2024-04-22T06:44:25.194372+0000","last_became_active":"2024-04-22T06:42:24.196859+0000","last_became_peered":"2024-04-22T06:42:24.196859+0000","last_unstale":"2024-04-22T06:44:25.194372+0000","last_undegraded":"2024-04-22T06:44:25.194372+0000","last_fullsized":"2024-04-22T06:44:25.194372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:25.529658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023295+0000","last_change":"2024-04-22T06:42:23.666499+0000","last_active":"2024-04-22T06:44:25.023295+0000","last_peered":"2024-04-22T06:44:25.023295+0000","last_clean":"2024-04-22T06:44:25.023295+0000","last_became_active":"2024-04-22T06:42:23.666127+0000","last_became_peered":"2024-04-22T06:42:23.666127+0000","last_unstale":"2024-04-22T06:44:25.023295+0000","last_undegraded":"2024-04-22T06:44:25.023295+0000","last_fullsized":"2024-04-22T06:44:25.023295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:37.774472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.043912+0000","last_change":"2024-04-22T06:42:24.200540+0000","last_active":"2024-04-22T06:44:25.043912+0000","last_peered":"2024-04-22T06:44:25.043912+0000","last_clean":"2024-04-22T06:44:25.043912+0000","last_became_active":"2024-04-22T06:42:24.199598+0000","last_became_peered":"2024-04-22T06:42:24.199598+0000","last_unstale":"2024-04-22T06:44:25.043912+0000","last_undegraded":"2024-04-22T06:44:25.043912+0000","last_fullsized":"2024-04-22T06:44:25.043912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:03:48.183341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044235+0000","last_change":"2024-04-22T06:42:23.677902+0000","last_active":"2024-04-22T06:44:25.044235+0000","last_peered":"2024-04-22T06:44:25.044235+0000","last_clean":"2024-04-22T06:44:25.044235+0000","last_became_active":"2024-04-22T06:42:23.677577+0000","last_became_peered":"2024-04-22T06:42:23.677577+0000","last_unstale":"2024-04-22T06:44:25.044235+0000","last_undegraded":"2024-04-22T06:44:25.044235+0000","last_fullsized":"2024-04-22T06:44:25.044235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:44.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952874+0000","last_change":"2024-04-22T06:42:24.190079+0000","last_active":"2024-04-22T06:44:24.952874+0000","last_peered":"2024-04-22T06:44:24.952874+0000","last_clean":"2024-04-22T06:44:24.952874+0000","last_became_active":"2024-04-22T06:42:24.189594+0000","last_became_peered":"2024-04-22T06:42:24.189594+0000","last_unstale":"2024-04-22T06:44:24.952874+0000","last_undegraded":"2024-04-22T06:44:24.952874+0000","last_fullsized":"2024-04-22T06:44:24.952874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:20:01.741241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'2","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953015+0000","last_change":"2024-04-22T06:43:17.817300+0000","last_active":"2024-04-22T06:44:24.953015+0000","last_peered":"2024-04-22T06:44:24.953015+0000","last_clean":"2024-04-22T06:44:24.953015+0000","last_became_active":"2024-04-22T06:43:17.813266+0000","last_became_peered":"2024-04-22T06:43:17.813266+0000","last_unstale":"2024-04-22T06:44:24.953015+0000","last_undegraded":"2024-04-22T06:44:24.953015+0000","last_fullsized":"2024-04-22T06:44:24.953015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:38:40.405800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044498+0000","last_change":"2024-04-22T06:42:24.188233+0000","last_active":"2024-04-22T06:44:25.044498+0000","last_peered":"2024-04-22T06:44:25.044498+0000","last_clean":"2024-04-22T06:44:25.044498+0000","last_became_active":"2024-04-22T06:42:24.187057+0000","last_became_peered":"2024-04-22T06:42:24.187057+0000","last_unstale":"2024-04-22T06:44:25.044498+0000","last_undegraded":"2024-04-22T06:44:25.044498+0000","last_fullsized":"2024-04-22T06:44:25.044498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:27:30.995503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.952952+0000","last_change":"2024-04-22T06:43:17.820815+0000","last_active":"2024-04-22T06:44:24.952952+0000","last_peered":"2024-04-22T06:44:24.952952+0000","last_clean":"2024-04-22T06:44:24.952952+0000","last_became_active":"2024-04-22T06:43:17.813797+0000","last_became_peered":"2024-04-22T06:43:17.813797+0000","last_unstale":"2024-04-22T06:44:24.952952+0000","last_undegraded":"2024-04-22T06:44:24.952952+0000","last_fullsized":"2024-04-22T06:44:24.952952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:27.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195201+0000","last_change":"2024-04-22T06:42:24.192104+0000","last_active":"2024-04-22T06:44:25.195201+0000","last_peered":"2024-04-22T06:44:25.195201+0000","last_clean":"2024-04-22T06:44:25.195201+0000","last_became_active":"2024-04-22T06:42:24.191715+0000","last_became_peered":"2024-04-22T06:42:24.191715+0000","last_unstale":"2024-04-22T06:44:25.195201+0000","last_undegraded":"2024-04-22T06:44:25.195201+0000","last_fullsized":"2024-04-22T06:44:25.195201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:36:16.269334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195645+0000","last_change":"2024-04-22T06:42:23.679854+0000","last_active":"2024-04-22T06:44:25.195645+0000","last_peered":"2024-04-22T06:44:25.195645+0000","last_clean":"2024-04-22T06:44:25.195645+0000","last_became_active":"2024-04-22T06:42:23.679412+0000","last_became_peered":"2024-04-22T06:42:23.679412+0000","last_unstale":"2024-04-22T06:44:25.195645+0000","last_undegraded":"2024-04-22T06:44:25.195645+0000","last_fullsized":"2024-04-22T06:44:25.195645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:28.801385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195419+0000","last_change":"2024-04-22T06:42:24.192912+0000","last_active":"2024-04-22T06:44:25.195419+0000","last_peered":"2024-04-22T06:44:25.195419+0000","last_clean":"2024-04-22T06:44:25.195419+0000","last_became_active":"2024-04-22T06:42:24.191792+0000","last_became_peered":"2024-04-22T06:42:24.191792+0000","last_unstale":"2024-04-22T06:44:25.195419+0000","last_undegraded":"2024-04-22T06:44:25.195419+0000","last_fullsized":"2024-04-22T06:44:25.195419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:16.761886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.150698+0000","last_change":"2024-04-22T06:42:23.668219+0000","last_active":"2024-04-22T06:44:25.150698+0000","last_peered":"2024-04-22T06:44:25.150698+0000","last_clean":"2024-04-22T06:44:25.150698+0000","last_became_active":"2024-04-22T06:42:23.667889+0000","last_became_peered":"2024-04-22T06:42:23.667889+0000","last_unstale":"2024-04-22T06:44:25.150698+0000","last_undegraded":"2024-04-22T06:44:25.150698+0000","last_fullsized":"2024-04-22T06:44:25.150698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:27:31.162636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195126+0000","last_change":"2024-04-22T06:42:24.196957+0000","last_active":"2024-04-22T06:44:25.195126+0000","last_peered":"2024-04-22T06:44:25.195126+0000","last_clean":"2024-04-22T06:44:25.195126+0000","last_became_active":"2024-04-22T06:42:24.196597+0000","last_became_peered":"2024-04-22T06:42:24.196597+0000","last_unstale":"2024-04-22T06:44:25.195126+0000","last_undegraded":"2024-04-22T06:44:25.195126+0000","last_fullsized":"2024-04-22T06:44:25.195126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:41:56.878866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.046237+0000","last_change":"2024-04-22T06:42:23.679232+0000","last_active":"2024-04-22T06:44:25.046237+0000","last_peered":"2024-04-22T06:44:25.046237+0000","last_clean":"2024-04-22T06:44:25.046237+0000","last_became_active":"2024-04-22T06:42:23.678882+0000","last_became_peered":"2024-04-22T06:42:23.678882+0000","last_unstale":"2024-04-22T06:44:25.046237+0000","last_undegraded":"2024-04-22T06:44:25.046237+0000","last_fullsized":"2024-04-22T06:44:25.046237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:22.643709+0000","last_clean_scrub_stamp":"2024-04-22T06:42:22.643709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:57:40.204888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195405+0000","last_change":"2024-04-22T06:42:24.190431+0000","last_active":"2024-04-22T06:44:25.195405+0000","last_peered":"2024-04-22T06:44:25.195405+0000","last_clean":"2024-04-22T06:44:25.195405+0000","last_became_active":"2024-04-22T06:42:24.190064+0000","last_became_peered":"2024-04-22T06:42:24.190064+0000","last_unstale":"2024-04-22T06:44:25.195405+0000","last_undegraded":"2024-04-22T06:44:25.195405+0000","last_fullsized":"2024-04-22T06:44:25.195405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.713456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953214+0000","last_change":"2024-04-22T06:43:17.814239+0000","last_active":"2024-04-22T06:44:24.953214+0000","last_peered":"2024-04-22T06:44:24.953214+0000","last_clean":"2024-04-22T06:44:24.953214+0000","last_became_active":"2024-04-22T06:43:17.813785+0000","last_became_peered":"2024-04-22T06:43:17.813785+0000","last_unstale":"2024-04-22T06:44:24.953214+0000","last_undegraded":"2024-04-22T06:44:24.953214+0000","last_fullsized":"2024-04-22T06:44:24.953214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:16.521621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195606+0000","last_change":"2024-04-22T06:43:17.814744+0000","last_active":"2024-04-22T06:44:25.195606+0000","last_peered":"2024-04-22T06:44:25.195606+0000","last_clean":"2024-04-22T06:44:25.195606+0000","last_became_active":"2024-04-22T06:43:17.814391+0000","last_became_peered":"2024-04-22T06:43:17.814391+0000","last_unstale":"2024-04-22T06:44:25.195606+0000","last_undegraded":"2024-04-22T06:44:25.195606+0000","last_fullsized":"2024-04-22T06:44:25.195606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:25:27.576649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195452+0000","last_change":"2024-04-22T06:42:24.194088+0000","last_active":"2024-04-22T06:44:25.195452+0000","last_peered":"2024-04-22T06:44:25.195452+0000","last_clean":"2024-04-22T06:44:25.195452+0000","last_became_active":"2024-04-22T06:42:24.192651+0000","last_became_peered":"2024-04-22T06:42:24.192651+0000","last_unstale":"2024-04-22T06:44:25.195452+0000","last_undegraded":"2024-04-22T06:44:25.195452+0000","last_fullsized":"2024-04-22T06:44:25.195452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:41:48.155231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.194286+0000","last_change":"2024-04-22T06:42:24.197502+0000","last_active":"2024-04-22T06:44:25.194286+0000","last_peered":"2024-04-22T06:44:25.194286+0000","last_clean":"2024-04-22T06:44:25.194286+0000","last_became_active":"2024-04-22T06:42:24.197190+0000","last_became_peered":"2024-04-22T06:42:24.197190+0000","last_unstale":"2024-04-22T06:44:25.194286+0000","last_undegraded":"2024-04-22T06:44:25.194286+0000","last_fullsized":"2024-04-22T06:44:25.194286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:18.347378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045832+0000","last_change":"2024-04-22T06:42:24.196837+0000","last_active":"2024-04-22T06:44:25.045832+0000","last_peered":"2024-04-22T06:44:25.045832+0000","last_clean":"2024-04-22T06:44:25.045832+0000","last_became_active":"2024-04-22T06:42:24.196505+0000","last_became_peered":"2024-04-22T06:42:24.196505+0000","last_unstale":"2024-04-22T06:44:25.045832+0000","last_undegraded":"2024-04-22T06:44:25.045832+0000","last_fullsized":"2024-04-22T06:44:25.045832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:53.672010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953143+0000","last_change":"2024-04-22T06:42:24.199747+0000","last_active":"2024-04-22T06:44:24.953143+0000","last_peered":"2024-04-22T06:44:24.953143+0000","last_clean":"2024-04-22T06:44:24.953143+0000","last_became_active":"2024-04-22T06:42:24.199371+0000","last_became_peered":"2024-04-22T06:42:24.199371+0000","last_unstale":"2024-04-22T06:44:24.953143+0000","last_undegraded":"2024-04-22T06:44:24.953143+0000","last_fullsized":"2024-04-22T06:44:24.953143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:43.441630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.195715+0000","last_change":"2024-04-22T06:42:24.193130+0000","last_active":"2024-04-22T06:44:25.195715+0000","last_peered":"2024-04-22T06:44:25.195715+0000","last_clean":"2024-04-22T06:44:25.195715+0000","last_became_active":"2024-04-22T06:42:24.192297+0000","last_became_peered":"2024-04-22T06:42:24.192297+0000","last_unstale":"2024-04-22T06:44:25.195715+0000","last_undegraded":"2024-04-22T06:44:25.195715+0000","last_fullsized":"2024-04-22T06:44:25.195715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:22.422649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044467+0000","last_change":"2024-04-22T06:42:24.196568+0000","last_active":"2024-04-22T06:44:25.044467+0000","last_peered":"2024-04-22T06:44:25.044467+0000","last_clean":"2024-04-22T06:44:25.044467+0000","last_became_active":"2024-04-22T06:42:24.196260+0000","last_became_peered":"2024-04-22T06:42:24.196260+0000","last_unstale":"2024-04-22T06:44:25.044467+0000","last_undegraded":"2024-04-22T06:44:25.044467+0000","last_fullsized":"2024-04-22T06:44:25.044467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:21.391251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.050646+0000","last_change":"2024-04-22T06:43:17.812891+0000","last_active":"2024-04-22T06:44:25.050646+0000","last_peered":"2024-04-22T06:44:25.050646+0000","last_clean":"2024-04-22T06:44:25.050646+0000","last_became_active":"2024-04-22T06:43:17.812517+0000","last_became_peered":"2024-04-22T06:43:17.812517+0000","last_unstale":"2024-04-22T06:44:25.050646+0000","last_undegraded":"2024-04-22T06:44:25.050646+0000","last_fullsized":"2024-04-22T06:44:25.050646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:40.099311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.044351+0000","last_change":"2024-04-22T06:42:24.192024+0000","last_active":"2024-04-22T06:44:25.044351+0000","last_peered":"2024-04-22T06:44:25.044351+0000","last_clean":"2024-04-22T06:44:25.044351+0000","last_became_active":"2024-04-22T06:42:24.191517+0000","last_became_peered":"2024-04-22T06:42:24.191517+0000","last_unstale":"2024-04-22T06:44:25.044351+0000","last_undegraded":"2024-04-22T06:44:25.044351+0000","last_fullsized":"2024-04-22T06:44:25.044351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.874471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.023389+0000","last_change":"2024-04-22T06:42:24.190696+0000","last_active":"2024-04-22T06:44:25.023389+0000","last_peered":"2024-04-22T06:44:25.023389+0000","last_clean":"2024-04-22T06:44:25.023389+0000","last_became_active":"2024-04-22T06:42:24.190364+0000","last_became_peered":"2024-04-22T06:42:24.190364+0000","last_unstale":"2024-04-22T06:44:25.023389+0000","last_undegraded":"2024-04-22T06:44:25.023389+0000","last_fullsized":"2024-04-22T06:44:25.023389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:32:39.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:25.045996+0000","last_change":"2024-04-22T06:42:24.187614+0000","last_active":"2024-04-22T06:44:25.045996+0000","last_peered":"2024-04-22T06:44:25.045996+0000","last_clean":"2024-04-22T06:44:25.045996+0000","last_became_active":"2024-04-22T06:42:24.187313+0000","last_became_peered":"2024-04-22T06:42:24.187313+0000","last_unstale":"2024-04-22T06:44:25.045996+0000","last_undegraded":"2024-04-22T06:44:25.045996+0000","last_fullsized":"2024-04-22T06:44:25.045996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:55.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:24.953024+0000","last_change":"2024-04-22T06:42:24.203030+0000","last_active":"2024-04-22T06:44:24.953024+0000","last_peered":"2024-04-22T06:44:24.953024+0000","last_clean":"2024-04-22T06:44:24.953024+0000","last_became_active":"2024-04-22T06:42:24.202680+0000","last_became_peered":"2024-04-22T06:42:24.202680+0000","last_unstale":"2024-04-22T06:44:24.953024+0000","last_undegraded":"2024-04-22T06:44:24.953024+0000","last_fullsized":"2024-04-22T06:44:24.953024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:23.645235+0000","last_clean_scrub_stamp":"2024-04-22T06:42:23.645235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:26:05.284310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107121,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599909,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":866,"num_read_kb":544,"num_write":296,"num_write_kb":1445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1799727,"data_compressed":81528,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":465124,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257983,"data_compressed":46638,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858576,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858496,"data_compressed":22047,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1059838,"data_compressed":38608,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":464749,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":533006,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":466020,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":536061,"data_compressed":4775,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989876,"data_compressed":28203,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":466500,"data_compressed":2901,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1189808,"data_compressed":44764,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35707,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68464,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71519,"data_compressed":1874,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134781,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202956,"data_compressed":8030,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:36.948 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 -- 172.21.15.79:0/3103033142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048101a60 msgr2=0x7f6048105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 --2- 172.21.15.79:0/3103033142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048101a60 0x7f6048105aa0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6030009900 tx=0x7f603002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 -- 172.21.15.79:0/3103033142 shutdown_connections 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 --2- 172.21.15.79:0/3103033142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6048106180 0x7f604810dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 --2- 172.21.15.79:0/3103033142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048101a60 0x7f6048105aa0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 --2- 172.21.15.79:0/3103033142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 0x7f6048101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 -- 172.21.15.79:0/3103033142 >> 172.21.15.79:0/3103033142 conn(0x7f60480771d0 msgr2=0x7f60480775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 -- 172.21.15.79:0/3103033142 shutdown_connections 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.084+0000 7f604d411640 1 -- 172.21.15.79:0/3103033142 wait complete. 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 Processor -- start 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 -- start start 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 0x7f6048198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 0x7f6048198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f604819cee0 0x7f604819d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6048112200 con 0x7f6048106180 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6048112080 con 0x7f60481010b0 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f604d411640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6048112380 con 0x7f604819cee0 2024-04-22T06:44:37.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 0x7f6048198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 0x7f6048198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53760/0 (socket says 172.21.15.79:53760) 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 -- 172.21.15.79:0/4194231288 learned_addr learned my addr 172.21.15.79:0/4194231288 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f60477fe640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f604819cee0 0x7f604819d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f6046ffd640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 0x7f6048198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 -- 172.21.15.79:0/4194231288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f604819cee0 msgr2=0x7f604819d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f604819cee0 0x7f604819d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 -- 172.21.15.79:0/4194231288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 msgr2=0x7f6048198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 0x7f6048198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f603ffff640 1 -- 172.21.15.79:0/4194231288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604819d9d0 con 0x7f6048106180 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f60477fe640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f604819cee0 0x7f604819d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.085+0000 7f6046ffd640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 0x7f6048198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.086+0000 7f603ffff640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 0x7f6048198b60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6030036040 tx=0x7f603002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.086+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030043020 con 0x7f6048106180 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.086+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f603002cbe0 con 0x7f6048106180 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.086+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604819dc60 con 0x7f6048106180 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.086+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030003bf0 con 0x7f6048106180 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.086+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60481a5480 con 0x7f6048106180 2024-04-22T06:44:37.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.087+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6048069230 con 0x7f6048106180 2024-04-22T06:44:37.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.088+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f6030046020 con 0x7f6048106180 2024-04-22T06:44:37.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.089+0000 7f6044ff9640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f60340a3e40 0x7f60340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.089+0000 7f6046ffd640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f60340a3e40 0x7f60340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.089+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f60300f32b0 con 0x7f6048106180 2024-04-22T06:44:37.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.090+0000 7f6046ffd640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f60340a3e40 0x7f60340a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f603800baa0 tx=0x7f6038005d90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.093+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f603004d020 con 0x7f6048106180 2024-04-22T06:44:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:37 smithi079 ceph-mon[27946]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:37 smithi079 ceph-mon[27946]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.305+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6048199970 con 0x7f6048106180 2024-04-22T06:44:37.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.306+0000 7f6044ff9640 1 -- 172.21.15.79:0/4194231288 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7f6048199970 con 0x7f6048106180 2024-04-22T06:44:37.304 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:37.304 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:44:37.304 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:44:37.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f60340a3e40 msgr2=0x7f60340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f60340a3e40 0x7f60340a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f603800baa0 tx=0x7f6038005d90 comp rx=0 tx=0).stop 2024-04-22T06:44:37.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 msgr2=0x7f6048198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 0x7f6048198b60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6030036040 tx=0x7f603002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 shutdown_connections 2024-04-22T06:44:37.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f604819cee0 0x7f604819d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6048106180 0x7f6048198b60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f60340a3e40 0x7f60340a6300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 --2- 172.21.15.79:0/4194231288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60481010b0 0x7f6048198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.307+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 >> 172.21.15.79:0/4194231288 conn(0x7f60480771d0 msgr2=0x7f60480fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.308+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 shutdown_connections 2024-04-22T06:44:37.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.308+0000 7f604d411640 1 -- 172.21.15.79:0/4194231288 wait complete. 2024-04-22T06:44:37.317 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 -- 172.21.15.79:0/3393826962 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678101ad0 msgr2=0x7fb67810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/3393826962 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678101ad0 0x7fb67810dcb0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb670009ca0 tx=0x7fb67002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 -- 172.21.15.79:0/3393826962 shutdown_connections 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/3393826962 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678101ad0 0x7fb67810dcb0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/3393826962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101130 0x7fb678101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/3393826962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678106f10 0x7fb6781072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 -- 172.21.15.79:0/3393826962 >> 172.21.15.79:0/3393826962 conn(0x7fb678077e30 msgr2=0x7fb678078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 -- 172.21.15.79:0/3393826962 shutdown_connections 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.417+0000 7fb67e5ef640 1 -- 172.21.15.79:0/3393826962 wait complete. 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 Processor -- start 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 -- start start 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678101130 0x7fb678198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101ad0 0x7fb678198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 0x7fb67819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb678112260 con 0x7fb678106f10 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781120e0 con 0x7fb678101130 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67e5ef640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781123e0 con 0x7fb678101ad0 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 0x7fb67819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 0x7fb67819cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53774/0 (socket says 172.21.15.79:53774) 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 -- 172.21.15.79:0/1111808725 learned_addr learned my addr 172.21.15.79:0/1111808725 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:37.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb677fff640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678101130 0x7fb678198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb6777fe640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101ad0 0x7fb678198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 -- 172.21.15.79:0/1111808725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101ad0 msgr2=0x7fb678198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101ad0 0x7fb678198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 -- 172.21.15.79:0/1111808725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678101130 msgr2=0x7fb678198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678101130 0x7fb678198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb67cb8c640 1 -- 172.21.15.79:0/1111808725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67819d6f0 con 0x7fb678106f10 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb6777fe640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101ad0 0x7fb678198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.418+0000 7fb677fff640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678101130 0x7fb678198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.419+0000 7fb67cb8c640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 0x7fb67819cfc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb6700098d0 tx=0x7fb670035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.419+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670039040 con 0x7fb678106f10 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.419+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67819d980 con 0x7fb678106f10 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.419+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb670044070 con 0x7fb678106f10 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.419+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6700042a0 con 0x7fb678106f10 2024-04-22T06:44:37.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.419+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6781a5270 con 0x7fb678106f10 2024-04-22T06:44:37.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.420+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb678069230 con 0x7fb678106f10 2024-04-22T06:44:37.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.421+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb670004440 con 0x7fb678106f10 2024-04-22T06:44:37.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.421+0000 7fb6757fa640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6600a3e40 0x7fb6600a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.421+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fb6700f3a10 con 0x7fb678106f10 2024-04-22T06:44:37.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.422+0000 7fb677fff640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6600a3e40 0x7fb6600a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.422+0000 7fb677fff640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6600a3e40 0x7fb6600a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb66800abe0 tx=0x7fb66800a5c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.425+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb670051e20 con 0x7fb678106f10 2024-04-22T06:44:37.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:37 smithi193 ceph-mon[31815]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:37 smithi193 ceph-mon[31815]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:37 smithi190 ceph-mon[31421]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:37 smithi190 ceph-mon[31421]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.604+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb678109370 con 0x7fb678106f10 2024-04-22T06:44:37.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.606+0000 7fb6757fa640 1 -- 172.21.15.79:0/1111808725 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7fb6700bffa0 con 0x7fb678106f10 2024-04-22T06:44:37.604 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:37.605 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:37.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6600a3e40 msgr2=0x7fb6600a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6600a3e40 0x7fb6600a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb66800abe0 tx=0x7fb66800a5c0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 msgr2=0x7fb67819cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 0x7fb67819cfc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb6700098d0 tx=0x7fb670035d60 comp rx=0 tx=0).stop 2024-04-22T06:44:37.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 shutdown_connections 2024-04-22T06:44:37.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb678106f10 0x7fb67819cfc0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb678101ad0 0x7fb678198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb6600a3e40 0x7fb6600a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 --2- 172.21.15.79:0/1111808725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb678101130 0x7fb678198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 >> 172.21.15.79:0/1111808725 conn(0x7fb678077e30 msgr2=0x7fb6780ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 shutdown_connections 2024-04-22T06:44:37.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.608+0000 7fb67e5ef640 1 -- 172.21.15.79:0/1111808725 wait complete. 2024-04-22T06:44:37.618 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:37.619 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:37.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.721+0000 7f3e3c752640 1 -- 172.21.15.79:0/2222209518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34106160 msgr2=0x7f3e3410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.721+0000 7f3e3c752640 1 --2- 172.21.15.79:0/2222209518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34106160 0x7f3e3410dd30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009ca0 tx=0x7f3e2c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- 172.21.15.79:0/2222209518 shutdown_connections 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 --2- 172.21.15.79:0/2222209518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34101a40 0x7f3e34105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 --2- 172.21.15.79:0/2222209518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34106160 0x7f3e3410dd30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 --2- 172.21.15.79:0/2222209518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101090 0x7f3e34101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- 172.21.15.79:0/2222209518 >> 172.21.15.79:0/2222209518 conn(0x7f3e34077f50 msgr2=0x7f3e34078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- 172.21.15.79:0/2222209518 shutdown_connections 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- 172.21.15.79:0/2222209518 wait complete. 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 Processor -- start 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- start start 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 0x7f3e34199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34106160 0x7f3e34199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34199fe0 0x7f3e341a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341110d0 con 0x7f3e34101090 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e34110f50 con 0x7f3e34106160 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3c752640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e34111250 con 0x7f3e34199fe0 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.722+0000 7f3e3a4ee640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 0x7f3e34199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 0x7f3e34199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53800/0 (socket says 172.21.15.79:53800) 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 -- 172.21.15.79:0/271694210 learned_addr learned my addr 172.21.15.79:0/271694210 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e39ced640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34106160 0x7f3e34199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3acef640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34199fe0 0x7f3e341a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 -- 172.21.15.79:0/271694210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34199fe0 msgr2=0x7f3e341a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34199fe0 0x7f3e341a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 -- 172.21.15.79:0/271694210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34106160 msgr2=0x7f3e34199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34106160 0x7f3e34199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 -- 172.21.15.79:0/271694210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e341a4d90 con 0x7f3e34101090 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e39ced640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34106160 0x7f3e34199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3acef640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34199fe0 0x7f3e341a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3a4ee640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 0x7f3e34199510 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000d770 tx=0x7f3e3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30017070 con 0x7f3e34101090 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e341a5080 con 0x7f3e34101090 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e300054d0 con 0x7f3e34101090 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e30004c30 con 0x7f3e34101090 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.723+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e341a5580 con 0x7f3e34101090 2024-04-22T06:44:37.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.724+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e34068d70 con 0x7f3e34101090 2024-04-22T06:44:37.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.725+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3e30004dd0 con 0x7f3e34101090 2024-04-22T06:44:37.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.726+0000 7f3e277fe640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e200a3e40 0x7f3e200a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.726+0000 7f3e39ced640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e200a3e40 0x7f3e200a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.726+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3e300d0a40 con 0x7f3e34101090 2024-04-22T06:44:37.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.726+0000 7f3e39ced640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e200a3e40 0x7f3e200a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3e34069c50 tx=0x7f3e28005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.730+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e3009cec0 con 0x7f3e34101090 2024-04-22T06:44:37.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.906+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e341093c0 con 0x7f3e34101090 2024-04-22T06:44:37.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.906+0000 7f3e277fe640 1 -- 172.21.15.79:0/271694210 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f3e3009c7c0 con 0x7f3e34101090 2024-04-22T06:44:37.906 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:44:37.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e200a3e40 msgr2=0x7f3e200a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e200a3e40 0x7f3e200a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3e34069c50 tx=0x7f3e28005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:37.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 msgr2=0x7f3e34199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 0x7f3e34199510 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000d770 tx=0x7f3e3000dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:37.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 shutdown_connections 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e200a3e40 0x7f3e200a6300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e34106160 0x7f3e34199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34199fe0 0x7f3e341a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 --2- 172.21.15.79:0/271694210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e34101090 0x7f3e34199510 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 >> 172.21.15.79:0/271694210 conn(0x7f3e34077f50 msgr2=0x7f3e340fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 shutdown_connections 2024-04-22T06:44:37.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:37.908+0000 7f3e3c752640 1 -- 172.21.15.79:0/271694210 wait complete. 2024-04-22T06:44:37.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 -- 172.21.15.79:0/566982178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 msgr2=0x7f413c107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 --2- 172.21.15.79:0/566982178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c107410 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f412c009930 tx=0x7f412c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 -- 172.21.15.79:0/566982178 shutdown_connections 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 --2- 172.21.15.79:0/566982178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f413c101b40 0x7f413c10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 --2- 172.21.15.79:0/566982178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 0x7f413c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 --2- 172.21.15.79:0/566982178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c107410 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 -- 172.21.15.79:0/566982178 >> 172.21.15.79:0/566982178 conn(0x7f413c076c50 msgr2=0x7f413c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 -- 172.21.15.79:0/566982178 shutdown_connections 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 -- 172.21.15.79:0/566982178 wait complete. 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 Processor -- start 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.024+0000 7f4142575640 1 -- start start 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 0x7f413c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f413c101b40 0x7f413c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c1a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f413c1110c0 con 0x7f413c107030 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f413c110f40 con 0x7f413c101b40 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f413c111240 con 0x7f413c1011a0 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c1a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c1a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53830/0 (socket says 172.21.15.79:53830) 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 -- 172.21.15.79:0/219532003 learned_addr learned my addr 172.21.15.79:0/219532003 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f413bfff640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 0x7f413c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 -- 172.21.15.79:0/219532003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 msgr2=0x7f413c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 0x7f413c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 -- 172.21.15.79:0/219532003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f413c101b40 msgr2=0x7f413c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f413c101b40 0x7f413c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 -- 172.21.15.79:0/219532003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c1a4ba0 con 0x7f413c107030 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f413bfff640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 0x7f413c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4140b12640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c1a4660 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f413400cad0 tx=0x7f413400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413400ae20 con 0x7f413c107030 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4134005020 con 0x7f413c107030 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c1a4d70 con 0x7f413c107030 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413400f6b0 con 0x7f413c107030 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.025+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c1a5240 con 0x7f413c107030 2024-04-22T06:44:38.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.026+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413c102590 con 0x7f413c107030 2024-04-22T06:44:38.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.027+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f41340051d0 con 0x7f413c107030 2024-04-22T06:44:38.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.028+0000 7f41397fa640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f41240a3e40 0x7f41240a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.028+0000 7f413bfff640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f41240a3e40 0x7f41240a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.028+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f41340d0bc0 con 0x7f413c107030 2024-04-22T06:44:38.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.028+0000 7f413bfff640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f41240a3e40 0x7f41240a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f412c006010 tx=0x7f412c009610 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.032+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4134014030 con 0x7f413c107030 2024-04-22T06:44:38.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.210+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f413c108d90 con 0x7f413c107030 2024-04-22T06:44:38.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.210+0000 7f41397fa640 1 -- 172.21.15.79:0/219532003 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f413409d040 con 0x7f413c107030 2024-04-22T06:44:38.209 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f41240a3e40 msgr2=0x7f41240a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f41240a3e40 0x7f41240a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f412c006010 tx=0x7f412c009610 comp rx=0 tx=0).stop 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 msgr2=0x7f413c1a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c1a4660 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f413400cad0 tx=0x7f413400cf90 comp rx=0 tx=0).stop 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 shutdown_connections 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f41240a3e40 0x7f41240a6300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f413c101b40 0x7f413c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f413c107030 0x7f413c1a4660 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 --2- 172.21.15.79:0/219532003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c1011a0 0x7f413c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.212+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 >> 172.21.15.79:0/219532003 conn(0x7f413c076c50 msgr2=0x7f413c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.213+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 shutdown_connections 2024-04-22T06:44:38.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.213+0000 7f4142575640 1 -- 172.21.15.79:0/219532003 wait complete. 2024-04-22T06:44:38.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:38 smithi079 ceph-mon[27946]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:38.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:38 smithi079 ceph-mon[27946]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-22T06:44:38.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:38 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4194231288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:38 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1111808725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:38 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/271694210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.222 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 -- 172.21.15.79:0/3393363926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc101090 msgr2=0x7f77fc101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 --2- 172.21.15.79:0/3393363926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc101090 0x7f77fc101470 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009900 tx=0x7f77f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 -- 172.21.15.79:0/3393363926 shutdown_connections 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 --2- 172.21.15.79:0/3393363926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77fc106160 0x7f77fc10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 --2- 172.21.15.79:0/3393363926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77fc101a40 0x7f77fc105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 --2- 172.21.15.79:0/3393363926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc101090 0x7f77fc101470 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 -- 172.21.15.79:0/3393363926 >> 172.21.15.79:0/3393363926 conn(0x7f77fc077f50 msgr2=0x7f77fc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 -- 172.21.15.79:0/3393363926 shutdown_connections 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 -- 172.21.15.79:0/3393363926 wait complete. 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 Processor -- start 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.327+0000 7f78045a4640 1 -- start start 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77fc101090 0x7f77fc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77fc106160 0x7f77fc198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 0x7f77fc19d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc1110a0 con 0x7f77fc19cf70 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc110f20 con 0x7f77fc106160 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc111220 con 0x7f77fc101090 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 0x7f77fc19d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 0x7f77fc19d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53842/0 (socket says 172.21.15.79:53842) 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 -- 172.21.15.79:0/732282558 learned_addr learned my addr 172.21.15.79:0/732282558 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802340640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77fc101090 0x7f77fc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7801b3f640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77fc106160 0x7f77fc198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 -- 172.21.15.79:0/732282558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77fc101090 msgr2=0x7f77fc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77fc101090 0x7f77fc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 -- 172.21.15.79:0/732282558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77fc106160 msgr2=0x7f77fc198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77fc106160 0x7f77fc198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 -- 172.21.15.79:0/732282558 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77fc19da60 con 0x7f77fc19cf70 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f7802b41640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 0x7f77fc19d420 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009fd0 tx=0x7f77f400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4017070 con 0x7f77fc19cf70 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77fc19dd50 con 0x7f77fc19cf70 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77f4003e20 con 0x7f77fc19cf70 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.328+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4010460 con 0x7f77fc19cf70 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.329+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77fc1a5540 con 0x7f77fc19cf70 2024-04-22T06:44:38.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.330+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77fc068d70 con 0x7f77fc19cf70 2024-04-22T06:44:38.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.331+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f77f4003970 con 0x7f77fc19cf70 2024-04-22T06:44:38.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.331+0000 7f77f37fe640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e80a3e40 0x7f77e80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.331+0000 7f7802340640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e80a3e40 0x7f77e80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.331+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f77f40d0870 con 0x7f77fc19cf70 2024-04-22T06:44:38.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.332+0000 7f7802340640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e80a3e40 0x7f77e80a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77f8000c00 tx=0x7f77f8004040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.336+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77f409ccf0 con 0x7f77fc19cf70 2024-04-22T06:44:38.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:38 smithi193 ceph-mon[31815]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:38 smithi193 ceph-mon[31815]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-22T06:44:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:38 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4194231288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:38 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1111808725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:38 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/271694210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:38 smithi190 ceph-mon[31421]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:38.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:38 smithi190 ceph-mon[31421]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-22T06:44:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:38 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4194231288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:38 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1111808725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:38 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/271694210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.545+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f77fc10a400 con 0x7f77fc19cf70 2024-04-22T06:44:38.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.545+0000 7f77f37fe640 1 -- 172.21.15.79:0/732282558 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f77f409c5f0 con 0x7f77fc19cf70 2024-04-22T06:44:38.544 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:44:38.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e80a3e40 msgr2=0x7f77e80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e80a3e40 0x7f77e80a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77f8000c00 tx=0x7f77f8004040 comp rx=0 tx=0).stop 2024-04-22T06:44:38.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 msgr2=0x7f77fc19d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 0x7f77fc19d420 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009fd0 tx=0x7f77f400b260 comp rx=0 tx=0).stop 2024-04-22T06:44:38.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 shutdown_connections 2024-04-22T06:44:38.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f77e80a3e40 0x7f77e80a6300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77fc106160 0x7f77fc198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77fc19cf70 0x7f77fc19d420 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 --2- 172.21.15.79:0/732282558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77fc101090 0x7f77fc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 >> 172.21.15.79:0/732282558 conn(0x7f77fc077f50 msgr2=0x7f77fc069520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 shutdown_connections 2024-04-22T06:44:38.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.547+0000 7f78045a4640 1 -- 172.21.15.79:0/732282558 wait complete. 2024-04-22T06:44:38.557 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:38.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 -- 172.21.15.79:0/3324619736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4101af0 msgr2=0x7fbfa410dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 --2- 172.21.15.79:0/3324619736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4101af0 0x7fbfa410dd10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009ca0 tx=0x7fbf9c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 -- 172.21.15.79:0/3324619736 shutdown_connections 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 --2- 172.21.15.79:0/3324619736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa4101130 0x7fbfa41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 --2- 172.21.15.79:0/3324619736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4101af0 0x7fbfa410dd10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 --2- 172.21.15.79:0/3324619736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4105e80 0x7fbfa4106260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 -- 172.21.15.79:0/3324619736 >> 172.21.15.79:0/3324619736 conn(0x7fbfa4077e30 msgr2=0x7fbfa4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 -- 172.21.15.79:0/3324619736 shutdown_connections 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 -- 172.21.15.79:0/3324619736 wait complete. 2024-04-22T06:44:38.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.670+0000 7fbfa9915640 1 Processor -- start 2024-04-22T06:44:38.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 -- start start 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4101130 0x7fbfa41976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa4101af0 0x7fbfa4197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 0x7fbfa419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41111f0 con 0x7fbfa4105e80 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4111070 con 0x7fbfa4101af0 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa9915640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4111370 con 0x7fbfa4101130 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 0x7fbfa419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 0x7fbfa419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53868/0 (socket says 172.21.15.79:53868) 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 -- 172.21.15.79:0/1565477880 learned_addr learned my addr 172.21.15.79:0/1565477880 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:38.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa27fc640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa4101af0 0x7fbfa4197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa2ffd640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4101130 0x7fbfa41976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 -- 172.21.15.79:0/1565477880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4101130 msgr2=0x7fbfa41976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4101130 0x7fbfa41976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 -- 172.21.15.79:0/1565477880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa4101af0 msgr2=0x7fbfa4197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa4101af0 0x7fbfa4197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa37fe640 1 -- 172.21.15.79:0/1565477880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa419c650 con 0x7fbfa4105e80 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.671+0000 7fbfa2ffd640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4101130 0x7fbfa41976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.672+0000 7fbfa37fe640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 0x7fbfa419bf20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c02c700 tx=0x7fbf9c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.672+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c041020 con 0x7fbfa4105e80 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.672+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa419c8e0 con 0x7fbfa4105e80 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.672+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf9c032960 con 0x7fbfa4105e80 2024-04-22T06:44:38.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.672+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c039040 con 0x7fbfa4105e80 2024-04-22T06:44:38.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.672+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa41a41d0 con 0x7fbfa4105e80 2024-04-22T06:44:38.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.673+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa4068550 con 0x7fbfa4105e80 2024-04-22T06:44:38.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.674+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fbf9c044020 con 0x7fbfa4105e80 2024-04-22T06:44:38.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.674+0000 7fbfa8913640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf8c0a3e40 0x7fbf8c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.674+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fbf9c0f2770 con 0x7fbfa4105e80 2024-04-22T06:44:38.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.674+0000 7fbfa2ffd640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf8c0a3e40 0x7fbf8c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.675+0000 7fbfa2ffd640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf8c0a3e40 0x7fbf8c0a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf98005fd0 tx=0x7fbf98005ea0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.679+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf9c04b020 con 0x7fbfa4105e80 2024-04-22T06:44:38.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.899+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfa4109340 con 0x7fbfa4105e80 2024-04-22T06:44:38.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.899+0000 7fbfa8913640 1 -- 172.21.15.79:0/1565477880 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7fbf9c0bed00 con 0x7fbfa4105e80 2024-04-22T06:44:38.898 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:38.898 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:44:38.898 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf8c0a3e40 msgr2=0x7fbf8c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf8c0a3e40 0x7fbf8c0a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf98005fd0 tx=0x7fbf98005ea0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 msgr2=0x7fbfa419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 0x7fbfa419bf20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c02c700 tx=0x7fbf9c0346d0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 shutdown_connections 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbf8c0a3e40 0x7fbf8c0a6300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa4101af0 0x7fbfa4197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbfa4105e80 0x7fbfa419bf20 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 --2- 172.21.15.79:0/1565477880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfa4101130 0x7fbfa41976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.901+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 >> 172.21.15.79:0/1565477880 conn(0x7fbfa4077e30 msgr2=0x7fbfa40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.902+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 shutdown_connections 2024-04-22T06:44:38.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:38.902+0000 7fbfa9915640 1 -- 172.21.15.79:0/1565477880 wait complete. 2024-04-22T06:44:38.911 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 -- 172.21.15.79:0/806325644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 msgr2=0x7fac44106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/806325644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac44106210 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fac34009930 tx=0x7fac3402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 -- 172.21.15.79:0/806325644 shutdown_connections 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/806325644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac441011a0 0x7fac44101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/806325644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac44101b60 0x7fac4410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/806325644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac44106210 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 -- 172.21.15.79:0/806325644 >> 172.21.15.79:0/806325644 conn(0x7fac44077e30 msgr2=0x7fac44078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 -- 172.21.15.79:0/806325644 shutdown_connections 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.013+0000 7fac4a7c3640 1 -- 172.21.15.79:0/806325644 wait complete. 2024-04-22T06:44:39.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 Processor -- start 2024-04-22T06:44:39.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 -- start start 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac441011a0 0x7fac441986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101b60 0x7fac44198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac4419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fac441110e0 con 0x7fac44105e30 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fac44110f60 con 0x7fac44101b60 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac4a7c3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac44111260 con 0x7fac441011a0 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac4419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac4419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53880/0 (socket says 172.21.15.79:53880) 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 -- 172.21.15.79:0/2734030888 learned_addr learned my addr 172.21.15.79:0/2734030888 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac437fe640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101b60 0x7fac44198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac43fff640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac441011a0 0x7fac441986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 -- 172.21.15.79:0/2734030888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac441011a0 msgr2=0x7fac441986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac441011a0 0x7fac441986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 -- 172.21.15.79:0/2734030888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101b60 msgr2=0x7fac44198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101b60 0x7fac44198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.014+0000 7fac48d60640 1 -- 172.21.15.79:0/2734030888 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4419d6b0 con 0x7fac44105e30 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.015+0000 7fac48d60640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac4419cf80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00eaa0 tx=0x7fac3c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.015+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00cdb0 con 0x7fac44105e30 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.015+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c005020 con 0x7fac44105e30 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.015+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4419d9a0 con 0x7fac44105e30 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.015+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0116d0 con 0x7fac44105e30 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.015+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac441a5230 con 0x7fac44105e30 2024-04-22T06:44:39.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.016+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac44102570 con 0x7fac44105e30 2024-04-22T06:44:39.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.018+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fac3c0051d0 con 0x7fac44105e30 2024-04-22T06:44:39.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.018+0000 7fac417fa640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fac2c0a3e40 0x7fac2c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.018+0000 7fac43fff640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fac2c0a3e40 0x7fac2c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.018+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fac3c0d0fb0 con 0x7fac44105e30 2024-04-22T06:44:39.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.019+0000 7fac43fff640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fac2c0a3e40 0x7fac2c0a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fac34036040 tx=0x7fac34000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.022+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac3c016030 con 0x7fac44105e30 2024-04-22T06:44:39.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.205+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac44068de0 con 0x7fac44105e30 2024-04-22T06:44:39.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.207+0000 7fac417fa640 1 -- 172.21.15.79:0/2734030888 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23025 (secure 0 0 0) 0x7fac44068de0 con 0x7fac44105e30 2024-04-22T06:44:39.206 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:39.206 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":88,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:44:24.943986+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:39.207 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fac2c0a3e40 msgr2=0x7fac2c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fac2c0a3e40 0x7fac2c0a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fac34036040 tx=0x7fac34000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 msgr2=0x7fac4419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac4419cf80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00eaa0 tx=0x7fac3c00ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 shutdown_connections 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fac2c0a3e40 0x7fac2c0a6300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac44101b60 0x7fac44198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac44105e30 0x7fac4419cf80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 --2- 172.21.15.79:0/2734030888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac441011a0 0x7fac441986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 >> 172.21.15.79:0/2734030888 conn(0x7fac44077e30 msgr2=0x7fac44069420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 shutdown_connections 2024-04-22T06:44:39.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.209+0000 7fac4a7c3640 1 -- 172.21.15.79:0/2734030888 wait complete. 2024-04-22T06:44:39.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:39 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/219532003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:39.217 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:39 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/732282558' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:39.218 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:39 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1565477880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:39.219 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 922603 2024-04-22T06:44:39.219 DEBUG:tasks.quiescer:Yielding 2024-04-22T06:44:39.219 INFO:teuthology.run_tasks:Running task workunit... 2024-04-22T06:44:39.233 INFO:tasks.workunit:Pulling workunits from ref 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:44:39.234 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T06:44:39.234 INFO:tasks.workunit:timeout=3h 2024-04-22T06:44:39.234 INFO:tasks.workunit:cleanup=True 2024-04-22T06:44:39.234 DEBUG:teuthology.orchestra.run.smithi079:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:39.235 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:39.250 INFO:teuthology.orchestra.run.smithi079.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:39.250 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T06:44:39.250 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-22T06:44:39.250 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:44:39.250 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T06:44:39.250 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-04-22 06:42:30.101728176 +0000 2024-04-22T06:44:39.251 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-04-22 06:42:30.101728176 +0000 2024-04-22T06:44:39.251 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-04-22 06:44:22.988687441 +0000 2024-04-22T06:44:39.251 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-04-22 06:42:30.101728176 +0000 2024-04-22T06:44:39.251 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:39.251 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T06:44:39.327 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:44:39.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 -- 172.21.15.79:0/1424624771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 msgr2=0x7f382c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 --2- 172.21.15.79:0/1424624771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c101620 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3818009930 tx=0x7f381802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 -- 172.21.15.79:0/1424624771 shutdown_connections 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 --2- 172.21.15.79:0/1424624771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f382c101b60 0x7f382c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 --2- 172.21.15.79:0/1424624771 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c101620 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 --2- 172.21.15.79:0/1424624771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f382c105e30 0x7f382c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 -- 172.21.15.79:0/1424624771 >> 172.21.15.79:0/1424624771 conn(0x7f382c077e30 msgr2=0x7f382c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 -- 172.21.15.79:0/1424624771 shutdown_connections 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 -- 172.21.15.79:0/1424624771 wait complete. 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 Processor -- start 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.337+0000 7f38330c1640 1 -- start start 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f382c101b60 0x7f382c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f382c105e30 0x7f382c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f382c112240 con 0x7f382c1011a0 2024-04-22T06:44:39.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f382c1120c0 con 0x7f382c101b60 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f382c1123c0 con 0x7f382c105e30 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c19e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53892/0 (socket says 172.21.15.79:53892) 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 -- 172.21.15.79:0/291784799 learned_addr learned my addr 172.21.15.79:0/291784799 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f383165e640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f382c105e30 0x7f382c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 -- 172.21.15.79:0/291784799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f382c105e30 msgr2=0x7f382c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f382c105e30 0x7f382c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 -- 172.21.15.79:0/291784799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f382c101b60 msgr2=0x7f382c19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f382c101b60 0x7f382c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 -- 172.21.15.79:0/291784799 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382c198d60 con 0x7f382c1011a0 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3830e5d640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c19e3e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f382000d950 tx=0x7f382000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3820017070 con 0x7f382c1011a0 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3820009c30 con 0x7f382c1011a0 2024-04-22T06:44:39.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382c199050 con 0x7f382c1011a0 2024-04-22T06:44:39.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.338+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382000f620 con 0x7f382c1011a0 2024-04-22T06:44:39.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.339+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f382c1a52a0 con 0x7f382c1011a0 2024-04-22T06:44:39.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.340+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c068de0 con 0x7f382c1011a0 2024-04-22T06:44:39.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.341+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3820005030 con 0x7f382c1011a0 2024-04-22T06:44:39.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.341+0000 7f3829ffb640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f381c0a3e40 0x7f381c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.341+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f38200d0750 con 0x7f382c1011a0 2024-04-22T06:44:39.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.341+0000 7f382bfff640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f381c0a3e40 0x7f381c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.342+0000 7f382bfff640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f381c0a3e40 0x7f381c0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3818005770 tx=0x7f3818000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.345+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f382009cbd0 con 0x7f382c1011a0 2024-04-22T06:44:39.352 INFO:tasks.workunit.client.0.smithi079.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T06:44:39.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:39 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/219532003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:39.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:39 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/732282558' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:39.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:39 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1565477880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:39.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:39 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/219532003' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:39.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:39 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/732282558' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:39.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:39 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1565477880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.579+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f382c199ac0 con 0x7f382c1011a0 2024-04-22T06:44:39.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.579+0000 7f3829ffb640 1 -- 172.21.15.79:0/291784799 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7f3820009de0 con 0x7f382c1011a0 2024-04-22T06:44:39.578 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:39.578 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:44:39.578 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.581+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f381c0a3e40 msgr2=0x7f381c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.581+0000 7f38330c1640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f381c0a3e40 0x7f381c0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3818005770 tx=0x7f3818000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.581+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 msgr2=0x7f382c19e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.581+0000 7f38330c1640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c19e3e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f382000d950 tx=0x7f382000de10 comp rx=0 tx=0).stop 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.581+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 shutdown_connections 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.581+0000 7f38330c1640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f381c0a3e40 0x7f381c0a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.582+0000 7f38330c1640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f382c101b60 0x7f382c19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.582+0000 7f38330c1640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f382c105e30 0x7f382c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.582+0000 7f38330c1640 1 --2- 172.21.15.79:0/291784799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f382c1011a0 0x7f382c19e3e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.582+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 >> 172.21.15.79:0/291784799 conn(0x7f382c077e30 msgr2=0x7f382c0feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.582+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 shutdown_connections 2024-04-22T06:44:39.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.582+0000 7f38330c1640 1 -- 172.21.15.79:0/291784799 wait complete. 2024-04-22T06:44:39.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:42:24.011937+0000', 'modified': '2024-04-22T06:44:23.666906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24467, 'mds_1': 24440, 'mds_2': 24436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24406': {'gid': 24406, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3775212644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3775212644}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3775212644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24413': {'gid': 24413, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6835/2543246858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 2543246858}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 2543246858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/678626858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 678626858}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 678626858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.193:6839/746968688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 746968688}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 746968688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.190:6837/2879949884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 2879949884}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 2879949884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.190:6839/4154999918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 4154999918}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 4154999918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24467, 'qdb_cluster': [24467, 24436, 24440]} 2024-04-22T06:44:39.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T06:44:39.592 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 -- 172.21.15.79:0/4270767462 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec1031e0 msgr2=0x7f28ec1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 --2- 172.21.15.79:0/4270767462 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec1031e0 0x7f28ec1035c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009900 tx=0x7f28e802bd60 comp rx=0 tx=0).stop 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 -- 172.21.15.79:0/4270767462 shutdown_connections 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 --2- 172.21.15.79:0/4270767462 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec103b00 0x7f28ec10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 --2- 172.21.15.79:0/4270767462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec10c520 0x7f28ec10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 --2- 172.21.15.79:0/4270767462 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec1031e0 0x7f28ec1035c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 -- 172.21.15.79:0/4270767462 >> 172.21.15.79:0/4270767462 conn(0x7f28ec0771d0 msgr2=0x7f28ec0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 -- 172.21.15.79:0/4270767462 shutdown_connections 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.736+0000 7f28f33b2640 1 -- 172.21.15.79:0/4270767462 wait complete. 2024-04-22T06:44:39.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 Processor -- start 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 -- start start 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec1031e0 0x7f28ec100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 0x7f28ec101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec10c520 0x7f28ec1018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec112210 con 0x7f28ec103b00 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec112090 con 0x7f28ec10c520 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f33b2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec112390 con 0x7f28ec1031e0 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 0x7f28ec101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 0x7f28ec101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53914/0 (socket says 172.21.15.79:53914) 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 -- 172.21.15.79:0/2937785186 learned_addr learned my addr 172.21.15.79:0/2937785186 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f194f640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec10c520 0x7f28ec1018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f114e640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec1031e0 0x7f28ec100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 -- 172.21.15.79:0/2937785186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec1031e0 msgr2=0x7f28ec100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec1031e0 0x7f28ec100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 -- 172.21.15.79:0/2937785186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec10c520 msgr2=0x7f28ec1018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec10c520 0x7f28ec1018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f094d640 1 -- 172.21.15.79:0/2937785186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec1a6cd0 con 0x7f28ec103b00 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f114e640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec1031e0 0x7f28ec100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.737+0000 7f28f194f640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec10c520 0x7f28ec1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.738+0000 7f28f094d640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 0x7f28ec101370 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00d950 tx=0x7f28dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.738+0000 7f28e27fc640 1 -- 172.21.15.79:0/2937785186 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc017070 con 0x7f28ec103b00 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.738+0000 7f28e27fc640 1 -- 172.21.15.79:0/2937785186 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28dc009c30 con 0x7f28ec103b00 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.738+0000 7f28e27fc640 1 -- 172.21.15.79:0/2937785186 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00f620 con 0x7f28ec103b00 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.738+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec1a6fc0 con 0x7f28ec103b00 2024-04-22T06:44:39.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.738+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ec1a74d0 con 0x7f28ec103b00 2024-04-22T06:44:39.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.740+0000 7f28e27fc640 1 -- 172.21.15.79:0/2937785186 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f28dc010460 con 0x7f28ec103b00 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.740+0000 7f28f5fa1740 1 Processor -- start 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.740+0000 7f28f5fa1740 1 -- start start 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.740+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.740+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 0x5615b79cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5615b7928b10 con 0x5615b79c5160 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5615b78c6240 con 0x5615b79cdce0 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615b7924ad0 con 0x5615b79c47b0 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28e27fc640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28d80a3ec0 0x7f28d80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 0x5615b79cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 0x5615b79cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53918/0 (socket says 172.21.15.79:53918) 2024-04-22T06:44:39.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 -- 172.21.15.79:0/209164304 learned_addr learned my addr 172.21.15.79:0/209164304 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f194f640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28d80a3ec0 0x7f28d80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28e27fc640 1 -- 172.21.15.79:0/2937785186 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f28dc0d06d0 con 0x7f28ec103b00 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f194f640 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f094d640 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 -- 172.21.15.79:0/209164304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 msgr2=0x5615b79c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 -- 172.21.15.79:0/209164304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 msgr2=0x5615b79d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 -- 172.21.15.79:0/209164304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615b763b2d0 con 0x5615b79c5160 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f094d640 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f194f640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28d80a3ec0 0x7f28d80a6380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28e400eba0 tx=0x7f28e4009d50 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28f114e640 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 0x5615b79cd710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f28e8036040 tx=0x7f28e802c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81de581ad15d57ea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.741+0000 7f28d77fe640 1 -- 172.21.15.79:0/209164304 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8041020 con 0x5615b79c5160 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28d77fe640 1 -- 172.21.15.79:0/209164304 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e802c910 con 0x5615b79c5160 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28d77fe640 1 -- 172.21.15.79:0/209164304 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e80053c0 con 0x5615b79c5160 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 -- 172.21.15.79:0/209164304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 msgr2=0x5615b79cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 0x5615b79cd710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f28e8036040 tx=0x7f28e802c7b0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 -- 172.21.15.79:0/209164304 shutdown_connections 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c5160 0x5615b79cd710 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/209164304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 -- 172.21.15.79:0/209164304 >> 172.21.15.79:0/209164304 conn(0x5615b79bea60 msgr2=0x5615b79bee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 -- 172.21.15.79:0/209164304 shutdown_connections 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.742+0000 7f28f5fa1740 1 -- 172.21.15.79:0/209164304 wait complete. 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 -- start start 2024-04-22T06:44:39.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 Processor -- start 2024-04-22T06:44:39.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c5160 0x5615b79cb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79cb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5615b79d0bf0 con 0x5615b79cdce0 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5615b78b1190 con 0x5615b79c5160 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615b79d0d70 con 0x5615b79c47b0 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f5fa1740 10 client.0 _ll_register_callbacks cb 0x5615b78ac8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f194f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f194f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79cb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53932/0 (socket says 172.21.15.79:53932) 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f194f640 1 -- 172.21.15.79:0/3253807056 learned_addr learned my addr 172.21.15.79:0/3253807056 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f094d640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c5160 0x5615b79cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.743+0000 7f28f114e640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f194f640 1 -- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 msgr2=0x5615b79cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f194f640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f194f640 1 -- 172.21.15.79:0/3253807056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c5160 msgr2=0x5615b79cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f194f640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c5160 0x5615b79cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f194f640 1 -- 172.21.15.79:0/3253807056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615b77056b0 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f114e640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f194f640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79cb770 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009ca0 tx=0x7f28e4008220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1041b4f4d2980026 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28d4ff9640 1 -- 172.21.15.79:0/3253807056 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e400cd20 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28d4ff9640 1 -- 172.21.15.79:0/3253807056 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f28e4027480 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e802c910 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28d4ff9640 1 -- 172.21.15.79:0/3253807056 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f28e4027680 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 10 client.15639 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5615b79dcfa0 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28d4ff9640 1 -- 172.21.15.79:0/3253807056 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e802c910 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28d4ff9640 1 -- 172.21.15.79:0/3253807056 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f28e401b050 con 0x5615b79cdce0 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 10 client.15639 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:39.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 10 client.15639 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 20 client.15639 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b79df630 0x5615b79e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 4 client.15639 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.744+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615b7936530 con 0x5615b79df630 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.745+0000 7f28f114e640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b79df630 0x5615b79e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.745+0000 7f28d4ff9640 10 client.15639 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.745+0000 7f28f114e640 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b79df630 0x5615b79e0aa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ed4c1560870f0d9 server_cookie=1c6492bdac2a03b3 in_seq=0 out_seq=0 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28d4ff9640 1 -- 172.21.15.79:0/3253807056 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615b7936530 con 0x5615b79df630 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28d4ff9640 10 client.15639 handle_command_reply: tid=0 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 client.15639 shutdown 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 msgr2=0x5615b79cb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79cb770 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009ca0 tx=0x7f28e4008220 comp rx=0 tx=0).stop 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 shutdown_connections 2024-04-22T06:44:39.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b79df630 0x5615b79e0aa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c5160 0x5615b79cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28d4ff9640 0 client.15639 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79cb770 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3253807056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c47b0 0x5615b79cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 >> 172.21.15.79:0/3253807056 conn(0x5615b79bea60 msgr2=0x5615b79c0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 shutdown_connections 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3253807056 wait complete. 2024-04-22T06:44:39.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.746+0000 7f28f5fa1740 20 client.15639 trim_cache size 0 max 16384 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 Processor -- start 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 -- start start 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c47b0 0x5615b79cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79c8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5615b79dede0 con 0x5615b79cdce0 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5615b79df4b0 con 0x5615b79c47b0 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615b79dfb80 con 0x5615b79c5160 2024-04-22T06:44:39.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f194f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f194f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53938/0 (socket says 172.21.15.79:53938) 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.764+0000 7f28f194f640 1 -- 172.21.15.79:0/3955855278 learned_addr learned my addr 172.21.15.79:0/3955855278 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f194f640 1 -- 172.21.15.79:0/3955855278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 msgr2=0x5615b79c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f094d640 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f194f640 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f194f640 1 -- 172.21.15.79:0/3955855278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c47b0 msgr2=0x5615b79cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f194f640 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c47b0 0x5615b79cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f194f640 1 -- 172.21.15.79:0/3955855278 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615b79c0040 con 0x5615b79cdce0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f094d640 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f194f640 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79c8490 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f28e4004a60 tx=0x7f28e401af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bead3b5dc595b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28d6ffd640 1 -- 172.21.15.79:0/3955855278 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e400e3f0 con 0x5615b79cdce0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28d6ffd640 1 -- 172.21.15.79:0/3955855278 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e4027480 con 0x5615b79cdce0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28d6ffd640 1 -- 172.21.15.79:0/3955855278 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40305c0 con 0x5615b79cdce0 2024-04-22T06:44:39.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3955855278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 msgr2=0x5615b79c8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79c8490 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f28e4004a60 tx=0x7f28e401af90 comp rx=0 tx=0).stop 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3955855278 shutdown_connections 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79cdce0 0x5615b79c8490 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/3955855278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79c47b0 0x5615b79cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3955855278 >> 172.21.15.79:0/3955855278 conn(0x5615b79bea60 msgr2=0x5615b79b9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3955855278 shutdown_connections 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.765+0000 7f28f5fa1740 1 -- 172.21.15.79:0/3955855278 wait complete. 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 -- start start 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 Processor -- start 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 0x5615b79e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5615b79e0920 con 0x5615b79c47b0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5615b79e0250 con 0x5615b79cdce0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615b79e0ff0 con 0x5615b79c5160 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f5fa1740 10 client.0 _ll_register_callbacks cb 0x7f28e4009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f114e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 0x5615b79e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f114e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 0x5615b79e28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53948/0 (socket says 172.21.15.79:53948) 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f114e640 1 -- 172.21.15.79:0/1767054503 learned_addr learned my addr 172.21.15.79:0/1767054503 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f094d640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.766+0000 7f28f194f640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f114e640 1 -- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 msgr2=0x5615b79e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f114e640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79e2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f114e640 1 -- 172.21.15.79:0/1767054503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 msgr2=0x5615b79e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f114e640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f114e640 1 -- 172.21.15.79:0/1767054503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615b79e71a0 con 0x5615b79c47b0 2024-04-22T06:44:39.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f194f640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f114e640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 0x5615b79e28b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f28e802c820 tx=0x7f28e80041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b82c8e480d45bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28d4ff9640 1 -- 172.21.15.79:0/1767054503 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e804a070 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28d4ff9640 1 -- 172.21.15.79:0/1767054503 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f28e802cd20 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e4027480 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28d4ff9640 1 -- 172.21.15.79:0/1767054503 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f28e8056050 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 10 client.15645 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5615b7a459b0 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28d4ff9640 1 -- 172.21.15.79:0/1767054503 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e4027480 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28d4ff9640 1 -- 172.21.15.79:0/1767054503 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f28e8034440 con 0x5615b79c47b0 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 10 client.15645 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 10 client.15645 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 20 client.15645 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b7a4a130 0x5615b7a48e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 4 client.15645 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.767+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "status"}) -- 0x5615b79303a0 con 0x5615b7a4a130 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f094d640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b7a4a130 0x5615b7a48e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28d4ff9640 10 client.15645 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f094d640 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b7a4a130 0x5615b7a48e40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f386fc36f818d8d5 server_cookie=8f7a0da5ee4642b3 in_seq=0 out_seq=0 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28d4ff9640 1 -- 172.21.15.79:0/1767054503 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5615b79303a0 con 0x5615b7a4a130 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28d4ff9640 10 client.15645 handle_command_reply: tid=0 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 client.15645 shutdown 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 msgr2=0x5615b79e28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 0x5615b79e28b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f28e802c820 tx=0x7f28e80041e0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 shutdown_connections 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5615b7a4a130 0x5615b7a48e40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615b79cdce0 0x5615b79e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28d4ff9640 0 client.15645 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b79c5160 0x5615b79e2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 --2- 172.21.15.79:0/1767054503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615b79c47b0 0x5615b79e28b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 >> 172.21.15.79:0/1767054503 conn(0x5615b79bea60 msgr2=0x5615b79c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 shutdown_connections 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.768+0000 7f28f5fa1740 1 -- 172.21.15.79:0/1767054503 wait complete. 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f5fa1740 20 client.15645 trim_cache size 0 max 16384 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 2, 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24436, 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:44:39.774 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 132.065822693, 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 22, 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 88, 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 149.906775595 2024-04-22T06:44:39.775 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28d80a3ec0 msgr2=0x7f28d80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28d80a3ec0 0x7f28d80a6380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28e400eba0 tx=0x7f28e4009d50 comp rx=0 tx=0).stop 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 msgr2=0x7f28ec101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 0x7f28ec101370 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00d950 tx=0x7f28dc00de10 comp rx=0 tx=0).stop 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 shutdown_connections 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28d80a3ec0 0x7f28d80a6380 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28ec10c520 0x7f28ec1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f28ec103b00 0x7f28ec101370 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 --2- 172.21.15.79:0/2937785186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28ec1031e0 0x7f28ec100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 >> 172.21.15.79:0/2937785186 conn(0x7f28ec0771d0 msgr2=0x7f28ec0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 shutdown_connections 2024-04-22T06:44:39.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.769+0000 7f28f33b2640 1 -- 172.21.15.79:0/2937785186 wait complete. 2024-04-22T06:44:39.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T06:44:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/4081034070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48101090 msgr2=0x7f3e48101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/4081034070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48101090 0x7f3e48101470 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009900 tx=0x7f3e3802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/4081034070 shutdown_connections 2024-04-22T06:44:39.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/4081034070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e48106160 0x7f3e4810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/4081034070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e48101a40 0x7f3e48105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/4081034070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48101090 0x7f3e48101470 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/4081034070 >> 172.21.15.79:0/4081034070 conn(0x7f3e48077f50 msgr2=0x7f3e48078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/4081034070 shutdown_connections 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/4081034070 wait complete. 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 Processor -- start 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 -- start start 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.884+0000 7f3e4fbf4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e48101090 0x7f3e4819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 0x7f3e4819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e48198590 0x7f3e48198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3e481110a0 con 0x7f3e48106160 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48110f20 con 0x7f3e48101090 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48111220 con 0x7f3e48198590 2024-04-22T06:44:39.885 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 0x7f3e4819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 0x7f3e4819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53970/0 (socket says 172.21.15.79:53970) 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 -- 172.21.15.79:0/2280671730 learned_addr learned my addr 172.21.15.79:0/2280671730 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e48198590 0x7f3e48198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d990640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e48101090 0x7f3e4819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 -- 172.21.15.79:0/2280671730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e48198590 msgr2=0x7f3e48198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e48198590 0x7f3e48198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 -- 172.21.15.79:0/2280671730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e48101090 msgr2=0x7f3e4819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e48101090 0x7f3e4819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 -- 172.21.15.79:0/2280671730 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e481990b0 con 0x7f3e48106160 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4d18f640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 0x7f3e4819e920 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d770 tx=0x7f3e4400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e3effd640 1 -- 172.21.15.79:0/2280671730 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44017070 con 0x7f3e48106160 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e3effd640 1 -- 172.21.15.79:0/2280671730 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e44009c30 con 0x7f3e48106160 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e481993a0 con 0x7f3e48106160 2024-04-22T06:44:39.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e3effd640 1 -- 172.21.15.79:0/2280671730 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4400f620 con 0x7f3e48106160 2024-04-22T06:44:39.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.885+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e481a55f0 con 0x7f3e48106160 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.887+0000 7f3e3effd640 1 -- 172.21.15.79:0/2280671730 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3e4401a070 con 0x7f3e48106160 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 Processor -- start 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 -- start start 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e3effd640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e340a3e40 0x7f3e340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e400130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e358b10 con 0x55ff3e3fdce0 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e2f6240 con 0x55ff3e3f47b0 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.888+0000 7f3e527e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e354ad0 con 0x55ff3e3f5160 2024-04-22T06:44:39.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4d18f640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e340a3e40 0x7f3e340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e400130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e400130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53980/0 (socket says 172.21.15.79:53980) 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 -- 172.21.15.79:0/2494812010 learned_addr learned my addr 172.21.15.79:0/2494812010 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4d990640 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e3effd640 1 -- 172.21.15.79:0/2280671730 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3e440d06a0 con 0x7f3e48106160 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 -- 172.21.15.79:0/2494812010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 msgr2=0x55ff3e3fd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 -- 172.21.15.79:0/2494812010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 msgr2=0x55ff3e3f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 -- 172.21.15.79:0/2494812010 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff3e06b2d0 con 0x55ff3e3fdce0 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4d18f640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e340a3e40 0x7f3e340a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e440cdd00 tx=0x7f3e4400df10 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e400130 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3e40013a80 tx=0x7f3e4000a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f956c7f47b6bbc12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e33fff640 1 -- 172.21.15.79:0/2494812010 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40004b20 con 0x55ff3e3fdce0 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e33fff640 1 -- 172.21.15.79:0/2494812010 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e40003e20 con 0x55ff3e3fdce0 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.889+0000 7f3e33fff640 1 -- 172.21.15.79:0/2494812010 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40004020 con 0x55ff3e3fdce0 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 -- 172.21.15.79:0/2494812010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 msgr2=0x55ff3e400130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e400130 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3e40013a80 tx=0x7f3e4000a590 comp rx=0 tx=0).stop 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 -- 172.21.15.79:0/2494812010 shutdown_connections 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e400130 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2494812010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 -- 172.21.15.79:0/2494812010 >> 172.21.15.79:0/2494812010 conn(0x55ff3e3eea60 msgr2=0x55ff3e3eee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 -- 172.21.15.79:0/2494812010 shutdown_connections 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.890+0000 7f3e527e9740 1 -- 172.21.15.79:0/2494812010 wait complete. 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 -- start start 2024-04-22T06:44:39.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 Processor -- start 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3fb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e400c50 con 0x55ff3e3f47b0 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e2e1190 con 0x55ff3e3f5160 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e400dd0 con 0x55ff3e3fdce0 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e527e9740 10 client.0 _ll_register_callbacks cb 0x55ff3e2dc8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53994/0 (socket says 172.21.15.79:53994) 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 -- 172.21.15.79:0/2931900359 learned_addr learned my addr 172.21.15.79:0/2931900359 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4d990640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 -- 172.21.15.79:0/2931900359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3fdce0 msgr2=0x55ff3e3fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3fb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 -- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f5160 msgr2=0x55ff3e3fb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4e191640 1 -- 172.21.15.79:0/2931900359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff3e1356b0 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.891+0000 7f3e4d990640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e4e191640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3facc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000a550 tx=0x7f3e40013730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08b75486556c2bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e317fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e317fa640 1 -- 172.21.15.79:0/2931900359 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40003b70 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e317fa640 1 -- 172.21.15.79:0/2931900359 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3e40004a40 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e40003e20 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e317fa640 1 -- 172.21.15.79:0/2931900359 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3e4002f050 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 10 client.15657 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ff3e40cf40 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e317fa640 1 -- 172.21.15.79:0/2931900359 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e40003e20 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e317fa640 1 -- 172.21.15.79:0/2931900359 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3e40028060 con 0x55ff3e3f47b0 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 10 client.15657 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:39.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 10 client.15657 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 20 client.15657 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e40f220 0x55ff3e410690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 4 client.15657 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.892+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff3e366530 con 0x55ff3e40f220 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.893+0000 7f3e4d990640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e40f220 0x55ff3e410690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.893+0000 7f3e4d990640 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e40f220 0x55ff3e410690 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ebfc357e466dd68 server_cookie=6de4a448c42a4f6c in_seq=0 out_seq=0 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.893+0000 7f3e317fa640 10 client.15657 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e317fa640 1 -- 172.21.15.79:0/2931900359 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff3e366530 con 0x55ff3e40f220 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e317fa640 10 client.15657 handle_command_reply: tid=0 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 client.15657 shutdown 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 msgr2=0x55ff3e3facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3facc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000a550 tx=0x7f3e40013730 comp rx=0 tx=0).stop 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 shutdown_connections 2024-04-22T06:44:39.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e40f220 0x55ff3e410690 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f5160 0x55ff3e3fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3fb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 --2- 172.21.15.79:0/2931900359 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3facc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 >> 172.21.15.79:0/2931900359 conn(0x55ff3e3eea60 msgr2=0x55ff3e3f0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e317fa640 0 client.15657 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 shutdown_connections 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 1 -- 172.21.15.79:0/2931900359 wait complete. 2024-04-22T06:44:39.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.894+0000 7f3e527e9740 20 client.15657 trim_cache size 0 max 16384 2024-04-22T06:44:39.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 Processor -- start 2024-04-22T06:44:39.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 -- start start 2024-04-22T06:44:39.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e418d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3eee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e40e9d0 con 0x55ff3e3fdce0 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e40f0a0 con 0x55ff3e3f47b0 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.912+0000 7f3e527e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e40f770 con 0x55ff3e3f5160 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3eee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54002/0 (socket says 172.21.15.79:54002) 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 -- 172.21.15.79:0/202443376 learned_addr learned my addr 172.21.15.79:0/202443376 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4d990640 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e418d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 -- 172.21.15.79:0/202443376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 msgr2=0x55ff3e418d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e418d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 -- 172.21.15.79:0/202443376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 msgr2=0x55ff3e3fd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3fd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 -- 172.21.15.79:0/202443376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff3e3f68c0 con 0x55ff3e3fdce0 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e4e191640 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3eee40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000ec90 tx=0x7f3e40015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c62c77fc0e049e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e337fe640 1 -- 172.21.15.79:0/202443376 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4000ae30 con 0x55ff3e3fdce0 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e337fe640 1 -- 172.21.15.79:0/202443376 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e400055c0 con 0x55ff3e3fdce0 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.913+0000 7f3e337fe640 1 -- 172.21.15.79:0/202443376 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4001e780 con 0x55ff3e3fdce0 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 -- 172.21.15.79:0/202443376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 msgr2=0x55ff3e3eee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3eee40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000ec90 tx=0x7f3e40015040 comp rx=0 tx=0).stop 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 -- 172.21.15.79:0/202443376 shutdown_connections 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3fdce0 0x55ff3e3eee40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f5160 0x55ff3e418d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 --2- 172.21.15.79:0/202443376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 -- 172.21.15.79:0/202443376 >> 172.21.15.79:0/202443376 conn(0x55ff3e3eea60 msgr2=0x55ff3e3f5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 -- 172.21.15.79:0/202443376 shutdown_connections 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 -- 172.21.15.79:0/202443376 wait complete. 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 -- start start 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.914+0000 7f3e527e9740 1 Processor -- start 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 0x55ff3e3f8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3fdce0 0x55ff3e417e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e410510 con 0x55ff3e3f5160 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e40fe40 con 0x55ff3e3fdce0 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff3e410be0 con 0x55ff3e3f47b0 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 10 client.0 _ll_register_callbacks cb 0x7f3e40013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 0x55ff3e3f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 0x55ff3e3f8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54008/0 (socket says 172.21.15.79:54008) 2024-04-22T06:44:39.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 -- 172.21.15.79:0/3421209752 learned_addr learned my addr 172.21.15.79:0/3421209752 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4e191640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4e191640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3fdce0 0x55ff3e417e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 -- 172.21.15.79:0/3421209752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f47b0 msgr2=0x55ff3e3f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 -- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3fdce0 msgr2=0x55ff3e417e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3fdce0 0x55ff3e417e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 -- 172.21.15.79:0/3421209752 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff3e418650 con 0x55ff3e3f5160 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4e191640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e4d990640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 0x55ff3e3f8940 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3e38005ec0 tx=0x7f3e38004110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479322307b080389 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e317fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e317fa640 1 -- 172.21.15.79:0/3421209752 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3804c070 con 0x55ff3e3f5160 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e317fa640 1 -- 172.21.15.79:0/3421209752 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f3e3802cbf0 con 0x55ff3e3f5160 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e400055c0 con 0x55ff3e3f5160 2024-04-22T06:44:39.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e317fa640 1 -- 172.21.15.79:0/3421209752 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3e38057050 con 0x55ff3e3f5160 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 10 client.15663 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.915+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ff3e478e90 con 0x55ff3e3f5160 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e317fa640 1 -- 172.21.15.79:0/3421209752 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e400055c0 con 0x55ff3e3f5160 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e317fa640 1 -- 172.21.15.79:0/3421209752 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3e38004be0 con 0x55ff3e3f5160 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e527e9740 10 client.15663 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e527e9740 10 client.15663 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e527e9740 20 client.15663 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e527e9740 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e47da60 0x55ff3e47c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e527e9740 4 client.15663 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "status"}) -- 0x55ff3e3603a0 con 0x55ff3e47da60 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e4e191640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e47da60 0x55ff3e47c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e4e191640 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e47da60 0x55ff3e47c770 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d3a9ebe722fcb63 server_cookie=69d7d41109e9b4ba in_seq=0 out_seq=0 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.916+0000 7f3e317fa640 10 client.15663 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e317fa640 1 -- 172.21.15.79:0/3421209752 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ff3e3603a0 con 0x55ff3e47da60 2024-04-22T06:44:39.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e317fa640 10 client.15663 handle_command_reply: tid=0 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 client.15663 shutdown 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 msgr2=0x55ff3e3f8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 0x55ff3e3f8940 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3e38005ec0 tx=0x7f3e38004110 comp rx=0 tx=0).stop 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 shutdown_connections 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x55ff3e47da60 0x55ff3e47c770 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff3e3fdce0 0x55ff3e417e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff3e3f5160 0x55ff3e3f8940 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e317fa640 0 client.15663 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 --2- 172.21.15.79:0/3421209752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff3e3f47b0 0x55ff3e3f8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 >> 172.21.15.79:0/3421209752 conn(0x55ff3e3eea60 msgr2=0x55ff3e3eee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 shutdown_connections 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 1 -- 172.21.15.79:0/3421209752 wait complete. 2024-04-22T06:44:39.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e527e9740 20 client.15663 trim_cache size 0 max 16384 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 1, 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24440, 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 133.22148609199999, 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 22, 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 88, 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T06:44:39.923 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 152.89857746999999 2024-04-22T06:44:39.924 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:39.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e340a3e40 msgr2=0x7f3e340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e340a3e40 0x7f3e340a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e440cdd00 tx=0x7f3e4400df10 comp rx=0 tx=0).stop 2024-04-22T06:44:39.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 msgr2=0x7f3e4819e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.917+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 0x7f3e4819e920 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d770 tx=0x7f3e4400dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 shutdown_connections 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e48198590 0x7f3e48198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3e48106160 0x7f3e4819e920 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3e340a3e40 0x7f3e340a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 --2- 172.21.15.79:0/2280671730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e48101090 0x7f3e4819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 >> 172.21.15.79:0/2280671730 conn(0x7f3e48077f50 msgr2=0x7f3e48069580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 shutdown_connections 2024-04-22T06:44:39.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:39.918+0000 7f3e4fbf4640 1 -- 172.21.15.79:0/2280671730 wait complete. 2024-04-22T06:44:39.926 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:44:40.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.027+0000 7f5e2743f640 1 -- 172.21.15.79:0/3881652341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e20103af0 msgr2=0x7f5e2010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.027+0000 7f5e2743f640 1 --2- 172.21.15.79:0/3881652341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e20103af0 0x7f5e2010bfd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009900 tx=0x7f5e1002bd60 comp rx=0 tx=0).stop 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- 172.21.15.79:0/3881652341 shutdown_connections 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 --2- 172.21.15.79:0/3881652341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e2010c510 0x7f5e2010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 --2- 172.21.15.79:0/3881652341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e20103af0 0x7f5e2010bfd0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 --2- 172.21.15.79:0/3881652341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e201031d0 0x7f5e201035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- 172.21.15.79:0/3881652341 >> 172.21.15.79:0/3881652341 conn(0x7f5e20077f50 msgr2=0x7f5e20078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- 172.21.15.79:0/3881652341 shutdown_connections 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- 172.21.15.79:0/3881652341 wait complete. 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 Processor -- start 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- start start 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e201031d0 0x7f5e20100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e20103af0 0x7f5e20101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 0x7f5e201018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201121a0 con 0x7f5e2010c510 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e20112020 con 0x7f5e20103af0 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.028+0000 7f5e2743f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e20112320 con 0x7f5e201031d0 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 0x7f5e201018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 0x7f5e201018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54034/0 (socket says 172.21.15.79:54034) 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 -- 172.21.15.79:0/1606293506 learned_addr learned my addr 172.21.15.79:0/1606293506 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e249da640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e20103af0 0x7f5e20101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e251db640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e201031d0 0x7f5e20100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 -- 172.21.15.79:0/1606293506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e201031d0 msgr2=0x7f5e20100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e201031d0 0x7f5e20100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 -- 172.21.15.79:0/1606293506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e20103af0 msgr2=0x7f5e20101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e20103af0 0x7f5e20101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 -- 172.21.15.79:0/1606293506 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e201a6c40 con 0x7f5e2010c510 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e249da640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e20103af0 0x7f5e20101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e259dc640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 0x7f5e201018a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800eaa0 tx=0x7f5e1800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e167fc640 1 -- 172.21.15.79:0/1606293506 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1800cdb0 con 0x7f5e2010c510 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e167fc640 1 -- 172.21.15.79:0/1606293506 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e18005020 con 0x7f5e2010c510 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e201a6f30 con 0x7f5e2010c510 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e167fc640 1 -- 172.21.15.79:0/1606293506 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e180116d0 con 0x7f5e2010c510 2024-04-22T06:44:40.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.029+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e201a7450 con 0x7f5e2010c510 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.031+0000 7f5e167fc640 1 -- 172.21.15.79:0/1606293506 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5e180051d0 con 0x7f5e2010c510 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 Processor -- start 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 -- start start 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873e1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873e4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ef8733cb10 con 0x55ef873d87b0 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ef872da240 con 0x55ef873e1ce0 2024-04-22T06:44:40.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ef87338ad0 con 0x55ef873d9160 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e167fc640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5e0c0a3e40 0x7f5e0c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e249da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e249da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873d8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54044/0 (socket says 172.21.15.79:54044) 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e249da640 1 -- 172.21.15.79:0/1998627210 learned_addr learned my addr 172.21.15.79:0/1998627210 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e259dc640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5e0c0a3e40 0x7f5e0c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e251db640 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873e1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e167fc640 1 -- 172.21.15.79:0/1606293506 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f5e180d0c80 con 0x7f5e2010c510 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.032+0000 7f5e249da640 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e249da640 1 -- 172.21.15.79:0/1998627210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 msgr2=0x55ef873e1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e249da640 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873e1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e249da640 1 -- 172.21.15.79:0/1998627210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 msgr2=0x55ef873e4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e249da640 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873e4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e249da640 1 -- 172.21.15.79:0/1998627210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef8704f2d0 con 0x55ef873d87b0 2024-04-22T06:44:40.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e259dc640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5e0c0a3e40 0x7f5e0c0a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e180ce2e0 tx=0x7f5e18087040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e249da640 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873d8b90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e10005c40 tx=0x7f5e10005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd641014732af97a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e0b7fe640 1 -- 172.21.15.79:0/1998627210 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10048070 con 0x55ef873d87b0 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e0b7fe640 1 -- 172.21.15.79:0/1998627210 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1002cdc0 con 0x55ef873d87b0 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.033+0000 7f5e0b7fe640 1 -- 172.21.15.79:0/1998627210 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10004900 con 0x55ef873d87b0 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1998627210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 msgr2=0x55ef873d8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873d8b90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e10005c40 tx=0x7f5e10005bd0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1998627210 shutdown_connections 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873e4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873e1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1998627210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873d8b90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1998627210 >> 172.21.15.79:0/1998627210 conn(0x55ef873d2a60 msgr2=0x55ef873d2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1998627210 shutdown_connections 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1998627210 wait complete. 2024-04-22T06:44:40.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- start start 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 Processor -- start 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873ded90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d9160 0x55ef873df2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873df810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ef873e4bf0 con 0x55ef873d87b0 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ef872c5190 con 0x55ef873d9160 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.034+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ef873e4d70 con 0x55ef873e1ce0 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e2a02d740 10 client.0 _ll_register_callbacks cb 0x55ef872c08b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873ded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873ded90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54056/0 (socket says 172.21.15.79:54056) 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 -- 172.21.15.79:0/4210826773 learned_addr learned my addr 172.21.15.79:0/4210826773 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e251db640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d9160 0x55ef873df2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 -- 172.21.15.79:0/4210826773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 msgr2=0x55ef873df810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873df810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 -- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d9160 msgr2=0x55ef873df2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d9160 0x55ef873df2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 -- 172.21.15.79:0/4210826773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef871196b0 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e249da640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873ded90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5e10005770 tx=0x7f5e100055b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1f8769cd6924a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e08ff9640 1 -- 172.21.15.79:0/4210826773 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10048070 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e08ff9640 1 -- 172.21.15.79:0/4210826773 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f5e100037f0 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e1002cdc0 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e08ff9640 1 -- 172.21.15.79:0/4210826773 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5e100039f0 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e2a02d740 10 client.15675 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ef873f0fa0 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.035+0000 7f5e08ff9640 1 -- 172.21.15.79:0/4210826773 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1002cdc0 con 0x55ef873d87b0 2024-04-22T06:44:40.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e08ff9640 1 -- 172.21.15.79:0/4210826773 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5e10043ae0 con 0x55ef873d87b0 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e2a02d740 10 client.15675 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e2a02d740 10 client.15675 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e2a02d740 20 client.15675 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef873f22f0 0x55ef873f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e2a02d740 4 client.15675 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef8734a530 con 0x55ef873f22f0 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e251db640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef873f22f0 0x55ef873f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e08ff9640 10 client.15675 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.036+0000 7f5e251db640 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef873f22f0 0x55ef873f43f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8077c5686ccfe76 server_cookie=c96165520d039ec2 in_seq=0 out_seq=0 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e08ff9640 1 -- 172.21.15.79:0/4210826773 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef8734a530 con 0x55ef873f22f0 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e08ff9640 10 client.15675 handle_command_reply: tid=0 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 client.15675 shutdown 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 msgr2=0x55ef873ded90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873ded90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5e10005770 tx=0x7f5e100055b0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 shutdown_connections 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef873f22f0 0x55ef873f43f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d9160 0x55ef873df2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e08ff9640 0 client.15675 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/4210826773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873ded90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 >> 172.21.15.79:0/4210826773 conn(0x55ef873d2a60 msgr2=0x55ef873d4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 shutdown_connections 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 1 -- 172.21.15.79:0/4210826773 wait complete. 2024-04-22T06:44:40.044 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.037+0000 7f5e2a02d740 20 client.15675 trim_cache size 0 max 16384 2024-04-22T06:44:40.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.055+0000 7f5e2a02d740 1 Processor -- start 2024-04-22T06:44:40.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.055+0000 7f5e2a02d740 1 -- start start 2024-04-22T06:44:40.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d87b0 0x55ef873e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.059 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 0x55ef873ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873d2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ef873f2730 con 0x55ef873d9160 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ef873f2e00 con 0x55ef873d87b0 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ef873f34d0 con 0x55ef873e1ce0 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 0x55ef873ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 0x55ef873ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54060/0 (socket says 172.21.15.79:54060) 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 -- 172.21.15.79:0/544002867 learned_addr learned my addr 172.21.15.79:0/544002867 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 -- 172.21.15.79:0/544002867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 msgr2=0x55ef873d2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873d2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 -- 172.21.15.79:0/544002867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d87b0 msgr2=0x55ef873e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d87b0 0x55ef873e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 -- 172.21.15.79:0/544002867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef873f32b0 con 0x55ef873d9160 2024-04-22T06:44:40.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e251db640 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 0x55ef873ee6c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x55ef873d9560 tx=0x7f5e1c00b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e28b5cd0ccd167e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e0affd640 1 -- 172.21.15.79:0/544002867 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c01f020 con 0x55ef873d9160 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e0affd640 1 -- 172.21.15.79:0/544002867 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1c01bdc0 con 0x55ef873d9160 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.056+0000 7f5e0affd640 1 -- 172.21.15.79:0/544002867 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0035f0 con 0x55ef873d9160 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 -- 172.21.15.79:0/544002867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 msgr2=0x55ef873ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 0x55ef873ee6c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x55ef873d9560 tx=0x7f5e1c00b4d0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 -- 172.21.15.79:0/544002867 shutdown_connections 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873e1ce0 0x55ef873d2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d9160 0x55ef873ee6c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/544002867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873d87b0 0x55ef873e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 -- 172.21.15.79:0/544002867 >> 172.21.15.79:0/544002867 conn(0x55ef873d2a60 msgr2=0x55ef873de620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 -- 172.21.15.79:0/544002867 shutdown_connections 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 -- 172.21.15.79:0/544002867 wait complete. 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 -- start start 2024-04-22T06:44:40.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.057+0000 7f5e2a02d740 1 Processor -- start 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873dbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873dc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873dc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ef873f4270 con 0x55ef873d87b0 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ef873f3ba0 con 0x55ef873e1ce0 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ef873f4940 con 0x55ef873d9160 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 10 client.0 _ll_register_callbacks cb 0x7f5e10046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873dbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873dbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54074/0 (socket says 172.21.15.79:54074) 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 -- 172.21.15.79:0/1083262646 learned_addr learned my addr 172.21.15.79:0/1083262646 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e251db640 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873dc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 -- 172.21.15.79:0/1083262646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 msgr2=0x55ef873dc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873dc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 -- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 msgr2=0x55ef873dc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873dc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 -- 172.21.15.79:0/1083262646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef873fa900 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e249da640 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873dbc80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5e10001eb0 tx=0x7f5e1005a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7152ad401277a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e08ff9640 1 -- 172.21.15.79:0/1083262646 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10048070 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e08ff9640 1 -- 172.21.15.79:0/1083262646 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f5e100035f0 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e1c01bdc0 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e08ff9640 1 -- 172.21.15.79:0/1083262646 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5e1002c910 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 10 client.15681 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.058+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ef87459250 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e08ff9640 1 -- 172.21.15.79:0/1083262646 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1c01bdc0 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e08ff9640 1 -- 172.21.15.79:0/1083262646 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5e1005a920 con 0x55ef873d87b0 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e2a02d740 10 client.15681 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:40.063 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e2a02d740 10 client.15681 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e2a02d740 20 client.15681 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef8745ddd0 0x55ef8745cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e2a02d740 4 client.15681 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "status"}) -- 0x55ef873443a0 con 0x55ef8745ddd0 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e251db640 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef8745ddd0 0x55ef8745cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e08ff9640 10 client.15681 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.059+0000 7f5e251db640 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef8745ddd0 0x55ef8745cae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf21ebfe1b944450 server_cookie=cd005fe9f333da69 in_seq=0 out_seq=0 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e08ff9640 1 -- 172.21.15.79:0/1083262646 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ef873443a0 con 0x55ef8745ddd0 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e08ff9640 10 client.15681 handle_command_reply: tid=0 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 client.15681 shutdown 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 msgr2=0x55ef873dbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.064 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873dbc80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5e10001eb0 tx=0x7f5e1005a4c0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 shutdown_connections 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef8745ddd0 0x55ef8745cae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef873e1ce0 0x55ef873dc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e08ff9640 0 client.15681 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef873d9160 0x55ef873dc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 --2- 172.21.15.79:0/1083262646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef873d87b0 0x55ef873dbc80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 >> 172.21.15.79:0/1083262646 conn(0x55ef873d2a60 msgr2=0x55ef873d2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 shutdown_connections 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 1 -- 172.21.15.79:0/1083262646 wait complete. 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2a02d740 20 client.15681 trim_cache size 0 max 16384 2024-04-22T06:44:40.065 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "cluster_fsid": "146994a8-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "whoami": 0, 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "id": 24467, 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "want_state": "up:active", 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "state": "up:active", 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "fs_name": "cephfs", 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "rank_uptime": 135.38862030999999, 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "mdsmap_epoch": 22, 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch": 88, 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout: "uptime": 141.684595121 2024-04-22T06:44:40.066 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:40.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5e0c0a3e40 msgr2=0x7f5e0c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2743f640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5e0c0a3e40 0x7f5e0c0a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e180ce2e0 tx=0x7f5e18087040 comp rx=0 tx=0).stop 2024-04-22T06:44:40.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 msgr2=0x7f5e201018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.060+0000 7f5e2743f640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 0x7f5e201018a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800eaa0 tx=0x7f5e1800ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:40.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 shutdown_connections 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5e0c0a3e40 0x7f5e0c0a6300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e20103af0 0x7f5e20101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e2010c510 0x7f5e201018a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 --2- 172.21.15.79:0/1606293506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e201031d0 0x7f5e20100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 >> 172.21.15.79:0/1606293506 conn(0x7f5e20077f50 msgr2=0x7f5e200fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 shutdown_connections 2024-04-22T06:44:40.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:40.061+0000 7f5e2743f640 1 -- 172.21.15.79:0/1606293506 wait complete. 2024-04-22T06:44:40.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:40 smithi079 ceph-mon[27946]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:40.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:40 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2734030888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:40.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:40 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/291784799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:40.069 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-22T06:44:40.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:40 smithi193 ceph-mon[31815]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:40.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:40 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2734030888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:40.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:40 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/291784799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:40.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:40 smithi190 ceph-mon[31421]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:40.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:40 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2734030888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:40.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:40 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/291784799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:42.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:42 smithi193 ceph-mon[31815]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-22T06:44:42.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:42 smithi190 ceph-mon[31421]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-22T06:44:42.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:42 smithi079 ceph-mon[27946]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-22T06:44:44.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:44 smithi193 ceph-mon[31815]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 40 op/s 2024-04-22T06:44:44.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:44 smithi190 ceph-mon[31421]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 40 op/s 2024-04-22T06:44:44.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:44 smithi079 ceph-mon[27946]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 40 op/s 2024-04-22T06:44:46.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:46 smithi193 ceph-mon[31815]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-22T06:44:46.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:46 smithi190 ceph-mon[31421]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-22T06:44:46.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:46 smithi079 ceph-mon[27946]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-22T06:44:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.639+0000 7faed8175640 1 -- 172.21.15.79:0/2558886882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed0107270 msgr2=0x7faed010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.639+0000 7faed8175640 1 --2- 172.21.15.79:0/2558886882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed0107270 0x7faed010ee40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7faec8009ca0 tx=0x7faec802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 -- 172.21.15.79:0/2558886882 shutdown_connections 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 --2- 172.21.15.79:0/2558886882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed0102b50 0x7faed0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 --2- 172.21.15.79:0/2558886882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed0107270 0x7faed010ee40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 --2- 172.21.15.79:0/2558886882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed01021a0 0x7faed0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 -- 172.21.15.79:0/2558886882 >> 172.21.15.79:0/2558886882 conn(0x7faed00793e0 msgr2=0x7faed00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 -- 172.21.15.79:0/2558886882 shutdown_connections 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 -- 172.21.15.79:0/2558886882 wait complete. 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 Processor -- start 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 -- start start 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed01021a0 0x7faed0198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed0107270 0x7faed0198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.640+0000 7faed8175640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 0x7faed019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed8175640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faed01121e0 con 0x7faed019cf20 2024-04-22T06:44:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed8175640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faed0112060 con 0x7faed0107270 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed8175640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faed0112360 con 0x7faed01021a0 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 0x7faed019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 0x7faed019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49036/0 (socket says 172.21.15.79:49036) 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 -- 172.21.15.79:0/1385162862 learned_addr learned my addr 172.21.15.79:0/1385162862 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed5710640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed0107270 0x7faed0198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed5f11640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed01021a0 0x7faed0198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 -- 172.21.15.79:0/1385162862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed01021a0 msgr2=0x7faed0198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed01021a0 0x7faed0198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 -- 172.21.15.79:0/1385162862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed0107270 msgr2=0x7faed0198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed0107270 0x7faed0198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 -- 172.21.15.79:0/1385162862 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed019da10 con 0x7faed019cf20 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed6712640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 0x7faed019d3d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7faec8005e30 tx=0x7faec8032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faec6ffd640 1 -- 172.21.15.79:0/1385162862 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8041020 con 0x7faed019cf20 2024-04-22T06:44:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faec6ffd640 1 -- 172.21.15.79:0/1385162862 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec8032970 con 0x7faed019cf20 2024-04-22T06:44:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed019dca0 con 0x7faed019cf20 2024-04-22T06:44:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faec6ffd640 1 -- 172.21.15.79:0/1385162862 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8039040 con 0x7faed019cf20 2024-04-22T06:44:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.641+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed01a54c0 con 0x7faed019cf20 2024-04-22T06:44:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.644+0000 7faec6ffd640 1 -- 172.21.15.79:0/1385162862 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7faec8044020 con 0x7faed019cf20 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.644+0000 7faedad67740 1 Processor -- start 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 -- start start 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a00383620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00383fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562a002e88d0 con 0x562a003914f0 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562a00285e80 con 0x562a00383b60 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faedad67740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a002e4890 con 0x562a003848d0 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faec6ffd640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faebc0a3f60 0x7faebc0a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003938e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49038/0 (socket says 172.21.15.79:49038) 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 -- 172.21.15.79:0/350181178 learned_addr learned my addr 172.21.15.79:0/350181178 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a00383620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5f11640 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00383fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed6712640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faebc0a3f60 0x7faebc0a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faec6ffd640 1 -- 172.21.15.79:0/1385162862 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7faec80f2670 con 0x7faed019cf20 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 -- 172.21.15.79:0/350181178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 msgr2=0x562a00383620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a00383620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 -- 172.21.15.79:0/350181178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 msgr2=0x562a00383fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00383fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.645+0000 7faed5710640 1 -- 172.21.15.79:0/350181178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a002a7320 con 0x562a003914f0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.646+0000 7faed5710640 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003938e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7faec0012700 tx=0x7faec000b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21ae9baa16577e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.646+0000 7faed6712640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faebc0a3f60 0x7faebc0a6420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faec80efde0 tx=0x7faec809d040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.646+0000 7faebbfff640 1 -- 172.21.15.79:0/350181178 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0012c80 con 0x562a003914f0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.646+0000 7faebbfff640 1 -- 172.21.15.79:0/350181178 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec0012e20 con 0x562a003914f0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.646+0000 7faebbfff640 1 -- 172.21.15.79:0/350181178 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00035f0 con 0x562a003914f0 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 -- 172.21.15.79:0/350181178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 msgr2=0x562a003938e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003938e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7faec0012700 tx=0x7faec000b6e0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 -- 172.21.15.79:0/350181178 shutdown_connections 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00383fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003938e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 --2- 172.21.15.79:0/350181178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a00383620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 -- 172.21.15.79:0/350181178 >> 172.21.15.79:0/350181178 conn(0x562a0034ac90 msgr2=0x562a0037e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 -- 172.21.15.79:0/350181178 shutdown_connections 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 -- 172.21.15.79:0/350181178 wait complete. 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 -- start start 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 Processor -- start 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.647+0000 7faedad67740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a00390f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a0038ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562a00394220 con 0x562a003914f0 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562a00270eb0 con 0x562a003848d0 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a003943a0 con 0x562a00383b60 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 10 client.0 _ll_register_callbacks cb 0x562a0029db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a0038ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a0038ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49054/0 (socket says 172.21.15.79:49054) 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 -- 172.21.15.79:0/2292285107 learned_addr learned my addr 172.21.15.79:0/2292285107 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a00390f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5f11640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 -- 172.21.15.79:0/2292285107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 msgr2=0x562a00390f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a00390f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 -- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 msgr2=0x562a0038e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 -- 172.21.15.79:0/2292285107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629ffffb210 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5f11640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faed5710640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a0038ea00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faec0001eb0 tx=0x7faec001b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e12af30e131509 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faeb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faeb97fa640 1 -- 172.21.15.79:0/2292285107 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0005030 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faeb97fa640 1 -- 172.21.15.79:0/2292285107 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7faec00051d0 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec0012e20 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faeb97fa640 1 -- 172.21.15.79:0/2292285107 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faec0033050 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 10 client.15693 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.648+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562a000c7620 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faeb97fa640 1 -- 172.21.15.79:0/2292285107 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec0012e20 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faeb97fa640 1 -- 172.21.15.79:0/2292285107 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faec001f460 con 0x562a003914f0 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faedad67740 10 client.15693 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faedad67740 10 client.15693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faedad67740 20 client.15693 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faedad67740 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a003a1440 0x562a003a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faedad67740 4 client.15693 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a002fc880 con 0x562a003a1440 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faed5f11640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a003a1440 0x562a003a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faed5f11640 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a003a1440 0x562a003a3830 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703b25c1b56ca9cc server_cookie=548558e62b0ac1c7 in_seq=0 out_seq=0 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.649+0000 7faeb97fa640 10 client.15693 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faeb97fa640 1 -- 172.21.15.79:0/2292285107 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a002fc880 con 0x562a003a1440 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faeb97fa640 10 client.15693 handle_command_reply: tid=0 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 client.15693 shutdown 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 msgr2=0x562a0038ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a0038ea00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faec0001eb0 tx=0x7faec001b040 comp rx=0 tx=0).stop 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 shutdown_connections 2024-04-22T06:44:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a003a1440 0x562a003a3830 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a0038ea00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 --2- 172.21.15.79:0/2292285107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a00390f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faeb97fa640 0 client.15693 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.650+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 >> 172.21.15.79:0/2292285107 conn(0x562a0034ac90 msgr2=0x562a003805a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.651+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 shutdown_connections 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.651+0000 7faedad67740 1 -- 172.21.15.79:0/2292285107 wait complete. 2024-04-22T06:44:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.651+0000 7faedad67740 20 client.15693 trim_cache size 0 max 16384 2024-04-22T06:44:46.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.665+0000 7faedad67740 1 Processor -- start 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.665+0000 7faedad67740 1 -- start start 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faedad67740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a0039d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faedad67740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faedad67740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faedad67740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faed01ae0f0 con 0x562a003914f0 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faedad67740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faed01ab9c0 con 0x562a003848d0 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faedad67740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faed01a9830 con 0x562a00383b60 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003a8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49070/0 (socket says 172.21.15.79:49070) 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 -- 172.21.15.79:0/3921176657 learned_addr learned my addr 172.21.15.79:0/3921176657 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5f11640 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a0039d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 -- 172.21.15.79:0/3921176657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 msgr2=0x562a0039d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a0039d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 -- 172.21.15.79:0/3921176657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 msgr2=0x562a0038d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 -- 172.21.15.79:0/3921176657 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a00382e30 con 0x562a003914f0 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5f11640 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faed5710640 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003a8f50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7faec0031040 tx=0x7faec0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92eebd37a3d87acd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faebb7fe640 1 -- 172.21.15.79:0/3921176657 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec001fbd0 con 0x562a003914f0 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faebb7fe640 1 -- 172.21.15.79:0/3921176657 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec00120b0 con 0x562a003914f0 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.666+0000 7faebb7fe640 1 -- 172.21.15.79:0/3921176657 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec000b9f0 con 0x562a003914f0 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 -- 172.21.15.79:0/3921176657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 msgr2=0x562a003a8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003a8f50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7faec0031040 tx=0x7faec0014040 comp rx=0 tx=0).stop 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 -- 172.21.15.79:0/3921176657 shutdown_connections 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a003848d0 0x562a0038d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a003a8f50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 --2- 172.21.15.79:0/3921176657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a00383b60 0x562a0039d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 -- 172.21.15.79:0/3921176657 >> 172.21.15.79:0/3921176657 conn(0x562a0034ac90 msgr2=0x562a0038cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 -- 172.21.15.79:0/3921176657 shutdown_connections 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 -- 172.21.15.79:0/3921176657 wait complete. 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 -- start start 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.667+0000 7faedad67740 1 Processor -- start 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00392160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a003926a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a00398660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faed01a9e80 con 0x562a003914f0 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faed01ab2f0 con 0x562a00383b60 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faed01aa550 con 0x562a003848d0 2024-04-22T06:44:46.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faedad67740 10 client.0 _ll_register_callbacks cb 0x7faecc00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a00398660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a00398660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49086/0 (socket says 172.21.15.79:49086) 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 -- 172.21.15.79:0/2065983348 learned_addr learned my addr 172.21.15.79:0/2065983348 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00392160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5f11640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a003926a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 -- 172.21.15.79:0/2065983348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 msgr2=0x562a003926a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a003926a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 -- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 msgr2=0x562a00392160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00392160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.668+0000 7faed5710640 1 -- 172.21.15.79:0/2065983348 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a00398bd0 con 0x562a003914f0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faed5710640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a00398660 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faec000b560 tx=0x7faec0012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8486b7a73eeddec1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faeb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faeb97fa640 1 -- 172.21.15.79:0/2065983348 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec003c040 con 0x562a003914f0 2024-04-22T06:44:46.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faeb97fa640 1 -- 172.21.15.79:0/2065983348 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7faec0005310 con 0x562a003914f0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec00120b0 con 0x562a003914f0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faeb97fa640 1 -- 172.21.15.79:0/2065983348 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faec001f670 con 0x562a003914f0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 10 client.15699 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x562a00399ee0 con 0x562a003914f0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faeb97fa640 1 -- 172.21.15.79:0/2065983348 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec00120b0 con 0x562a003914f0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faeb97fa640 1 -- 172.21.15.79:0/2065983348 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7faec0003c30 con 0x562a003914f0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 10 client.15699 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 20 client.15699 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a00407b50 0x562a00409f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 4 client.15699 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a002883d0 con 0x562a00407b50 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.669+0000 7faed5f11640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a00407b50 0x562a00409f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.670+0000 7faeb97fa640 10 client.15699 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:46.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.670+0000 7faed5f11640 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a00407b50 0x562a00409f70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36c1b8e2f5d6cdf server_cookie=8a798819b7c8bd74 in_seq=0 out_seq=0 2024-04-22T06:44:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.680+0000 7faeb97fa640 1 -- 172.21.15.79:0/2065983348 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a002883d0 con 0x562a00407b50 2024-04-22T06:44:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.680+0000 7faeb97fa640 10 client.15699 handle_command_reply: tid=0 2024-04-22T06:44:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.680+0000 7faedad67740 1 client.15699 shutdown 2024-04-22T06:44:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.680+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 msgr2=0x562a00398660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.680+0000 7faedad67740 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a00398660 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faec000b560 tx=0x7faec0012040 comp rx=0 tx=0).stop 2024-04-22T06:44:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 shutdown_connections 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562a00407b50 0x562a00409f70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562a003914f0 0x562a00398660 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a003848d0 0x562a003926a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faeb97fa640 0 client.15699 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 --2- 172.21.15.79:0/2065983348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a00383b60 0x562a00392160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 >> 172.21.15.79:0/2065983348 conn(0x562a0034ac90 msgr2=0x562a0037d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 shutdown_connections 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 1 -- 172.21.15.79:0/2065983348 wait complete. 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faedad67740 20 client.15699 trim_cache size 0 max 16384 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:44:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:44:46.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:44:46.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:44:46.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faebc0a3f60 msgr2=0x7faebc0a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faebc0a3f60 0x7faebc0a6420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faec80efde0 tx=0x7faec809d040 comp rx=0 tx=0).stop 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 msgr2=0x7faed019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 0x7faed019d3d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7faec8005e30 tx=0x7faec8032420 comp rx=0 tx=0).stop 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 shutdown_connections 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faebc0a3f60 0x7faebc0a6420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed0107270 0x7faed0198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed019cf20 0x7faed019d3d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 --2- 172.21.15.79:0/1385162862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed01021a0 0x7faed0198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 >> 172.21.15.79:0/1385162862 conn(0x7faed00793e0 msgr2=0x7faed00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 shutdown_connections 2024-04-22T06:44:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:46.681+0000 7faed8175640 1 -- 172.21.15.79:0/1385162862 wait complete. 2024-04-22T06:44:48.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:48 smithi193 ceph-mon[31815]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-22T06:44:48.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:48 smithi190 ceph-mon[31421]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-22T06:44:48.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:48 smithi079 ceph-mon[27946]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-22T06:44:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:50 smithi193 ceph-mon[31815]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-22T06:44:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:50 smithi190 ceph-mon[31421]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-22T06:44:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:50 smithi079 ceph-mon[27946]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-22T06:44:52.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:52 smithi193 ceph-mon[31815]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-22T06:44:52.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:52 smithi190 ceph-mon[31421]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-22T06:44:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:52 smithi079 ceph-mon[27946]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-22T06:44:54.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:54 smithi193 ceph-mon[31815]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-22T06:44:54.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:54 smithi190 ceph-mon[31421]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-22T06:44:54.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:54 smithi079 ceph-mon[27946]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-22T06:44:55.070 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.2 2024-04-22T06:44:55.071 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:55.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 -- 172.21.15.79:0/3704589817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 msgr2=0x7fe93c101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 --2- 172.21.15.79:0/3704589817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c101130 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe938009900 tx=0x7fe93802bc50 comp rx=0 tx=0).stop 2024-04-22T06:44:55.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 -- 172.21.15.79:0/3704589817 shutdown_connections 2024-04-22T06:44:55.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 --2- 172.21.15.79:0/3704589817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe93c10e900 0x7fe93c110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 --2- 172.21.15.79:0/3704589817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c101670 0x7fe93c10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 --2- 172.21.15.79:0/3704589817 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c101130 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 -- 172.21.15.79:0/3704589817 >> 172.21.15.79:0/3704589817 conn(0x7fe93c0771d0 msgr2=0x7fe93c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 -- 172.21.15.79:0/3704589817 shutdown_connections 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 -- 172.21.15.79:0/3704589817 wait complete. 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 Processor -- start 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 -- start start 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.183+0000 7fe943c81640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe93c101670 0x7fe93c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c10e900 0x7fe93c19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c1144d0 con 0x7fe93c100d50 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c114350 con 0x7fe93c101670 2024-04-22T06:44:55.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c114650 con 0x7fe93c10e900 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c1a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43646/0 (socket says 172.21.15.79:43646) 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 -- 172.21.15.79:0/4167502007 learned_addr learned my addr 172.21.15.79:0/4167502007 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe94221e640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c10e900 0x7fe93c19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe94121c640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe93c101670 0x7fe93c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 -- 172.21.15.79:0/4167502007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c10e900 msgr2=0x7fe93c19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c10e900 0x7fe93c19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 -- 172.21.15.79:0/4167502007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe93c101670 msgr2=0x7fe93c1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe93c101670 0x7fe93c1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 -- 172.21.15.79:0/4167502007 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93c19afc0 con 0x7fe93c100d50 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe94221e640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c10e900 0x7fe93c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe941a1d640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c1a0680 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe938009230 tx=0x7fe93802c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938043020 con 0x7fe93c100d50 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe93802cc00 con 0x7fe93c100d50 2024-04-22T06:44:55.185 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93c19b250 con 0x7fe93c100d50 2024-04-22T06:44:55.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938003bf0 con 0x7fe93c100d50 2024-04-22T06:44:55.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.184+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe93c1a74d0 con 0x7fe93c100d50 2024-04-22T06:44:55.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.185+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe93c06ae60 con 0x7fe93c100d50 2024-04-22T06:44:55.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.187+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe938046020 con 0x7fe93c100d50 2024-04-22T06:44:55.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.187+0000 7fe932ffd640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9280a3e40 0x7fe9280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.187+0000 7fe94121c640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9280a3e40 0x7fe9280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.187+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fe9380f2fa0 con 0x7fe93c100d50 2024-04-22T06:44:55.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.188+0000 7fe94121c640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9280a3e40 0x7fe9280a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005fd0 tx=0x7fe92c005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.191+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe93804d020 con 0x7fe93c100d50 2024-04-22T06:44:55.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.406+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe93c109a90 con 0x7fe93c100d50 2024-04-22T06:44:55.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.406+0000 7fe932ffd640 1 -- 172.21.15.79:0/4167502007 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7fe9380473b0 con 0x7fe93c100d50 2024-04-22T06:44:55.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:44:55.405 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:44:55.405 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9280a3e40 msgr2=0x7fe9280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9280a3e40 0x7fe9280a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005fd0 tx=0x7fe92c005850 comp rx=0 tx=0).stop 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 msgr2=0x7fe93c1a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c1a0680 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe938009230 tx=0x7fe93802c800 comp rx=0 tx=0).stop 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 shutdown_connections 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9280a3e40 0x7fe9280a6300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe93c101670 0x7fe93c1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe93c10e900 0x7fe93c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.408+0000 7fe943c81640 1 --2- 172.21.15.79:0/4167502007 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe93c100d50 0x7fe93c1a0680 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.409+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 >> 172.21.15.79:0/4167502007 conn(0x7fe93c0771d0 msgr2=0x7fe93c10fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.409+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 shutdown_connections 2024-04-22T06:44:55.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.409+0000 7fe943c81640 1 -- 172.21.15.79:0/4167502007 wait complete. 2024-04-22T06:44:55.418 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 84.2 --await' 2024-04-22T06:44:55.418 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 84.2 --await 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.526+0000 7feac9b9f640 1 -- 172.21.15.79:0/474040122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 msgr2=0x7feac4069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.526+0000 7feac9b9f640 1 --2- 172.21.15.79:0/474040122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4069730 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feab4009900 tx=0x7feab402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- 172.21.15.79:0/474040122 shutdown_connections 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 --2- 172.21.15.79:0/474040122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac4069d00 0x7feac41069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 --2- 172.21.15.79:0/474040122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac41070f0 0x7feac410ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 --2- 172.21.15.79:0/474040122 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4069730 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- 172.21.15.79:0/474040122 >> 172.21.15.79:0/474040122 conn(0x7feac4076790 msgr2=0x7feac4076ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- 172.21.15.79:0/474040122 shutdown_connections 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- 172.21.15.79:0/474040122 wait complete. 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 Processor -- start 2024-04-22T06:44:55.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- start start 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41070f0 0x7feac4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac419cdd0 0x7feac419d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7feac4112030 con 0x7feac4069350 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feac4111eb0 con 0x7feac41070f0 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.527+0000 7feac9b9f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feac41121b0 con 0x7feac419cdd0 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43660/0 (socket says 172.21.15.79:43660) 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 -- 172.21.15.79:0/632033796 learned_addr learned my addr 172.21.15.79:0/632033796 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac939e640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac419cdd0 0x7feac419d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 -- 172.21.15.79:0/632033796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac419cdd0 msgr2=0x7feac419d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac419cdd0 0x7feac419d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac3fff640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41070f0 0x7feac4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 -- 172.21.15.79:0/632033796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41070f0 msgr2=0x7feac4198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41070f0 0x7feac4198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 -- 172.21.15.79:0/632033796 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac419d8c0 con 0x7feac4069350 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac939e640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac419cdd0 0x7feac419d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac3fff640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41070f0 0x7feac4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac8b9d640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4198510 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feab4005940 tx=0x7feab402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac1ffb640 1 -- 172.21.15.79:0/632033796 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4043020 con 0x7feac4069350 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac419db50 con 0x7feac4069350 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac1ffb640 1 -- 172.21.15.79:0/632033796 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab402cbe0 con 0x7feac4069350 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac1ffb640 1 -- 172.21.15.79:0/632033796 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4003bf0 con 0x7feac4069350 2024-04-22T06:44:55.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.528+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac41a5370 con 0x7feac4069350 2024-04-22T06:44:55.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac1ffb640 1 -- 172.21.15.79:0/632033796 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7feab4046020 con 0x7feac4069350 2024-04-22T06:44:55.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 Processor -- start 2024-04-22T06:44:55.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 -- start start 2024-04-22T06:44:55.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821cab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564a821220b0 con 0x564a821cb090 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564a820d3680 con 0x564a821c1b90 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feacd9ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564a820dfc60 con 0x564a821c24b0 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43676/0 (socket says 172.21.15.79:43676) 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 -- 172.21.15.79:0/1173975091 learned_addr learned my addr 172.21.15.79:0/1173975091 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac939e640 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac1ffb640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7feaac0a3e40 0x7feaac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac8b9d640 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac939e640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7feaac0a3e40 0x7feaac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 -- 172.21.15.79:0/1173975091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 msgr2=0x564a821cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 -- 172.21.15.79:0/1173975091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 msgr2=0x564a821c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac3fff640 1 -- 172.21.15.79:0/1173975091 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a81f04c90 con 0x564a821cb090 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.531+0000 7feac8b9d640 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feac1ffb640 1 -- 172.21.15.79:0/632033796 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7feab40f2c60 con 0x7feac4069350 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feac3fff640 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cd480 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feab800bb80 tx=0x7feab800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28977d84c105c2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feac939e640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7feaac0a3e40 0x7feaac0a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feabc005fd0 tx=0x7feabc004ab0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feaaaffd640 1 -- 172.21.15.79:0/1173975091 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8017620 con 0x564a821cb090 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feaaaffd640 1 -- 172.21.15.79:0/1173975091 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab8017c70 con 0x564a821cb090 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feaaaffd640 1 -- 172.21.15.79:0/1173975091 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab800fb30 con 0x564a821cb090 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feacd9ea740 1 -- 172.21.15.79:0/1173975091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 msgr2=0x564a821cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cd480 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feab800bb80 tx=0x7feab800d710 comp rx=0 tx=0).stop 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.532+0000 7feacd9ea740 1 -- 172.21.15.79:0/1173975091 shutdown_connections 2024-04-22T06:44:55.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cd480 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1173975091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- 172.21.15.79:0/1173975091 >> 172.21.15.79:0/1173975091 conn(0x564a821bc2c0 msgr2=0x564a821bc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- 172.21.15.79:0/1173975091 shutdown_connections 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- 172.21.15.79:0/1173975091 wait complete. 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- start start 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 Processor -- start 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 0x564a821c8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821cb090 0x564a821c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564a82125400 con 0x564a821c1b90 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564a820e1840 con 0x564a821c24b0 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564a8212a720 con 0x564a821cb090 2024-04-22T06:44:55.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feacd9ea740 10 client.0 _ll_register_callbacks cb 0x564a82132ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feac8b9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 0x564a821c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feac8b9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 0x564a821c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43692/0 (socket says 172.21.15.79:43692) 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.533+0000 7feac8b9d640 1 -- 172.21.15.79:0/800555735 learned_addr learned my addr 172.21.15.79:0/800555735 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac939e640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821cb090 0x564a821c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac8b9d640 1 -- 172.21.15.79:0/800555735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821cb090 msgr2=0x564a821c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac3fff640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac8b9d640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821cb090 0x564a821c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac8b9d640 1 -- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 msgr2=0x564a821c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac8b9d640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac8b9d640 1 -- 172.21.15.79:0/800555735 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a81f65420 con 0x564a821c1b90 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feac8b9d640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 0x564a821c8040 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feab40bea80 tx=0x7feab40be980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2073173bc628ce48 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feaa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feaa3fff640 1 -- 172.21.15.79:0/800555735 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab403c350 con 0x564a821c1b90 2024-04-22T06:44:55.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feaa3fff640 1 -- 172.21.15.79:0/800555735 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7feab40054d0 con 0x564a821c1b90 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feab8017c70 con 0x564a821c1b90 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feaa3fff640 1 -- 172.21.15.79:0/800555735 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7feab40ee050 con 0x564a821c1b90 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 10 client.15717 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564a81df8580 con 0x564a821c1b90 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feaa3fff640 1 -- 172.21.15.79:0/800555735 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab8017c70 con 0x564a821c1b90 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feaa3fff640 1 -- 172.21.15.79:0/800555735 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7feab40c72d0 con 0x564a821c1b90 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 10 client.15717 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 10 client.15717 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 20 client.15717 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a821dacd0 0x564a821dd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 4 client.15717 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.534+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a82133b10 con 0x564a821dacd0 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.535+0000 7feac3fff640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a821dacd0 0x564a821dd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.535+0000 7feac3fff640 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a821dacd0 0x564a821dd160 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59d975b0c016faed server_cookie=5dd4b76779586143 in_seq=0 out_seq=0 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.535+0000 7feaa3fff640 10 client.15717 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feaa3fff640 1 -- 172.21.15.79:0/800555735 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a82133b10 con 0x564a821dacd0 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feaa3fff640 10 client.15717 handle_command_reply: tid=0 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 client.15717 shutdown 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 msgr2=0x564a821c8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 0x564a821c8040 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feab40bea80 tx=0x7feab40be980 comp rx=0 tx=0).stop 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 shutdown_connections 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a821dacd0 0x564a821dd160 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821cb090 0x564a821c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 --2- 172.21.15.79:0/800555735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821c1b90 0x564a821c8040 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feaa3fff640 0 client.15717 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:55.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 >> 172.21.15.79:0/800555735 conn(0x564a821bc2c0 msgr2=0x564a821be010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 shutdown_connections 2024-04-22T06:44:55.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 1 -- 172.21.15.79:0/800555735 wait complete. 2024-04-22T06:44:55.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.536+0000 7feacd9ea740 20 client.15717 trim_cache size 0 max 16384 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 Processor -- start 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 -- start start 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c1b90 0x564a821ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821bb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7feac41ad9a0 con 0x564a821cb090 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feac41ab810 con 0x564a821c24b0 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.553+0000 7feacd9ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feac41ad2d0 con 0x564a821c1b90 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821bb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821bb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43694/0 (socket says 172.21.15.79:43694) 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 -- 172.21.15.79:0/2699145255 learned_addr learned my addr 172.21.15.79:0/2699145255 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac8b9d640 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c1b90 0x564a821ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 -- 172.21.15.79:0/2699145255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c1b90 msgr2=0x564a821ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c1b90 0x564a821ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 -- 172.21.15.79:0/2699145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 msgr2=0x564a821c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 -- 172.21.15.79:0/2699145255 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a821bc0f0 con 0x564a821cb090 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac8b9d640 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c1b90 0x564a821ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feac939e640 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821bb220 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feabc012910 tx=0x7feabc009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed0f34dd4e79fcf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feaaa7fc640 1 -- 172.21.15.79:0/2699145255 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc01a070 con 0x564a821cb090 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feaaa7fc640 1 -- 172.21.15.79:0/2699145255 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feabc026070 con 0x564a821cb090 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.554+0000 7feaaa7fc640 1 -- 172.21.15.79:0/2699145255 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc02bb70 con 0x564a821cb090 2024-04-22T06:44:55.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- 172.21.15.79:0/2699145255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 msgr2=0x564a821bb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821bb220 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feabc012910 tx=0x7feabc009880 comp rx=0 tx=0).stop 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- 172.21.15.79:0/2699145255 shutdown_connections 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c24b0 0x564a821c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821bb220 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- 172.21.15.79:0/2699145255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c1b90 0x564a821ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- 172.21.15.79:0/2699145255 >> 172.21.15.79:0/2699145255 conn(0x564a821bc2c0 msgr2=0x564a821bab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- 172.21.15.79:0/2699145255 shutdown_connections 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- 172.21.15.79:0/2699145255 wait complete. 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- start start 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 Processor -- start 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.555+0000 7feacd9ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7feac41ac530 con 0x564a821cb090 2024-04-22T06:44:55.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feacd9ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feac41abe60 con 0x564a821c1b90 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feacd9ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feac41acc00 con 0x564a821c24b0 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feacd9ea740 10 client.0 _ll_register_callbacks cb 0x7feab40057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43704/0 (socket says 172.21.15.79:43704) 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 -- 172.21.15.79:0/1072489623 learned_addr learned my addr 172.21.15.79:0/1072489623 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 -- 172.21.15.79:0/1072489623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 msgr2=0x564a821d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac3fff640 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 -- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 msgr2=0x564a821d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 -- 172.21.15.79:0/1072489623 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a821d0510 con 0x564a821cb090 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feac939e640 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cff50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7feabc004a80 tx=0x7feabc014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13054a03b467c652 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feaa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feaa3fff640 1 -- 172.21.15.79:0/1072489623 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc038060 con 0x564a821cb090 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feaa3fff640 1 -- 172.21.15.79:0/1072489623 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7feabc02bd80 con 0x564a821cb090 2024-04-22T06:44:55.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feabc026070 con 0x564a821cb090 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feaa3fff640 1 -- 172.21.15.79:0/1072489623 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7feabc01a040 con 0x564a821cb090 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feacd9ea740 10 client.15723 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.556+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564a82241a80 con 0x564a821cb090 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feaa3fff640 1 -- 172.21.15.79:0/1072489623 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feabc026070 con 0x564a821cb090 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feaa3fff640 1 -- 172.21.15.79:0/1072489623 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7feabc01d070 con 0x564a821cb090 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feacd9ea740 10 client.15723 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feacd9ea740 10 client.15723 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feacd9ea740 20 client.15723 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a82242160 0x564a82244580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feacd9ea740 4 client.15723 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.2, "await": true}] 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.2, "await": true}) -- 0x564a8212d980 con 0x564a82242160 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feac8b9d640 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a82242160 0x564a82244580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feaa3fff640 10 client.15723 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:55.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.557+0000 7feac8b9d640 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a82242160 0x564a82244580 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79efba0f6b837c4d server_cookie=88508172998559f7 in_seq=0 out_seq=0 2024-04-22T06:44:55.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feaa3fff640 1 -- 172.21.15.79:0/1072489623 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x564a8212d980 con 0x564a82242160 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feaa3fff640 10 client.15723 handle_command_reply: tid=0 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 client.15723 shutdown 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 msgr2=0x564a821cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cff50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7feabc004a80 tx=0x7feabc014e00 comp rx=0 tx=0).stop 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 shutdown_connections 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564a82242160 0x564a82244580 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564a821cb090 0x564a821cff50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feaa3fff640 0 client.15723 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a821c24b0 0x564a821d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 --2- 172.21.15.79:0/1072489623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564a821c1b90 0x564a821d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 >> 172.21.15.79:0/1072489623 conn(0x564a821bc2c0 msgr2=0x564a821bb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 shutdown_connections 2024-04-22T06:44:55.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 1 -- 172.21.15.79:0/1072489623 wait complete. 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.569+0000 7feacd9ea740 20 client.15723 trim_cache size 0 max 16384 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 22, 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 2, 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "c807895f": { 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 2, 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:44:55.569 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 84.2, 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:55.570 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:55.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:55.571 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:44:55.571 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:44:55.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7feaac0a3e40 msgr2=0x7feaac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7feaac0a3e40 0x7feaac0a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feabc005fd0 tx=0x7feabc004ab0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 msgr2=0x7feac4198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4198510 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7feab4005940 tx=0x7feab402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 shutdown_connections 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7feaac0a3e40 0x7feaac0a6300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feac41070f0 0x7feac4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feac419cdd0 0x7feac419d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 --2- 172.21.15.79:0/632033796 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac4069350 0x7feac4198510 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 >> 172.21.15.79:0/632033796 conn(0x7feac4076790 msgr2=0x7feac40ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 shutdown_connections 2024-04-22T06:44:55.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:44:55.570+0000 7feac9b9f640 1 -- 172.21.15.79:0/632033796 wait complete. 2024-04-22T06:44:55.579 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: c807895f, quiesce took 0.5 seconds. Will release after: 24.2 2024-04-22T06:44:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:56 smithi193 ceph-mon[31815]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-22T06:44:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:56 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4167502007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:56.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:56 smithi190 ceph-mon[31421]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-22T06:44:56.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:56 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4167502007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:56.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:56 smithi079 ceph-mon[27946]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-22T06:44:56.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:56 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4167502007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:44:58 smithi193 ceph-mon[31815]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-22T06:44:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:44:58 smithi190 ceph-mon[31421]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-22T06:44:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:44:58 smithi079 ceph-mon[27946]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-22T06:44:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 -- 172.21.15.79:0/2854756678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 msgr2=0x7fd52c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 --2- 172.21.15.79:0/2854756678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c101620 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd520009900 tx=0x7fd52002bc80 comp rx=0 tx=0).stop 2024-04-22T06:44:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 -- 172.21.15.79:0/2854756678 shutdown_connections 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 --2- 172.21.15.79:0/2854756678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c101b60 0x7fd52c10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 --2- 172.21.15.79:0/2854756678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c101620 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 --2- 172.21.15.79:0/2854756678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c106fb0 0x7fd52c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 -- 172.21.15.79:0/2854756678 >> 172.21.15.79:0/2854756678 conn(0x7fd52c076c50 msgr2=0x7fd52c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.777+0000 7fd532798640 1 -- 172.21.15.79:0/2854756678 shutdown_connections 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.778+0000 7fd532798640 1 -- 172.21.15.79:0/2854756678 wait complete. 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.778+0000 7fd532798640 1 Processor -- start 2024-04-22T06:44:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.778+0000 7fd532798640 1 -- start start 2024-04-22T06:44:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c101b60 0x7fd52c10ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c106fb0 0x7fd52c10f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1121e0 con 0x7fd52c1011a0 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112060 con 0x7fd52c106fb0 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112360 con 0x7fd52c101b60 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c10e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43716/0 (socket says 172.21.15.79:43716) 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 -- 172.21.15.79:0/337056539 learned_addr learned my addr 172.21.15.79:0/337056539 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c101b60 0x7fd52c10ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd530d35640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c106fb0 0x7fd52c10f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 -- 172.21.15.79:0/337056539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c101b60 msgr2=0x7fd52c10ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c101b60 0x7fd52c10ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 -- 172.21.15.79:0/337056539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c106fb0 msgr2=0x7fd52c10f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c106fb0 0x7fd52c10f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 -- 172.21.15.79:0/337056539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c1a6b00 con 0x7fd52c1011a0 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd530d35640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c106fb0 0x7fd52c10f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c101b60 0x7fd52c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd52bfff640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c10e700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c970 tx=0x7fd51c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd5297fa640 1 -- 172.21.15.79:0/337056539 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c007bb0 con 0x7fd52c1011a0 2024-04-22T06:44:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd5297fa640 1 -- 172.21.15.79:0/337056539 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd51c007d50 con 0x7fd52c1011a0 2024-04-22T06:44:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c1a6df0 con 0x7fd52c1011a0 2024-04-22T06:44:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd5297fa640 1 -- 172.21.15.79:0/337056539 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c00f450 con 0x7fd52c1011a0 2024-04-22T06:44:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.779+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1a7260 con 0x7fd52c1011a0 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd5297fa640 1 -- 172.21.15.79:0/337056539 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd51c004f20 con 0x7fd52c1011a0 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd535382740 1 Processor -- start 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd535382740 1 -- start start 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd5297fa640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd5140a3ee0 0x7fd5140a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd535382740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 0x55b5a41ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd535382740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.782+0000 7fd535382740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41d40e0 0x55b5a41d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd535382740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b5a412f8d0 con 0x55b5a41cb8d0 2024-04-22T06:44:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd535382740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5a40cce80 con 0x55b5a41cab60 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd535382740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5a412b890 con 0x55b5a41d40e0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd530d35640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd5140a3ee0 0x7fd5140a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd5297fa640 1 -- 172.21.15.79:0/337056539 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd51c0d0830 con 0x7fd52c1011a0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 0x55b5a41ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 0x55b5a41ca620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43720/0 (socket says 172.21.15.79:43720) 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 -- 172.21.15.79:0/213017516 learned_addr learned my addr 172.21.15.79:0/213017516 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 -- 172.21.15.79:0/213017516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41d40e0 msgr2=0x55b5a41d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41d40e0 0x55b5a41d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 -- 172.21.15.79:0/213017516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 msgr2=0x55b5a41cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 -- 172.21.15.79:0/213017516 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a40ee320 con 0x55b5a41cb8d0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd530d35640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd5140a3ee0 0x7fd5140a63a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd524005fd0 tx=0x7fd52400d710 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd52bfff640 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 0x55b5a41ca620 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c940 tx=0x7fd51c09c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae58a7efc3acc04 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd5127fc640 1 -- 172.21.15.79:0/213017516 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c01d070 con 0x55b5a41cb8d0 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd5127fc640 1 -- 172.21.15.79:0/213017516 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd51c0a0960 con 0x55b5a41cb8d0 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.783+0000 7fd5127fc640 1 -- 172.21.15.79:0/213017516 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0549e0 con 0x55b5a41cb8d0 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 -- 172.21.15.79:0/213017516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 msgr2=0x55b5a41ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 0x55b5a41ca620 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c940 tx=0x7fd51c09c200 comp rx=0 tx=0).stop 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 -- 172.21.15.79:0/213017516 shutdown_connections 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41cafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41d40e0 0x55b5a41d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 --2- 172.21.15.79:0/213017516 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cb8d0 0x55b5a41ca620 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 -- 172.21.15.79:0/213017516 >> 172.21.15.79:0/213017516 conn(0x55b5a4191c90 msgr2=0x55b5a41c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 -- 172.21.15.79:0/213017516 shutdown_connections 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 -- 172.21.15.79:0/213017516 wait complete. 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.784+0000 7fd535382740 1 -- start start 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 Processor -- start 2024-04-22T06:44:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 0x55b5a41d3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41d40e0 0x55b5a41d1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b5a41d6db0 con 0x55b5a41cab60 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b5a40b7eb0 con 0x55b5a41d40e0 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5a41d6f30 con 0x55b5a41cb8d0 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd535382740 10 client.0 _ll_register_callbacks cb 0x55b5a40e4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 0x55b5a41d3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 0x55b5a41d3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43730/0 (socket says 172.21.15.79:43730) 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 -- 172.21.15.79:0/117596909 learned_addr learned my addr 172.21.15.79:0/117596909 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd530d35640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41d40e0 0x55b5a41d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 -- 172.21.15.79:0/117596909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 msgr2=0x55b5a41d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 -- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41d40e0 msgr2=0x55b5a41d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41d40e0 0x55b5a41d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 -- 172.21.15.79:0/117596909 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a3e42210 con 0x55b5a41cab60 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd530d35640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41d40e0 0x55b5a41d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd52bfff640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 0x55b5a41d3ae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0a4cf0 tx=0x7fd51c004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff38474212bf78a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.785+0000 7fd50b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd50b7fe640 1 -- 172.21.15.79:0/117596909 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c01d070 con 0x55b5a41cab60 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd50b7fe640 1 -- 172.21.15.79:0/117596909 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd51c054990 con 0x55b5a41cab60 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd51c0a0960 con 0x55b5a41cab60 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd50b7fe640 1 -- 172.21.15.79:0/117596909 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd51c0ba050 con 0x55b5a41cab60 2024-04-22T06:44:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 10 client.15735 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b5a3f0e620 con 0x55b5a41cab60 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd50b7fe640 1 -- 172.21.15.79:0/117596909 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd51c0b4030 con 0x55b5a41cab60 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd50b7fe640 1 -- 172.21.15.79:0/117596909 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd51c09bb20 con 0x55b5a41cab60 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 10 client.15735 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 20 client.15735 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a41e3f60 0x55b5a41e6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 4 client.15735 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a4143880 con 0x55b5a41e3f60 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.786+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a41e3f60 0x55b5a41e6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.787+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a41e3f60 0x55b5a41e6350 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c63c09c4865dbc21 server_cookie=ada2ceed00f5d6ce in_seq=0 out_seq=0 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.787+0000 7fd50b7fe640 10 client.15735 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd50b7fe640 1 -- 172.21.15.79:0/117596909 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5a4143880 con 0x55b5a41e3f60 2024-04-22T06:44:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd50b7fe640 10 client.15735 handle_command_reply: tid=0 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 client.15735 shutdown 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 msgr2=0x55b5a41d3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 0x55b5a41d3ae0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0a4cf0 tx=0x7fd51c004980 comp rx=0 tx=0).stop 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 shutdown_connections 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a41e3f60 0x55b5a41e6350 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41d40e0 0x55b5a41d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd50b7fe640 0 client.15735 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 --2- 172.21.15.79:0/117596909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41cab60 0x55b5a41d3ae0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 >> 172.21.15.79:0/117596909 conn(0x55b5a4191c90 msgr2=0x55b5a41c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 shutdown_connections 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 1 -- 172.21.15.79:0/117596909 wait complete. 2024-04-22T06:44:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.788+0000 7fd535382740 20 client.15735 trim_cache size 0 max 16384 2024-04-22T06:44:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 Processor -- start 2024-04-22T06:44:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 -- start start 2024-04-22T06:44:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cab60 0x55b5a41d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b25c0 con 0x55b5a41d40e0 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1afe90 con 0x55b5a41cb8d0 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd535382740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1adce0 con 0x55b5a41cab60 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43738/0 (socket says 172.21.15.79:43738) 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 -- 172.21.15.79:0/3775344507 learned_addr learned my addr 172.21.15.79:0/3775344507 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd52bfff640 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cab60 0x55b5a41d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 -- 172.21.15.79:0/3775344507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cab60 msgr2=0x55b5a41d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cab60 0x55b5a41d35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 -- 172.21.15.79:0/3775344507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cb8d0 msgr2=0x55b5a41d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd530d35640 1 -- 172.21.15.79:0/3775344507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a41c9e30 con 0x55b5a41d40e0 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.803+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd530d35640 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cf740 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd524011570 tx=0x7fd524012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca70fbfc55b3323 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd511ffb640 1 -- 172.21.15.79:0/3775344507 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52400f9a0 con 0x55b5a41d40e0 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd511ffb640 1 -- 172.21.15.79:0/3775344507 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd524024070 con 0x55b5a41d40e0 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd511ffb640 1 -- 172.21.15.79:0/3775344507 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52401e3c0 con 0x55b5a41d40e0 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 -- 172.21.15.79:0/3775344507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 msgr2=0x55b5a41cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cf740 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd524011570 tx=0x7fd524012000 comp rx=0 tx=0).stop 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 -- 172.21.15.79:0/3775344507 shutdown_connections 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cb8d0 0x55b5a41d3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cf740 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 --2- 172.21.15.79:0/3775344507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cab60 0x55b5a41d35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 -- 172.21.15.79:0/3775344507 >> 172.21.15.79:0/3775344507 conn(0x55b5a4191c90 msgr2=0x55b5a41cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 -- 172.21.15.79:0/3775344507 shutdown_connections 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.804+0000 7fd535382740 1 -- 172.21.15.79:0/3775344507 wait complete. 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 -- start start 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 Processor -- start 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41e9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41e98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1ae350 con 0x55b5a41d40e0 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1af7c0 con 0x55b5a41cab60 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1aea20 con 0x55b5a41cb8d0 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd535382740 10 client.0 _ll_register_callbacks cb 0x7fd51c09bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd530d35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd530d35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43740/0 (socket says 172.21.15.79:43740) 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd530d35640 1 -- 172.21.15.79:0/2834134291 learned_addr learned my addr 172.21.15.79:0/2834134291 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:44:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd52b7fe640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41e98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.805+0000 7fd52bfff640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41e9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd530d35640 1 -- 172.21.15.79:0/2834134291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 msgr2=0x55b5a41e98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd530d35640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41e98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd530d35640 1 -- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 msgr2=0x55b5a41e9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd530d35640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41e9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd530d35640 1 -- 172.21.15.79:0/2834134291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a41cdef0 con 0x55b5a41d40e0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd530d35640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cd6c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd524005c10 tx=0x7fd524010a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1760cf294bfe7821 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd50b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd50b7fe640 1 -- 172.21.15.79:0/2834134291 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524034680 con 0x55b5a41d40e0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd50b7fe640 1 -- 172.21.15.79:0/2834134291 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd524034820 con 0x55b5a41d40e0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd524024070 con 0x55b5a41d40e0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd50b7fe640 1 -- 172.21.15.79:0/2834134291 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd524010e90 con 0x55b5a41d40e0 2024-04-22T06:44:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 10 client.15741 fetch_fsmap learned FSMap version 22 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b5a4249d40 con 0x55b5a41d40e0 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd50b7fe640 1 -- 172.21.15.79:0/2834134291 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd524024070 con 0x55b5a41d40e0 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd50b7fe640 1 -- 172.21.15.79:0/2834134291 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd524029770 con 0x55b5a41d40e0 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 10 client.15741 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 10 client.15741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 20 client.15741 populate_metadata read hostname 'smithi079' 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.806+0000 7fd535382740 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a424a460 0x55b5a424c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.807+0000 7fd535382740 4 client.15741 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.807+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5a40cf3d0 con 0x55b5a424a460 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.807+0000 7fd52bfff640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a424a460 0x55b5a424c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.807+0000 7fd52bfff640 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a424a460 0x55b5a424c880 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e249b1d91f532dfa server_cookie=4b3e4c92a1326da0 in_seq=0 out_seq=0 2024-04-22T06:44:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.807+0000 7fd50b7fe640 10 client.15741 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:59.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd50b7fe640 1 -- 172.21.15.79:0/2834134291 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5a40cf3d0 con 0x55b5a424a460 2024-04-22T06:44:59.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd50b7fe640 10 client.15741 handle_command_reply: tid=0 2024-04-22T06:44:59.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 client.15741 shutdown 2024-04-22T06:44:59.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 msgr2=0x55b5a41cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cd6c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd524005c10 tx=0x7fd524010a60 comp rx=0 tx=0).stop 2024-04-22T06:44:59.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 shutdown_connections 2024-04-22T06:44:59.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b5a424a460 0x55b5a424c880 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b5a41d40e0 0x55b5a41cd6c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a41cb8d0 0x55b5a41e98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd50b7fe640 0 client.15741 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 --2- 172.21.15.79:0/2834134291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5a41cab60 0x55b5a41e9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 >> 172.21.15.79:0/2834134291 conn(0x55b5a4191c90 msgr2=0x55b5a41cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 shutdown_connections 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 1 -- 172.21.15.79:0/2834134291 wait complete. 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.814+0000 7fd535382740 20 client.15741 trim_cache size 0 max 16384 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:44:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:44:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:44:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:44:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd5140a3ee0 msgr2=0x7fd5140a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd5140a3ee0 0x7fd5140a63a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd524005fd0 tx=0x7fd52400d710 comp rx=0 tx=0).stop 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 msgr2=0x7fd52c10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c10e700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00c970 tx=0x7fd51c00ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 shutdown_connections 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd5140a3ee0 0x7fd5140a63a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd52c106fb0 0x7fd52c10f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd52c101b60 0x7fd52c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 --2- 172.21.15.79:0/337056539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd52c1011a0 0x7fd52c10e700 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 >> 172.21.15.79:0/337056539 conn(0x7fd52c076c50 msgr2=0x7fd52c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 shutdown_connections 2024-04-22T06:44:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:44:59.815+0000 7fd532798640 1 -- 172.21.15.79:0/337056539 wait complete. 2024-04-22T06:45:00.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:00 smithi193 ceph-mon[31815]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-22T06:45:00.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:00 smithi190 ceph-mon[31421]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-22T06:45:00.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:00 smithi079 ceph-mon[27946]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-22T06:45:01.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:45:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:45:00.764+0000 7f9c2443c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:45:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:45:00.764+0000 7f9c2443c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:45:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:45:00.764+0000 7f9c2443c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:45:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:45:00.764+0000 7f9c2443c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:45:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:45:00.764+0000 7f9c2443c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:02.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:02.236 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:02.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 -- 172.21.15.79:0/744546836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50101a30 msgr2=0x7f8e50105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 --2- 172.21.15.79:0/744546836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50101a30 0x7f8e50105a70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009900 tx=0x7f8e4c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 -- 172.21.15.79:0/744546836 shutdown_connections 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 --2- 172.21.15.79:0/744546836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e50106150 0x7f8e5010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 --2- 172.21.15.79:0/744546836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50101a30 0x7f8e50105a70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 --2- 172.21.15.79:0/744546836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e50101080 0x7f8e50101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 -- 172.21.15.79:0/744546836 >> 172.21.15.79:0/744546836 conn(0x7f8e50078000 msgr2=0x7f8e50078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 -- 172.21.15.79:0/744546836 shutdown_connections 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 -- 172.21.15.79:0/744546836 wait complete. 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 Processor -- start 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.351+0000 7f8e576c1640 1 -- start start 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e576c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e50101080 0x7f8e50197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e576c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 0x7f8e50197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e576c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5019be50 0x7f8e5019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e576c1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8e50111090 con 0x7f8e50106150 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e576c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e50110f10 con 0x7f8e50101080 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e576c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8e50111210 con 0x7f8e5019be50 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 0x7f8e50197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 0x7f8e50197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43772/0 (socket says 172.21.15.79:43772) 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 -- 172.21.15.79:0/749513332 learned_addr learned my addr 172.21.15.79:0/749513332 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5019be50 0x7f8e5019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e5545d640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e50101080 0x7f8e50197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 -- 172.21.15.79:0/749513332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5019be50 msgr2=0x7f8e5019c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5019be50 0x7f8e5019c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 -- 172.21.15.79:0/749513332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e50101080 msgr2=0x7f8e50197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e50101080 0x7f8e50197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 -- 172.21.15.79:0/749513332 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5019c940 con 0x7f8e50106150 2024-04-22T06:45:02.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 0x7f8e50197ad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c036040 tx=0x7f8e4c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.352+0000 7f8e427fc640 1 -- 172.21.15.79:0/749513332 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c043020 con 0x7f8e50106150 2024-04-22T06:45:02.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.353+0000 7f8e427fc640 1 -- 172.21.15.79:0/749513332 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e4c02cb70 con 0x7f8e50106150 2024-04-22T06:45:02.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.353+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5019cbd0 con 0x7f8e50106150 2024-04-22T06:45:02.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.353+0000 7f8e427fc640 1 -- 172.21.15.79:0/749513332 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c003b80 con 0x7f8e50106150 2024-04-22T06:45:02.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.353+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e501a43f0 con 0x7f8e50106150 2024-04-22T06:45:02.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.355+0000 7f8e427fc640 1 -- 172.21.15.79:0/749513332 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f8e4c046020 con 0x7f8e50106150 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.355+0000 7f8e5a391740 1 Processor -- start 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 -- start start 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015270 0x55f689015650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 0x55f689020750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689020c90 0x55f689023080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f688f7a1e0 con 0x55f689015b90 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f688f17390 con 0x55f689020c90 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5a391740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f688f761a0 con 0x55f689015270 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e427fc640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8e3c0a3ed0 0x7f8e3c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 0x55f689020750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 0x55f689020750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43782/0 (socket says 172.21.15.79:43782) 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 -- 172.21.15.79:0/2178977201 learned_addr learned my addr 172.21.15.79:0/2178977201 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.361 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8e3c0a3ed0 0x7f8e3c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e427fc640 1 -- 172.21.15.79:0/749513332 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8e4c0f2aa0 con 0x7f8e50106150 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015270 0x55f689015650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689020c90 0x55f689023080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 -- 172.21.15.79:0/2178977201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015270 msgr2=0x55f689015650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015270 0x55f689015650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 -- 172.21.15.79:0/2178977201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689020c90 msgr2=0x55f689023080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689020c90 0x55f689023080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 -- 172.21.15.79:0/2178977201 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f688d567d0 con 0x55f689015b90 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689020c90 0x55f689023080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.356+0000 7f8e5545d640 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 0x55f689020750 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400ad00 tx=0x7f8e4400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f0f6a599df4cbb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8e3c0a3ed0 0x7f8e3c0a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8e48005fd0 tx=0x7f8e48009b60 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e3b7fe640 1 -- 172.21.15.79:0/2178977201 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4401d070 con 0x55f689015b90 2024-04-22T06:45:02.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e3b7fe640 1 -- 172.21.15.79:0/2178977201 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e44018af0 con 0x55f689015b90 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e3b7fe640 1 -- 172.21.15.79:0/2178977201 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44010b30 con 0x55f689015b90 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 -- 172.21.15.79:0/2178977201 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 msgr2=0x55f689020750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 0x55f689020750 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400ad00 tx=0x7f8e4400f790 comp rx=0 tx=0).stop 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 -- 172.21.15.79:0/2178977201 shutdown_connections 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689020c90 0x55f689023080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015b90 0x55f689020750 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2178977201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015270 0x55f689015650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 -- 172.21.15.79:0/2178977201 >> 172.21.15.79:0/2178977201 conn(0x55f68900fcb0 msgr2=0x55f689010090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 -- 172.21.15.79:0/2178977201 shutdown_connections 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.357+0000 7f8e5a391740 1 -- 172.21.15.79:0/2178977201 wait complete. 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 -- start start 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 Processor -- start 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 0x55f68901dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015b90 0x55f68901e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689020c90 0x55f68901e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f689023a80 con 0x55f689015270 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f688f332f0 con 0x55f689015b90 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f689023c00 con 0x55f689020c90 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5a391740 10 client.0 _ll_register_callbacks cb 0x55f688f40580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5545d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 0x55f68901dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5545d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 0x55f68901dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43798/0 (socket says 172.21.15.79:43798) 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e5545d640 1 -- 172.21.15.79:0/978159382 learned_addr learned my addr 172.21.15.79:0/978159382 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.358+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689020c90 0x55f68901e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5545d640 1 -- 172.21.15.79:0/978159382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689020c90 msgr2=0x55f68901e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5545d640 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689020c90 0x55f68901e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5545d640 1 -- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015b90 msgr2=0x55f68901e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5545d640 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015b90 0x55f68901e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5545d640 1 -- 172.21.15.79:0/978159382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f688c8c3f0 con 0x55f689015270 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5545d640 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 0x55f68901dc50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8e44010470 tx=0x7f8e440186d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff8e60e7bae4aea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e38ff9640 1 -- 172.21.15.79:0/978159382 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4401d040 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e38ff9640 1 -- 172.21.15.79:0/978159382 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8e44010870 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e44018af0 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e38ff9640 1 -- 172.21.15.79:0/978159382 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8e44031050 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 10 client.15759 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f688c4c0f0 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e38ff9640 1 -- 172.21.15.79:0/978159382 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e44018af0 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e38ff9640 1 -- 172.21.15.79:0/978159382 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8e44020030 con 0x55f689015270 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 10 client.15759 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 20 client.15759 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f689030b20 0x55f689032f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 4 client.15759 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:02.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.359+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f688f87c00 con 0x55f689030b20 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.360+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f689030b20 0x55f689032f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.360+0000 7f8e38ff9640 10 client.15759 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.360+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f689030b20 0x55f689032f10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb71c7d548298636 server_cookie=7b39b863a80c3f8d in_seq=0 out_seq=0 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e38ff9640 1 -- 172.21.15.79:0/978159382 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f688f87c00 con 0x55f689030b20 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e38ff9640 10 client.15759 handle_command_reply: tid=0 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 client.15759 shutdown 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 msgr2=0x55f68901dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 0x55f68901dc50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8e44010470 tx=0x7f8e440186d0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 shutdown_connections 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f689030b20 0x55f689032f10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015b90 0x55f68901e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689020c90 0x55f68901e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e38ff9640 0 client.15759 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 --2- 172.21.15.79:0/978159382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689015270 0x55f68901dc50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 >> 172.21.15.79:0/978159382 conn(0x55f68900fcb0 msgr2=0x55f689011a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 shutdown_connections 2024-04-22T06:45:02.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 1 -- 172.21.15.79:0/978159382 wait complete. 2024-04-22T06:45:02.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.361+0000 7f8e5a391740 20 client.15759 trim_cache size 0 max 16384 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 Processor -- start 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 -- start start 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f68902c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f68901d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f6890117a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501ae1a0 con 0x55f689020c90 2024-04-22T06:45:02.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501aba70 con 0x55f689015270 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5a391740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501a98c0 con 0x55f689015b90 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f6890117a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f6890117a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43814/0 (socket says 172.21.15.79:43814) 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 -- 172.21.15.79:0/2515423623 learned_addr learned my addr 172.21.15.79:0/2515423623 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f68901d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5545d640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f68902c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 -- 172.21.15.79:0/2515423623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 msgr2=0x55f68901d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f68901d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 -- 172.21.15.79:0/2515423623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 msgr2=0x55f68902c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f68902c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 -- 172.21.15.79:0/2515423623 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f689010880 con 0x55f689020c90 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e54c5c640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f68901d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e5545d640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f68902c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:02.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.379+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f6890117a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8e48003a90 tx=0x7f8e48009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92857bf8a908612b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e3affd640 1 -- 172.21.15.79:0/2515423623 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48025070 con 0x55f689020c90 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e3affd640 1 -- 172.21.15.79:0/2515423623 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e48004030 con 0x55f689020c90 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e3affd640 1 -- 172.21.15.79:0/2515423623 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48005260 con 0x55f689020c90 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 -- 172.21.15.79:0/2515423623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 msgr2=0x55f6890117a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f6890117a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8e48003a90 tx=0x7f8e48009500 comp rx=0 tx=0).stop 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 -- 172.21.15.79:0/2515423623 shutdown_connections 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f6890117a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f68901d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 --2- 172.21.15.79:0/2515423623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f68902c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 -- 172.21.15.79:0/2515423623 >> 172.21.15.79:0/2515423623 conn(0x55f68900fcb0 msgr2=0x55f689011090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 -- 172.21.15.79:0/2515423623 shutdown_connections 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 -- 172.21.15.79:0/2515423623 wait complete. 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.380+0000 7f8e5a391740 1 -- start start 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 Processor -- start 2024-04-22T06:45:02.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f689029180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f689024560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f689024aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501a9f30 con 0x55f689020c90 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501ab3a0 con 0x55f689015270 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501aa600 con 0x55f689015b90 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5a391740 10 client.0 _ll_register_callbacks cb 0x7f8e4401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f689024aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f689024aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43824/0 (socket says 172.21.15.79:43824) 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 -- 172.21.15.79:0/1149844966 learned_addr learned my addr 172.21.15.79:0/1149844966 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 -- 172.21.15.79:0/1149844966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 msgr2=0x55f689024560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f689024560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 -- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 msgr2=0x55f689029180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5545d640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f689029180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f689029180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 -- 172.21.15.79:0/1149844966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6890251e0 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e5545d640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f689029180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e55c5e640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f689024aa0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8e480049d0 tx=0x7f8e48021660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bda80af9d040392 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.381+0000 7f8e38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e38ff9640 1 -- 172.21.15.79:0/1149844966 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4802d070 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e38ff9640 1 -- 172.21.15.79:0/1149844966 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f8e48004c50 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e48004030 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e38ff9640 1 -- 172.21.15.79:0/1149844966 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8e48039050 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 10 client.15765 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f689096d30 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e38ff9640 1 -- 172.21.15.79:0/1149844966 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e48004030 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e38ff9640 1 -- 172.21.15.79:0/1149844966 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8e48005240 con 0x55f689020c90 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 10 client.15765 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:02.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 20 client.15765 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f6890973a0 0x55f6890997c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 4 client.15765 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f688f81a70 con 0x55f6890973a0 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.382+0000 7f8e5545d640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f6890973a0 0x55f6890997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e38ff9640 10 client.15765 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5545d640 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f6890973a0 0x55f6890997c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2701d90d389b1011 server_cookie=da33ca159364db06 in_seq=0 out_seq=0 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e38ff9640 1 -- 172.21.15.79:0/1149844966 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f688f81a70 con 0x55f6890973a0 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e38ff9640 10 client.15765 handle_command_reply: tid=0 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 client.15765 shutdown 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 msgr2=0x55f689024aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f689024aa0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8e480049d0 tx=0x7f8e48021660 comp rx=0 tx=0).stop 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 shutdown_connections 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f6890973a0 0x55f6890997c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f689020c90 0x55f689024aa0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e38ff9640 0 client.15765 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f689015b90 0x55f689024560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 --2- 172.21.15.79:0/1149844966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f689015270 0x55f689029180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 >> 172.21.15.79:0/1149844966 conn(0x55f68900fcb0 msgr2=0x55f689011920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 shutdown_connections 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 1 -- 172.21.15.79:0/1149844966 wait complete. 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.383+0000 7f8e5a391740 20 client.15765 trim_cache size 0 max 16384 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:45:02.389 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:02.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8e3c0a3ed0 msgr2=0x7f8e3c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8e3c0a3ed0 0x7f8e3c0a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8e48005fd0 tx=0x7f8e48009b60 comp rx=0 tx=0).stop 2024-04-22T06:45:02.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 msgr2=0x7f8e50197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 0x7f8e50197ad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c036040 tx=0x7f8e4c0045e0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 shutdown_connections 2024-04-22T06:45:02.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5019be50 0x7f8e5019c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8e50106150 0x7f8e50197ad0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8e3c0a3ed0 0x7f8e3c0a6390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 --2- 172.21.15.79:0/749513332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e50101080 0x7f8e50197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 >> 172.21.15.79:0/749513332 conn(0x7f8e50078000 msgr2=0x7f8e500fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 shutdown_connections 2024-04-22T06:45:02.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.384+0000 7f8e576c1640 1 -- 172.21.15.79:0/749513332 wait complete. 2024-04-22T06:45:02.392 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:02 smithi079 ceph-mon[27946]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 26 op/s 2024-04-22T06:45:02.392 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:02 smithi079 ceph-mon[27946]: mgrmap e25: x(active, since 36s), standbys: z, y 2024-04-22T06:45:02.393 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:45:02.393 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:02.433 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:02 smithi193 ceph-mon[31815]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 26 op/s 2024-04-22T06:45:02.434 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:02 smithi193 ceph-mon[31815]: mgrmap e25: x(active, since 36s), standbys: z, y 2024-04-22T06:45:02.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:02 smithi190 ceph-mon[31421]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 26 op/s 2024-04-22T06:45:02.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:02 smithi190 ceph-mon[31421]: mgrmap e25: x(active, since 36s), standbys: z, y 2024-04-22T06:45:02.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 -- 172.21.15.79:0/1732390799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc06b2a0 msgr2=0x7f61cc106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 --2- 172.21.15.79:0/1732390799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc06b2a0 0x7f61cc106bc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009900 tx=0x7f61c002bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 -- 172.21.15.79:0/1732390799 shutdown_connections 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 --2- 172.21.15.79:0/1732390799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61cc1072f0 0x7f61cc10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 --2- 172.21.15.79:0/1732390799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc06b2a0 0x7f61cc106bc0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 --2- 172.21.15.79:0/1732390799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61cc06a8f0 0x7f61cc06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 -- 172.21.15.79:0/1732390799 >> 172.21.15.79:0/1732390799 conn(0x7f61cc077e30 msgr2=0x7f61cc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 -- 172.21.15.79:0/1732390799 shutdown_connections 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.492+0000 7f61d2031640 1 -- 172.21.15.79:0/1732390799 wait complete. 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 Processor -- start 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 -- start start 2024-04-22T06:45:02.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61cc06a8f0 0x7f61cc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61cc1072f0 0x7f61cc198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 0x7f61cc19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc1121a0 con 0x7f61cc19d000 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc112020 con 0x7f61cc06a8f0 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61d2031640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc112320 con 0x7f61cc1072f0 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61cbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 0x7f61cc19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61cbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 0x7f61cc19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45688/0 (socket says 172.21.15.79:45688) 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.493+0000 7f61cbfff640 1 -- 172.21.15.79:0/1380293412 learned_addr learned my addr 172.21.15.79:0/1380293412 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cb7fe640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61cc06a8f0 0x7f61cc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cbfff640 1 -- 172.21.15.79:0/1380293412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61cc1072f0 msgr2=0x7f61cc198c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cbfff640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61cc1072f0 0x7f61cc198c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cbfff640 1 -- 172.21.15.79:0/1380293412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61cc06a8f0 msgr2=0x7f61cc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cbfff640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61cc06a8f0 0x7f61cc198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cbfff640 1 -- 172.21.15.79:0/1380293412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc19daf0 con 0x7f61cc19d000 2024-04-22T06:45:02.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61cbfff640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 0x7f61cc19d4b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f61c400eaa0 tx=0x7f61c400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c400cdb0 con 0x7f61cc19d000 2024-04-22T06:45:02.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61c4005020 con 0x7f61cc19d000 2024-04-22T06:45:02.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61cc19dde0 con 0x7f61cc19d000 2024-04-22T06:45:02.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c40116d0 con 0x7f61cc19d000 2024-04-22T06:45:02.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.494+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61cc1a5580 con 0x7f61cc19d000 2024-04-22T06:45:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.495+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61cc069230 con 0x7f61cc19d000 2024-04-22T06:45:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.496+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f61c40051d0 con 0x7f61cc19d000 2024-04-22T06:45:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.496+0000 7f61c8ff9640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f61b40a3ed0 0x7f61b40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.497+0000 7f61cb7fe640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f61b40a3ed0 0x7f61b40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.497+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f61c40d0c50 con 0x7f61cc19d000 2024-04-22T06:45:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.497+0000 7f61cb7fe640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f61b40a3ed0 0x7f61b40a6390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009bd0 tx=0x7f61bc005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.501+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61c4016030 con 0x7f61cc19d000 2024-04-22T06:45:02.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.716+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61cc06b2a0 con 0x7f61cc19d000 2024-04-22T06:45:02.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.717+0000 7f61c8ff9640 1 -- 172.21.15.79:0/1380293412 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7f61c409d0e0 con 0x7f61cc19d000 2024-04-22T06:45:02.715 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:45:02.715 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:45:02.716 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:45:02.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f61b40a3ed0 msgr2=0x7f61b40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f61b40a3ed0 0x7f61b40a6390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009bd0 tx=0x7f61bc005f90 comp rx=0 tx=0).stop 2024-04-22T06:45:02.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 msgr2=0x7f61cc19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 0x7f61cc19d4b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f61c400eaa0 tx=0x7f61c400ef60 comp rx=0 tx=0).stop 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 shutdown_connections 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f61cc19d000 0x7f61cc19d4b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61cc1072f0 0x7f61cc198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f61b40a3ed0 0x7f61b40a6390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 --2- 172.21.15.79:0/1380293412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61cc06a8f0 0x7f61cc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 >> 172.21.15.79:0/1380293412 conn(0x7f61cc077e30 msgr2=0x7f61cc0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 shutdown_connections 2024-04-22T06:45:02.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.719+0000 7f61d2031640 1 -- 172.21.15.79:0/1380293412 wait complete. 2024-04-22T06:45:02.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:45:02.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 -- 172.21.15.79:0/2344191286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a8074300 msgr2=0x7f64a80746e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 --2- 172.21.15.79:0/2344191286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a8074300 0x7f64a80746e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009900 tx=0x7f64a402bcd0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 -- 172.21.15.79:0/2344191286 shutdown_connections 2024-04-22T06:45:02.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 --2- 172.21.15.79:0/2344191286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a810e7f0 0x7f64a8110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 --2- 172.21.15.79:0/2344191286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64a8074c20 0x7f64a810e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 --2- 172.21.15.79:0/2344191286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a8074300 0x7f64a80746e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 -- 172.21.15.79:0/2344191286 >> 172.21.15.79:0/2344191286 conn(0x7f64a80fd9c0 msgr2=0x7f64a80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 -- 172.21.15.79:0/2344191286 shutdown_connections 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.830+0000 7f64b05ce640 1 -- 172.21.15.79:0/2344191286 wait complete. 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 Processor -- start 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 -- start start 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64a8074300 0x7f64a81a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a8074c20 0x7f64a81a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 0x7f64a819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81143a0 con 0x7f64a810e7f0 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8114220 con 0x7f64a8074c20 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64b05ce640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8114520 con 0x7f64a8074300 2024-04-22T06:45:02.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 0x7f64a819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 0x7f64a819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45714/0 (socket says 172.21.15.79:45714) 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3555335336 learned_addr learned my addr 172.21.15.79:0/3555335336 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64ae36a640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64a8074300 0x7f64a81a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64adb69640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a8074c20 0x7f64a81a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3555335336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64a8074300 msgr2=0x7f64a81a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64a8074300 0x7f64a81a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3555335336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a8074c20 msgr2=0x7f64a81a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a8074c20 0x7f64a81a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3555335336 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a819b000 con 0x7f64a810e7f0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.831+0000 7f64adb69640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a8074c20 0x7f64a81a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.832+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 0x7f64a819a740 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f64a000cce0 tx=0x7f64a000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.832+0000 7f649f7fe640 1 -- 172.21.15.79:0/3555335336 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a000f450 con 0x7f64a810e7f0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.832+0000 7f649f7fe640 1 -- 172.21.15.79:0/3555335336 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a000faa0 con 0x7f64a810e7f0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.832+0000 7f649f7fe640 1 -- 172.21.15.79:0/3555335336 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0018b70 con 0x7f64a810e7f0 2024-04-22T06:45:02.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.832+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a819b2f0 con 0x7f64a810e7f0 2024-04-22T06:45:02.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.832+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a81a75b0 con 0x7f64a810e7f0 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.834+0000 7f649f7fe640 1 -- 172.21.15.79:0/3555335336 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f64a0018d10 con 0x7f64a810e7f0 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.834+0000 7f64b31bf740 1 Processor -- start 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 -- start start 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22bc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f649f7fe640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64940a3ed0 0x7f64940a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 0x5571b22bee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5571b22181f0 con 0x5571b22bca00 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5571b21b54b0 con 0x5571b22b34d0 2024-04-22T06:45:02.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64b31bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571b22141b0 con 0x5571b22b3e80 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64ae36a640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64940a3ed0 0x7f64940a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 0x5571b22bee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 0x5571b22bee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45724/0 (socket says 172.21.15.79:45724) 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3795665572 learned_addr learned my addr 172.21.15.79:0/3795665572 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f649f7fe640 1 -- 172.21.15.79:0/3555335336 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f64a00d0ec0 con 0x7f64a810e7f0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64adb69640 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64adb69640 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22bc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3795665572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 msgr2=0x5571b22bc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22bc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3795665572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 msgr2=0x5571b22b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64aeb6b640 1 -- 172.21.15.79:0/3795665572 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b21a71b0 con 0x5571b22bca00 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.835+0000 7f64ae36a640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64940a3ed0 0x7f64940a6390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64a4006010 tx=0x7f64a4005e40 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 0x5571b22bee50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f64a009cbe0 tx=0x7f64a0013810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a54795876851265 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f649cbfe640 1 -- 172.21.15.79:0/3795665572 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a00aa6b0 con 0x5571b22bca00 2024-04-22T06:45:02.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f649cbfe640 1 -- 172.21.15.79:0/3795665572 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a00a6780 con 0x5571b22bca00 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f649cbfe640 1 -- 172.21.15.79:0/3795665572 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0087910 con 0x5571b22bca00 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f64b31bf740 1 -- 172.21.15.79:0/3795665572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 msgr2=0x5571b22bee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f64b31bf740 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 0x5571b22bee50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f64a009cbe0 tx=0x7f64a0013810 comp rx=0 tx=0).stop 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f64b31bf740 1 -- 172.21.15.79:0/3795665572 shutdown_connections 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f64b31bf740 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22bca00 0x5571b22bee50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.836+0000 7f64b31bf740 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22bc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 --2- 172.21.15.79:0/3795665572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- 172.21.15.79:0/3795665572 >> 172.21.15.79:0/3795665572 conn(0x5571b22adba0 msgr2=0x5571b22adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- 172.21.15.79:0/3795665572 shutdown_connections 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- 172.21.15.79:0/3795665572 wait complete. 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- start start 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 Processor -- start 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 0x5571b22bc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22b9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22bca00 0x5571b22b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5571b22bf8b0 con 0x5571b22b34d0 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5571b21d1500 con 0x5571b22bca00 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571b22bfa30 con 0x5571b22b3e80 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64ae36a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 0x5571b22bc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64b31bf740 10 client.0 _ll_register_callbacks cb 0x5571b219fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64ae36a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 0x5571b22bc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45738/0 (socket says 172.21.15.79:45738) 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64ae36a640 1 -- 172.21.15.79:0/352179913 learned_addr learned my addr 172.21.15.79:0/352179913 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64adb69640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.837+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22bca00 0x5571b22b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64ae36a640 1 -- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 msgr2=0x5571b22b9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64ae36a640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22b9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64ae36a640 1 -- 172.21.15.79:0/352179913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22bca00 msgr2=0x5571b22b9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64ae36a640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22bca00 0x5571b22b9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64ae36a640 1 -- 172.21.15.79:0/352179913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b1f2a3f0 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64ae36a640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 0x5571b22bc300 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009900 tx=0x7f64a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60230eb9bad5e0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f6491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f6491ffb640 1 -- 172.21.15.79:0/352179913 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4047040 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f6491ffb640 1 -- 172.21.15.79:0/352179913 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f64a4005520 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a00a6780 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f6491ffb640 1 -- 172.21.15.79:0/352179913 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f64a40046c0 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 10 client.15783 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5571b1ff47d0 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f6491ffb640 1 -- 172.21.15.79:0/352179913 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a00a6780 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f6491ffb640 1 -- 172.21.15.79:0/352179913 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f64a4003c70 con 0x5571b22b34d0 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 10 client.15783 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 10 client.15783 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 20 client.15783 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b22ccb60 0x5571b22cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 4 client.15783 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.838+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571b21b7930 con 0x5571b22ccb60 2024-04-22T06:45:02.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.839+0000 7f64adb69640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b22ccb60 0x5571b22cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.839+0000 7f6491ffb640 10 client.15783 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.839+0000 7f64adb69640 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b22ccb60 0x5571b22cefc0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5cd9b7edc00f5aa3 server_cookie=d15f85c7e7156b11 in_seq=0 out_seq=0 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f6491ffb640 1 -- 172.21.15.79:0/352179913 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571b21b7930 con 0x5571b22ccb60 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f6491ffb640 10 client.15783 handle_command_reply: tid=0 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 client.15783 shutdown 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 msgr2=0x5571b22bc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 0x5571b22bc300 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009900 tx=0x7f64a4009260 comp rx=0 tx=0).stop 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 shutdown_connections 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b22ccb60 0x5571b22cefc0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22bca00 0x5571b22b9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22b3e80 0x5571b22b9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f6491ffb640 0 client.15783 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:45:02.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 --2- 172.21.15.79:0/352179913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b34d0 0x5571b22bc300 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 >> 172.21.15.79:0/352179913 conn(0x5571b22adba0 msgr2=0x5571b22af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 shutdown_connections 2024-04-22T06:45:02.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 1 -- 172.21.15.79:0/352179913 wait complete. 2024-04-22T06:45:02.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.840+0000 7f64b31bf740 20 client.15783 trim_cache size 0 max 16384 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 Processor -- start 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 -- start start 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22aca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22bbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81b25b0 con 0x5571b22b3e80 2024-04-22T06:45:02.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81afe80 con 0x5571b22b34d0 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64b31bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81adcd0 con 0x5571b22bca00 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22aca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45752/0 (socket says 172.21.15.79:45752) 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 -- 172.21.15.79:0/1739707832 learned_addr learned my addr 172.21.15.79:0/1739707832 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 -- 172.21.15.79:0/1739707832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 msgr2=0x5571b22bbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22bbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 -- 172.21.15.79:0/1739707832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 msgr2=0x5571b22c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.853+0000 7f64adb69640 1 -- 172.21.15.79:0/1739707832 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b22b3090 con 0x5571b22b3e80 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64adb69640 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22aca80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x5571b22b2fa0 tx=0x7f649800b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4aeac9e5cd66bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f6493fff640 1 -- 172.21.15.79:0/1739707832 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498022070 con 0x5571b22b3e80 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f6493fff640 1 -- 172.21.15.79:0/1739707832 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f649801e870 con 0x5571b22b3e80 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f6493fff640 1 -- 172.21.15.79:0/1739707832 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64980047a0 con 0x5571b22b3e80 2024-04-22T06:45:02.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 -- 172.21.15.79:0/1739707832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 msgr2=0x5571b22aca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22aca80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x5571b22b2fa0 tx=0x7f649800b3a0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 -- 172.21.15.79:0/1739707832 shutdown_connections 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22bbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22aca80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 --2- 172.21.15.79:0/1739707832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 -- 172.21.15.79:0/1739707832 >> 172.21.15.79:0/1739707832 conn(0x5571b22adba0 msgr2=0x5571b22ac370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 -- 172.21.15.79:0/1739707832 shutdown_connections 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.854+0000 7f64b31bf740 1 -- 172.21.15.79:0/1739707832 wait complete. 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 -- start start 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 Processor -- start 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ae340 con 0x5571b22b3e80 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81af7b0 con 0x5571b22b34d0 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81aea10 con 0x5571b22bca00 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64b31bf740 10 client.0 _ll_register_callbacks cb 0x7f64a4005110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45754/0 (socket says 172.21.15.79:45754) 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 -- 172.21.15.79:0/2888597934 learned_addr learned my addr 172.21.15.79:0/2888597934 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64ae36a640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 -- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 msgr2=0x5571b22b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 -- 172.21.15.79:0/2888597934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 msgr2=0x5571b22c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 -- 172.21.15.79:0/2888597934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b22b6a10 con 0x5571b22b3e80 2024-04-22T06:45:02.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64aeb6b640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f64adb69640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22b5ee0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x5571b22b2fa0 tx=0x7f6498005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0b4359b2bbd98 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f6491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f6491ffb640 1 -- 172.21.15.79:0/2888597934 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498022040 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f649801e870 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.855+0000 7f6491ffb640 1 -- 172.21.15.79:0/2888597934 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f649801e5c0 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f6491ffb640 1 -- 172.21.15.79:0/2888597934 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6498005610 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 10 client.15789 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5571b2332f50 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f6491ffb640 1 -- 172.21.15.79:0/2888597934 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f649801e870 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f6491ffb640 1 -- 172.21.15.79:0/2888597934 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f649800f040 con 0x5571b22b3e80 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 10 client.15789 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 10 client.15789 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 20 client.15789 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:02.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b23335f0 0x5571b2335a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 4 client.15789 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 --> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5571b2225c10 con 0x5571b23335f0 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.856+0000 7f64ae36a640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b23335f0 0x5571b2335a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f6491ffb640 10 client.15789 ms_handle_connect on v2:172.21.15.193:6838/746968688 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64ae36a640 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b23335f0 0x5571b2335a10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=227af2b6e1be32eb server_cookie=b9d3c6c515f08c8 in_seq=0 out_seq=0 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f6491ffb640 1 -- 172.21.15.79:0/2888597934 <== mds.2 v2:172.21.15.193:6838/746968688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5571b2225c10 con 0x5571b23335f0 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f6491ffb640 10 client.15789 handle_command_reply: tid=0 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 client.15789 shutdown 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 msgr2=0x5571b22b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22b5ee0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x5571b22b2fa0 tx=0x7f6498005f90 comp rx=0 tx=0).stop 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 shutdown_connections 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571b22bca00 0x5571b22b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5571b22b3e80 0x5571b22b5ee0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.193:6838/746968688,v1:172.21.15.193:6839/746968688] conn(0x5571b23335f0 0x5571b2335a10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 --2- 172.21.15.79:0/2888597934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5571b22b34d0 0x5571b22c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f6491ffb640 0 client.15789 ms_handle_reset on v2:172.21.15.193:6838/746968688 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 >> 172.21.15.79:0/2888597934 conn(0x5571b22adba0 msgr2=0x5571b22acc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 shutdown_connections 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 1 -- 172.21.15.79:0/2888597934 wait complete. 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b31bf740 20 client.15789 trim_cache size 0 max 16384 2024-04-22T06:45:02.863 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64940a3ed0 msgr2=0x7f64940a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64940a3ed0 0x7f64940a6390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64a4006010 tx=0x7f64a4005e40 comp rx=0 tx=0).stop 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 msgr2=0x7f64a819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 0x7f64a819a740 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f64a000cce0 tx=0x7f64a000a620 comp rx=0 tx=0).stop 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 shutdown_connections 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64940a3ed0 0x7f64940a6390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64a8074c20 0x7f64a81a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64a810e7f0 0x7f64a819a740 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.857+0000 7f64b05ce640 1 --2- 172.21.15.79:0/3555335336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64a8074300 0x7f64a81a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.858+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 >> 172.21.15.79:0/3555335336 conn(0x7f64a80fd9c0 msgr2=0x7f64a80fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.858+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 shutdown_connections 2024-04-22T06:45:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.858+0000 7f64b05ce640 1 -- 172.21.15.79:0/3555335336 wait complete. 2024-04-22T06:45:02.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.969+0000 7fcc1c354640 1 -- 172.21.15.79:0/1971966575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc1410e900 msgr2=0x7fcc14110cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.969+0000 7fcc1c354640 1 --2- 172.21.15.79:0/1971966575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc1410e900 0x7fcc14110cf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009900 tx=0x7fcc0c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- 172.21.15.79:0/1971966575 shutdown_connections 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 --2- 172.21.15.79:0/1971966575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc14101670 0x7fcc1410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 --2- 172.21.15.79:0/1971966575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc1410e900 0x7fcc14110cf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 --2- 172.21.15.79:0/1971966575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 0x7fcc14101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- 172.21.15.79:0/1971966575 >> 172.21.15.79:0/1971966575 conn(0x7fcc14077f50 msgr2=0x7fcc14078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- 172.21.15.79:0/1971966575 shutdown_connections 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- 172.21.15.79:0/1971966575 wait complete. 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 Processor -- start 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- start start 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 0x7fcc141a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 0x7fcc141a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc1410e900 0x7fcc1419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141143c0 con 0x7fcc14101670 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14114240 con 0x7fcc1410e900 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc1c354640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14114540 con 0x7fcc14100d50 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc198ef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 0x7fcc141a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc198ef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 0x7fcc141a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45772/0 (socket says 172.21.15.79:45772) 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.970+0000 7fcc198ef640 1 -- 172.21.15.79:0/3708874398 learned_addr learned my addr 172.21.15.79:0/3708874398 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 0x7fcc141a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc1410e900 0x7fcc1419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc198ef640 1 -- 172.21.15.79:0/3708874398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 msgr2=0x7fcc141a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc198ef640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 0x7fcc141a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc198ef640 1 -- 172.21.15.79:0/3708874398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc1410e900 msgr2=0x7fcc1419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc198ef640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc1410e900 0x7fcc1419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc198ef640 1 -- 172.21.15.79:0/3708874398 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1419b000 con 0x7fcc14101670 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 0x7fcc141a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc198ef640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 0x7fcc141a0bc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400d950 tx=0x7fcc0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc0b7fe640 1 -- 172.21.15.79:0/3708874398 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04017030 con 0x7fcc14101670 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc0b7fe640 1 -- 172.21.15.79:0/3708874398 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc04004020 con 0x7fcc14101670 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc0b7fe640 1 -- 172.21.15.79:0/3708874398 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc040047a0 con 0x7fcc14101670 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1419b2f0 con 0x7fcc14101670 2024-04-22T06:45:02.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.971+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc141a7600 con 0x7fcc14101670 2024-04-22T06:45:02.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.973+0000 7fcc0b7fe640 1 -- 172.21.15.79:0/3708874398 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fcc04009a30 con 0x7fcc14101670 2024-04-22T06:45:02.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 Processor -- start 2024-04-22T06:45:02.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 -- start start 2024-04-22T06:45:02.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6aee80 0x5615bf6b7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6b7a00 0x5615bf6b9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5615bf6131f0 con 0x5615bf6ae4d0 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5615bf5b04b0 con 0x5615bf6aee80 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1f017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615bf60f1b0 con 0x5615bf6b7a00 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc0b7fe640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc000a3f50 0x7fcc000a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1a8f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1a8f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6ae8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45776/0 (socket says 172.21.15.79:45776) 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/1343223761 learned_addr learned my addr 172.21.15.79:0/1343223761 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc198ef640 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6b7a00 0x5615bf6b9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.974+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6aee80 0x5615bf6b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc000a3f50 0x7fcc000a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc0b7fe640 1 -- 172.21.15.79:0/3708874398 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fcc040d04c0 con 0x7fcc14101670 2024-04-22T06:45:02.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/1343223761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6b7a00 msgr2=0x5615bf6b9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6b7a00 0x5615bf6b9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/1343223761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6aee80 msgr2=0x5615bf6b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6aee80 0x5615bf6b7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/1343223761 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615bf5a21b0 con 0x5615bf6ae4d0 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6ae8b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036040 tx=0x7fcc0c02ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a14f2cb024cce8e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc08bfe640 1 -- 172.21.15.79:0/1343223761 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c005240 con 0x5615bf6ae4d0 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc000a3f50 0x7fcc000a6410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c049000 tx=0x7fcc0c048000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc08bfe640 1 -- 172.21.15.79:0/1343223761 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc0c0053e0 con 0x5615bf6ae4d0 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.975+0000 7fcc08bfe640 1 -- 172.21.15.79:0/1343223761 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c003810 con 0x5615bf6ae4d0 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 -- 172.21.15.79:0/1343223761 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 msgr2=0x5615bf6ae8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6ae8b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036040 tx=0x7fcc0c02ca90 comp rx=0 tx=0).stop 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 -- 172.21.15.79:0/1343223761 shutdown_connections 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6aee80 0x5615bf6b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6b7a00 0x5615bf6b9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 --2- 172.21.15.79:0/1343223761 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6ae8b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 -- 172.21.15.79:0/1343223761 >> 172.21.15.79:0/1343223761 conn(0x5615bf6a8ba0 msgr2=0x5615bf6a8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 -- 172.21.15.79:0/1343223761 shutdown_connections 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 -- 172.21.15.79:0/1343223761 wait complete. 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 -- start start 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 Processor -- start 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.976+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 0x5615bf6b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5615bf6ba970 con 0x5615bf6b7a00 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5615bf5cc500 con 0x5615bf6ae4d0 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615bf6baaf0 con 0x5615bf6aee80 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 10 client.0 _ll_register_callbacks cb 0x5615bf59ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:02.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 0x5615bf6b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 0x5615bf6b4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45792/0 (socket says 172.21.15.79:45792) 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/2955687996 learned_addr learned my addr 172.21.15.79:0/2955687996 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc198ef640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/2955687996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 msgr2=0x5615bf6b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6ae4d0 msgr2=0x5615bf6b7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 -- 172.21.15.79:0/2955687996 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615bf3253f0 con 0x5615bf6b7a00 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 0x5615bf6b4d10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c04f040 tx=0x7fcc0c005330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f18be3fa6a750 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcbfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcbfdffb640 1 -- 172.21.15.79:0/2955687996 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c05b060 con 0x5615bf6b7a00 2024-04-22T06:45:02.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcbfdffb640 1 -- 172.21.15.79:0/2955687996 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fcc0c03a040 con 0x5615bf6b7a00 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc0c0053e0 con 0x5615bf6b7a00 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcbfdffb640 1 -- 172.21.15.79:0/2955687996 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcc0c005590 con 0x5615bf6b7a00 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 10 client.15801 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.977+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5615bf3ef7d0 con 0x5615bf6b7a00 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcbfdffb640 1 -- 172.21.15.79:0/2955687996 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc0c0053e0 con 0x5615bf6b7a00 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcbfdffb640 1 -- 172.21.15.79:0/2955687996 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcc0c040070 con 0x5615bf6b7a00 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1f017740 10 client.15801 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1f017740 10 client.15801 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1f017740 20 client.15801 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf6c7b10 0x5615bf6c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1f017740 4 client.15801 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615bf5b2930 con 0x5615bf6c7b10 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.978+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf6c7b10 0x5615bf6c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcbfdffb640 10 client.15801 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf6c7b10 0x5615bf6c9f00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e001d87e3254c655 server_cookie=cfb672e6e78cd153 in_seq=0 out_seq=0 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcbfdffb640 1 -- 172.21.15.79:0/2955687996 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615bf5b2930 con 0x5615bf6c7b10 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcbfdffb640 10 client.15801 handle_command_reply: tid=0 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 client.15801 shutdown 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 msgr2=0x5615bf6b4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 0x5615bf6b4d10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c04f040 tx=0x7fcc0c005330 comp rx=0 tx=0).stop 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 shutdown_connections 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf6c7b10 0x5615bf6c9f00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6b7a00 0x5615bf6b4d10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2955687996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcbfdffb640 0 client.15801 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 >> 172.21.15.79:0/2955687996 conn(0x5615bf6a8ba0 msgr2=0x5615bf6aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 shutdown_connections 2024-04-22T06:45:02.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.979+0000 7fcc1f017740 1 -- 172.21.15.79:0/2955687996 wait complete. 2024-04-22T06:45:02.986 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.980+0000 7fcc1f017740 20 client.15801 trim_cache size 0 max 16384 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 Processor -- start 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 -- start start 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141b2610 con 0x5615bf6ae4d0 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141afee0 con 0x5615bf6b7a00 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1f017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141add30 con 0x5615bf6aee80 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45802/0 (socket says 172.21.15.79:45802) 2024-04-22T06:45:02.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/2188485065 learned_addr learned my addr 172.21.15.79:0/2188485065 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/2188485065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 msgr2=0x5615bf6a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/2188485065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 msgr2=0x5615bf6b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/2188485065 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615bf6b2c70 con 0x5615bf6ae4d0 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b3880 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5615bf6adfa0 tx=0x7fcc10009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f28808dbbbc01f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcbfffff640 1 -- 172.21.15.79:0/2188485065 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10004d20 con 0x5615bf6ae4d0 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.993+0000 7fcbfffff640 1 -- 172.21.15.79:0/2188485065 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc10023030 con 0x5615bf6ae4d0 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcbfffff640 1 -- 172.21.15.79:0/2188485065 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1001f790 con 0x5615bf6ae4d0 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 -- 172.21.15.79:0/2188485065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 msgr2=0x5615bf6b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b3880 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5615bf6adfa0 tx=0x7fcc10009500 comp rx=0 tx=0).stop 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 -- 172.21.15.79:0/2188485065 shutdown_connections 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6b6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 --2- 172.21.15.79:0/2188485065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6b3880 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 -- 172.21.15.79:0/2188485065 >> 172.21.15.79:0/2188485065 conn(0x5615bf6a8ba0 msgr2=0x5615bf6a7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 -- 172.21.15.79:0/2188485065 shutdown_connections 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 -- 172.21.15.79:0/2188485065 wait complete. 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 -- start start 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.994+0000 7fcc1f017740 1 Processor -- start 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6caa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141ae3a0 con 0x5615bf6ae4d0 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141af810 con 0x5615bf6b7a00 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141aea70 con 0x5615bf6aee80 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 10 client.0 _ll_register_callbacks cb 0x7fcc0c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:02.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45814/0 (socket says 172.21.15.79:45814) 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/267815482 learned_addr learned my addr 172.21.15.79:0/267815482 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc198ef640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/267815482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 msgr2=0x5615bf6ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 msgr2=0x5615bf6caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 -- 172.21.15.79:0/267815482 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615bf6cafb0 con 0x5615bf6ae4d0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a8f1640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1a0f0640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6cd080 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000a010 tx=0x7fcc1001f250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887ce7fb5ed318bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcbfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcbfdffb640 1 -- 172.21.15.79:0/267815482 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10004db0 con 0x5615bf6ae4d0 2024-04-22T06:45:03.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcbfdffb640 1 -- 172.21.15.79:0/267815482 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fcc1001f510 con 0x5615bf6ae4d0 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.995+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10023030 con 0x5615bf6ae4d0 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcbfdffb640 1 -- 172.21.15.79:0/267815482 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcc1001f710 con 0x5615bf6ae4d0 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 10 client.15807 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5615bf72da70 con 0x5615bf6ae4d0 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcbfdffb640 1 -- 172.21.15.79:0/267815482 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc10023030 con 0x5615bf6ae4d0 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcbfdffb640 1 -- 172.21.15.79:0/267815482 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fcc10005420 con 0x5615bf6ae4d0 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 10 client.15807 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 10 client.15807 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 20 client.15807 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf72e160 0x5615bf730580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 4 client.15807 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 --> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615bf620c10 con 0x5615bf72e160 2024-04-22T06:45:03.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc198ef640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf72e160 0x5615bf730580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcbfdffb640 10 client.15807 ms_handle_connect on v2:172.21.15.190:6836/2879949884 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.996+0000 7fcc198ef640 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf72e160 0x5615bf730580 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d5c880f96614d76 server_cookie=6a84df90c8d146c6 in_seq=0 out_seq=0 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcbfdffb640 1 -- 172.21.15.79:0/267815482 <== mds.1 v2:172.21.15.190:6836/2879949884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615bf620c10 con 0x5615bf72e160 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcbfdffb640 10 client.15807 handle_command_reply: tid=0 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 client.15807 shutdown 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 msgr2=0x5615bf6cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6cd080 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000a010 tx=0x7fcc1001f250 comp rx=0 tx=0).stop 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 shutdown_connections 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:6836/2879949884,v1:172.21.15.190:6837/2879949884] conn(0x5615bf72e160 0x5615bf730580 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615bf6b7a00 0x5615bf6caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615bf6aee80 0x5615bf6ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 --2- 172.21.15.79:0/267815482 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5615bf6ae4d0 0x5615bf6cd080 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 >> 172.21.15.79:0/267815482 conn(0x5615bf6a8ba0 msgr2=0x5615bf6c2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcbfdffb640 0 client.15807 ms_handle_reset on v2:172.21.15.190:6836/2879949884 2024-04-22T06:45:03.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 shutdown_connections 2024-04-22T06:45:03.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 1 -- 172.21.15.79:0/267815482 wait complete. 2024-04-22T06:45:03.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1f017740 20 client.15807 trim_cache size 0 max 16384 2024-04-22T06:45:03.003 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc000a3f50 msgr2=0x7fcc000a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc000a3f50 0x7fcc000a6410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c049000 tx=0x7fcc0c048000 comp rx=0 tx=0).stop 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 msgr2=0x7fcc141a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 0x7fcc141a0bc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400d950 tx=0x7fcc0400de10 comp rx=0 tx=0).stop 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 shutdown_connections 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc000a3f50 0x7fcc000a6410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc1410e900 0x7fcc1419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc14101670 0x7fcc141a0bc0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 --2- 172.21.15.79:0/3708874398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc14100d50 0x7fcc141a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.997+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 >> 172.21.15.79:0/3708874398 conn(0x7fcc14077f50 msgr2=0x7fcc14110030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.998+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 shutdown_connections 2024-04-22T06:45:03.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:02.998+0000 7fcc1c354640 1 -- 172.21.15.79:0/3708874398 wait complete. 2024-04-22T06:45:03.007 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:45:03.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 -- 172.21.15.79:0/1071259413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350105ea0 msgr2=0x7f4350106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1071259413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350105ea0 0x7f4350106280 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4344009900 tx=0x7f434402bc80 comp rx=0 tx=0).stop 2024-04-22T06:45:03.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 -- 172.21.15.79:0/1071259413 shutdown_connections 2024-04-22T06:45:03.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1071259413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4350101130 0x7f4350101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1071259413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4350101ad0 0x7f435010dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1071259413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350105ea0 0x7f4350106280 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 -- 172.21.15.79:0/1071259413 >> 172.21.15.79:0/1071259413 conn(0x7f4350077e30 msgr2=0x7f4350078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 -- 172.21.15.79:0/1071259413 shutdown_connections 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.111+0000 7f4355d9f640 1 -- 172.21.15.79:0/1071259413 wait complete. 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 Processor -- start 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 -- start start 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 0x7f4350197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4350101ad0 0x7f4350197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4350105ea0 0x7f435019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43501111d0 con 0x7f4350101130 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4350111050 con 0x7f4350105ea0 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f4355d9f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4350111350 con 0x7f4350101ad0 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 0x7f4350197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 0x7f4350197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45820/0 (socket says 172.21.15.79:45820) 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 -- 172.21.15.79:0/1435534821 learned_addr learned my addr 172.21.15.79:0/1435534821 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434ffff640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4350105ea0 0x7f435019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 -- 172.21.15.79:0/1435534821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4350101ad0 msgr2=0x7f4350197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4350101ad0 0x7f4350197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 -- 172.21.15.79:0/1435534821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4350105ea0 msgr2=0x7f435019bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4350105ea0 0x7f435019bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.112+0000 7f434f7fe640 1 -- 172.21.15.79:0/1435534821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435019c710 con 0x7f4350101130 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.113+0000 7f434f7fe640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 0x7f4350197770 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4344006010 tx=0x7f4344004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.113+0000 7f434cff9640 1 -- 172.21.15.79:0/1435534821 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344043020 con 0x7f4350101130 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.113+0000 7f434cff9640 1 -- 172.21.15.79:0/1435534821 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4344004440 con 0x7f4350101130 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.113+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435019c9a0 con 0x7f4350101130 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.113+0000 7f434cff9640 1 -- 172.21.15.79:0/1435534821 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344003b50 con 0x7f4350101130 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.113+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43501a4290 con 0x7f4350101130 2024-04-22T06:45:03.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f434cff9640 1 -- 172.21.15.79:0/1435534821 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4344046020 con 0x7f4350101130 2024-04-22T06:45:03.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f435898a740 1 Processor -- start 2024-04-22T06:45:03.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f435898a740 1 -- start start 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f434cff9640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43380a3ed0 0x7f43380a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f435898a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da526e0 0x55ed0da52ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f434ffff640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43380a3ed0 0x7f43380a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.115+0000 7f435898a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 0x55ed0da5b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f435898a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da5e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f435898a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d9b51f0 con 0x55ed0da53090 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434cff9640 1 -- 172.21.15.79:0/1435534821 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f43440f27f0 con 0x7f4350101130 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f435898a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d9524b0 con 0x55ed0da5bc10 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f435898a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d9b11b0 con 0x55ed0da526e0 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 0x55ed0da5b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 0x55ed0da5b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45832/0 (socket says 172.21.15.79:45832) 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 -- 172.21.15.79:0/3605573948 learned_addr learned my addr 172.21.15.79:0/3605573948 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:03.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434f7fe640 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da526e0 0x55ed0da52ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 -- 172.21.15.79:0/3605573948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da526e0 msgr2=0x55ed0da52ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da526e0 0x55ed0da52ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 -- 172.21.15.79:0/3605573948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 msgr2=0x55ed0da5e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da5e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434ffff640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43380a3ed0 0x7f43380a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4348005c40 tx=0x7f434800f730 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 -- 172.21.15.79:0/3605573948 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0d9441b0 con 0x55ed0da53090 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f434effd640 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 0x55ed0da5b640 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f434000a9c0 tx=0x7f434000ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d1c84f7427eed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f4335ffb640 1 -- 172.21.15.79:0/3605573948 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434000cde0 con 0x55ed0da53090 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f4335ffb640 1 -- 172.21.15.79:0/3605573948 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f434000f900 con 0x55ed0da53090 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.116+0000 7f4335ffb640 1 -- 172.21.15.79:0/3605573948 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340018910 con 0x55ed0da53090 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 -- 172.21.15.79:0/3605573948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 msgr2=0x55ed0da5b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 0x55ed0da5b640 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f434000a9c0 tx=0x7f434000ae80 comp rx=0 tx=0).stop 2024-04-22T06:45:03.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 -- 172.21.15.79:0/3605573948 shutdown_connections 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da5e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da53090 0x55ed0da5b640 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 --2- 172.21.15.79:0/3605573948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da526e0 0x55ed0da52ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 -- 172.21.15.79:0/3605573948 >> 172.21.15.79:0/3605573948 conn(0x55ed0da4cdb0 msgr2=0x55ed0da4d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 -- 172.21.15.79:0/3605573948 shutdown_connections 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 -- 172.21.15.79:0/3605573948 wait complete. 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 -- start start 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.117+0000 7f435898a740 1 Processor -- start 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 0x55ed0da5b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da58ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ed0da5eb80 con 0x55ed0da526e0 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d96e500 con 0x55ed0da5bc10 2024-04-22T06:45:03.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ed0da5ed00 con 0x55ed0da53090 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 10 client.0 _ll_register_callbacks cb 0x55ed0d93cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 0x55ed0da5b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 0x55ed0da5b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45840/0 (socket says 172.21.15.79:45840) 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 -- 172.21.15.79:0/735466232 learned_addr learned my addr 172.21.15.79:0/735466232 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434ffff640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da58ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434effd640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 -- 172.21.15.79:0/735466232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 msgr2=0x55ed0da588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 -- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 msgr2=0x55ed0da58ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da58ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 -- 172.21.15.79:0/735466232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0d6c73f0 con 0x55ed0da526e0 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434effd640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f434f7fe640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 0x55ed0da5b550 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f43440be540 tx=0x7f43440bdf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10ba65ba382b256 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f432effd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f432effd640 1 -- 172.21.15.79:0/735466232 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43440d7030 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f432effd640 1 -- 172.21.15.79:0/735466232 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f43440c6f90 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f434000f900 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f432effd640 1 -- 172.21.15.79:0/735466232 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f43440c2960 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 10 client.15819 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.118+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ed0d7917d0 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f432effd640 1 -- 172.21.15.79:0/735466232 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f434000f900 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f432effd640 1 -- 172.21.15.79:0/735466232 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f43440f00b0 con 0x55ed0da526e0 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f435898a740 10 client.15819 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f435898a740 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f435898a740 20 client.15819 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f435898a740 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0da6be60 0x55ed0da6e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f435898a740 4 client.15819 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:03.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed0d954930 con 0x55ed0da6be60 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.119+0000 7f434effd640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0da6be60 0x55ed0da6e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f432effd640 10 client.15819 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f434effd640 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0da6be60 0x55ed0da6e2d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=593c23df91bf1f3c server_cookie=8f8192b9498bb4de in_seq=0 out_seq=0 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f432effd640 1 -- 172.21.15.79:0/735466232 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed0d954930 con 0x55ed0da6be60 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f432effd640 10 client.15819 handle_command_reply: tid=0 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f435898a740 1 client.15819 shutdown 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 msgr2=0x55ed0da5b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f435898a740 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 0x55ed0da5b550 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f43440be540 tx=0x7f43440bdf10 comp rx=0 tx=0).stop 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 shutdown_connections 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.120+0000 7f435898a740 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0da6be60 0x55ed0da6e2d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da5bc10 0x55ed0da58ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f432effd640 0 client.15819 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:03.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 1 --2- 172.21.15.79:0/735466232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da526e0 0x55ed0da5b550 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 >> 172.21.15.79:0/735466232 conn(0x55ed0da4cdb0 msgr2=0x55ed0da4eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 shutdown_connections 2024-04-22T06:45:03.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 1 -- 172.21.15.79:0/735466232 wait complete. 2024-04-22T06:45:03.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.121+0000 7f435898a740 20 client.15819 trim_cache size 0 max 16384 2024-04-22T06:45:03.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.133+0000 7f435898a740 1 Processor -- start 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 -- start start 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da67a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da64360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0da5b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43501ae100 con 0x55ed0da5bc10 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43501ab9d0 con 0x55ed0da526e0 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f435898a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43501a9820 con 0x55ed0da53090 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0da5b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0da5b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45842/0 (socket says 172.21.15.79:45842) 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 -- 172.21.15.79:0/3469157931 learned_addr learned my addr 172.21.15.79:0/3469157931 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434effd640 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da64360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 -- 172.21.15.79:0/3469157931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 msgr2=0x55ed0da64360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da64360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 -- 172.21.15.79:0/3469157931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 msgr2=0x55ed0da67a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da67a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 -- 172.21.15.79:0/3469157931 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0da4c8e0 con 0x55ed0da5bc10 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f434ffff640 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0da5b010 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f43480115b0 tx=0x7f4348014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354cc6c9b1eb8e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f43357fa640 1 -- 172.21.15.79:0/3469157931 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348003b70 con 0x55ed0da5bc10 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f43357fa640 1 -- 172.21.15.79:0/3469157931 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4348003d10 con 0x55ed0da5bc10 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.134+0000 7f43357fa640 1 -- 172.21.15.79:0/3469157931 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434801f9c0 con 0x55ed0da5bc10 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 -- 172.21.15.79:0/3469157931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 msgr2=0x55ed0da5b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0da5b010 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f43480115b0 tx=0x7f4348014000 comp rx=0 tx=0).stop 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 -- 172.21.15.79:0/3469157931 shutdown_connections 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0da5b010 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da64360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 --2- 172.21.15.79:0/3469157931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da67a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 -- 172.21.15.79:0/3469157931 >> 172.21.15.79:0/3469157931 conn(0x55ed0da4cdb0 msgr2=0x55ed0da4bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 -- 172.21.15.79:0/3469157931 shutdown_connections 2024-04-22T06:45:03.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 -- 172.21.15.79:0/3469157931 wait complete. 2024-04-22T06:45:03.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.135+0000 7f435898a740 1 -- start start 2024-04-22T06:45:03.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 Processor -- start 2024-04-22T06:45:03.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da4a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da4a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0dacd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43501a9e90 con 0x55ed0da5bc10 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43501ab300 con 0x55ed0da526e0 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43501aa560 con 0x55ed0da53090 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 10 client.0 _ll_register_callbacks cb 0x7f43440be450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0dacd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0dacd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45854/0 (socket says 172.21.15.79:45854) 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 -- 172.21.15.79:0/1428243958 learned_addr learned my addr 172.21.15.79:0/1428243958 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434effd640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da4a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434f7fe640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da4a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 -- 172.21.15.79:0/1428243958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 msgr2=0x55ed0da4a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da4a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 -- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 msgr2=0x55ed0da4a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da4a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 -- 172.21.15.79:0/1428243958 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0dacdad0 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f434ffff640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0dacd490 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4348006010 tx=0x7f4348003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6410a405f0c02a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f432effd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f432effd640 1 -- 172.21.15.79:0/1428243958 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434801f8f0 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f432effd640 1 -- 172.21.15.79:0/1428243958 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f434801fa90 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.136+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4348003d10 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f432effd640 1 -- 172.21.15.79:0/1428243958 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f434801fc90 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 10 client.15825 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ed0dad4e40 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f432effd640 1 -- 172.21.15.79:0/1428243958 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4348003d10 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f432effd640 1 -- 172.21.15.79:0/1428243958 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4348032420 con 0x55ed0da5bc10 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 10 client.15825 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 20 client.15825 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:03.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0dad5560 0x55ed0dad7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 4 client.15825 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed0d9c2c10 con 0x55ed0dad5560 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.137+0000 7f434f7fe640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0dad5560 0x55ed0dad7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f432effd640 10 client.15825 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f434f7fe640 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0dad5560 0x55ed0dad7980 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73d60ab6835c3b51 server_cookie=24f6794bcf515b89 in_seq=0 out_seq=0 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f432effd640 1 -- 172.21.15.79:0/1428243958 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed0d9c2c10 con 0x55ed0dad5560 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f432effd640 10 client.15825 handle_command_reply: tid=0 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 client.15825 shutdown 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 msgr2=0x55ed0dacd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0dacd490 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4348006010 tx=0x7f4348003ba0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 shutdown_connections 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ed0dad5560 0x55ed0dad7980 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ed0da5bc10 0x55ed0dacd490 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f432effd640 0 client.15825 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0da53090 0x55ed0da4a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 --2- 172.21.15.79:0/1428243958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ed0da526e0 0x55ed0da4a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 >> 172.21.15.79:0/1428243958 conn(0x55ed0da4cdb0 msgr2=0x55ed0da4de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 shutdown_connections 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 1 -- 172.21.15.79:0/1428243958 wait complete. 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.138+0000 7f435898a740 20 client.15825 trim_cache size 0 max 16384 2024-04-22T06:45:03.143 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43380a3ed0 msgr2=0x7f43380a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43380a3ed0 0x7f43380a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4348005c40 tx=0x7f434800f730 comp rx=0 tx=0).stop 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 msgr2=0x7f4350197770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 0x7f4350197770 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4344006010 tx=0x7f4344004040 comp rx=0 tx=0).stop 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 shutdown_connections 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43380a3ed0 0x7f43380a6390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4350105ea0 0x7f435019bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4350101ad0 0x7f4350197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 --2- 172.21.15.79:0/1435534821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4350101130 0x7f4350197770 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 >> 172.21.15.79:0/1435534821 conn(0x7f4350077e30 msgr2=0x7f43500fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 shutdown_connections 2024-04-22T06:45:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:03.139+0000 7f4355d9f640 1 -- 172.21.15.79:0/1435534821 wait complete. 2024-04-22T06:45:03.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:03 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1380293412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:03.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:03 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1380293412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:03.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:03 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1380293412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:04.149 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:45:04.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:45:04.150 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 -- 172.21.15.79:0/2976925969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3641031d0 msgr2=0x7fd3641035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 --2- 172.21.15.79:0/2976925969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3641031d0 0x7fd3641035b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd354009900 tx=0x7fd35402bd60 comp rx=0 tx=0).stop 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 -- 172.21.15.79:0/2976925969 shutdown_connections 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 --2- 172.21.15.79:0/2976925969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd36410c510 0x7fd36410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 --2- 172.21.15.79:0/2976925969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364103af0 0x7fd36410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 --2- 172.21.15.79:0/2976925969 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3641031d0 0x7fd3641035b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 -- 172.21.15.79:0/2976925969 >> 172.21.15.79:0/2976925969 conn(0x7fd364077f50 msgr2=0x7fd364078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 -- 172.21.15.79:0/2976925969 shutdown_connections 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.259+0000 7fd36a881640 1 -- 172.21.15.79:0/2976925969 wait complete. 2024-04-22T06:45:04.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 Processor -- start 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 -- start start 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3641031d0 0x7fd364100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364103af0 0x7fd364101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 0x7fd3641018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641121f0 con 0x7fd36410c510 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd364112070 con 0x7fd3641031d0 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd36a881640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd364112370 con 0x7fd364103af0 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.260+0000 7fd368e1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 0x7fd3641018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 0x7fd3641018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45870/0 (socket says 172.21.15.79:45870) 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 -- 172.21.15.79:0/861040615 learned_addr learned my addr 172.21.15.79:0/861040615 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd3637fe640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364103af0 0x7fd364101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd363fff640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3641031d0 0x7fd364100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 -- 172.21.15.79:0/861040615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364103af0 msgr2=0x7fd364101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364103af0 0x7fd364101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 -- 172.21.15.79:0/861040615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3641031d0 msgr2=0x7fd364100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3641031d0 0x7fd364100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 -- 172.21.15.79:0/861040615 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3641a6c70 con 0x7fd36410c510 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd368e1e640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 0x7fd3641018a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00eaa0 tx=0x7fd35c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd3617fa640 1 -- 172.21.15.79:0/861040615 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00cdb0 con 0x7fd36410c510 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd3617fa640 1 -- 172.21.15.79:0/861040615 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd35c005020 con 0x7fd36410c510 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3641a6f60 con 0x7fd36410c510 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd3617fa640 1 -- 172.21.15.79:0/861040615 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c0116d0 con 0x7fd36410c510 2024-04-22T06:45:04.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.261+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3641a7430 con 0x7fd36410c510 2024-04-22T06:45:04.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd3617fa640 1 -- 172.21.15.79:0/861040615 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fd35c0051d0 con 0x7fd36410c510 2024-04-22T06:45:04.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 Processor -- start 2024-04-22T06:45:04.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 -- start start 2024-04-22T06:45:04.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bf8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534c01840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534c04260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b534b5cf00 con 0x55b534bf9290 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b534afa590 con 0x55b534c01e10 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd36d47a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b534b58ec0 con 0x55b534bf88e0 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd363fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534c01840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd363fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534c01840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45874/0 (socket says 172.21.15.79:45874) 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.264+0000 7fd363fff640 1 -- 172.21.15.79:0/1946153571 learned_addr learned my addr 172.21.15.79:0/1946153571 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bf8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd3617fa640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd34c0a3ed0 0x7fd34c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534c04260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd368e1e640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd34c0a3ed0 0x7fd34c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd363fff640 1 -- 172.21.15.79:0/1946153571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 msgr2=0x55b534bf8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd363fff640 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bf8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd363fff640 1 -- 172.21.15.79:0/1946153571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 msgr2=0x55b534c04260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd363fff640 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534c04260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd363fff640 1 -- 172.21.15.79:0/1946153571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5349377e0 con 0x55b534bf9290 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bf8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd3617fa640 1 -- 172.21.15.79:0/861040615 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd35c0d0ba0 con 0x7fd36410c510 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd363fff640 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534c01840 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd354000c00 tx=0x7fd35402c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97929d83d6fe7b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd368e1e640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd34c0a3ed0 0x7fd34c0a6390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0ce140 tx=0x7fd35c087040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd34a7fc640 1 -- 172.21.15.79:0/1946153571 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354004140 con 0x55b534bf9290 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd34a7fc640 1 -- 172.21.15.79:0/1946153571 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd35402cdc0 con 0x55b534bf9290 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.265+0000 7fd34a7fc640 1 -- 172.21.15.79:0/1946153571 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540046c0 con 0x55b534bf9290 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 -- 172.21.15.79:0/1946153571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 msgr2=0x55b534c01840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534c01840 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd354000c00 tx=0x7fd35402c7a0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 -- 172.21.15.79:0/1946153571 shutdown_connections 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534c04260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534c01840 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1946153571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bf8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 -- 172.21.15.79:0/1946153571 >> 172.21.15.79:0/1946153571 conn(0x55b534bf2fb0 msgr2=0x55b534bf3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 -- 172.21.15.79:0/1946153571 shutdown_connections 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 -- 172.21.15.79:0/1946153571 wait complete. 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 -- start start 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.266+0000 7fd36d47a740 1 Processor -- start 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bfedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534bff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b534c04d20 con 0x55b534bf9290 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b534b16580 con 0x55b534c01e10 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b534c04ea0 con 0x55b534bf88e0 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 10 client.0 _ll_register_callbacks cb 0x55b534b1c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:04.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45886/0 (socket says 172.21.15.79:45886) 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 -- 172.21.15.79:0/1647618649 learned_addr learned my addr 172.21.15.79:0/1647618649 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bfedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534bff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 -- 172.21.15.79:0/1647618649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 msgr2=0x55b534bfedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bfedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 -- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 msgr2=0x55b534bff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534bff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 -- 172.21.15.79:0/1647618649 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b534997f70 con 0x55b534bf9290 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534bff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd363fff640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff330 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd3540096c0 tx=0x7fd354005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168a0727a58fa0b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd3437fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd3437fe640 1 -- 172.21.15.79:0/1647618649 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540035f0 con 0x55b534bf9290 2024-04-22T06:45:04.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd3437fe640 1 -- 172.21.15.79:0/1647618649 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd354003790 con 0x55b534bf9290 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.267+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd35402cdc0 con 0x55b534bf9290 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd3437fe640 1 -- 172.21.15.79:0/1647618649 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd354003990 con 0x55b534bf9290 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 10 client.15837 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b53482f170 con 0x55b534bf9290 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd3437fe640 1 -- 172.21.15.79:0/1647618649 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd35402cdc0 con 0x55b534bf9290 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd3437fe640 1 -- 172.21.15.79:0/1647618649 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd354034c90 con 0x55b534bf9290 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 10 client.15837 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 20 client.15837 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c121c0 0x55b534c145b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 4 client.15837 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b534b6a920 con 0x55b534c121c0 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.268+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c121c0 0x55b534c145b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.269+0000 7fd3437fe640 10 client.15837 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.269+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c121c0 0x55b534c145b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b03b298939aa25 server_cookie=70099d19ce2d72af in_seq=0 out_seq=0 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.269+0000 7fd3437fe640 1 -- 172.21.15.79:0/1647618649 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b534b6a920 con 0x55b534c121c0 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.269+0000 7fd3437fe640 10 client.15837 handle_command_reply: tid=0 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.269+0000 7fd36d47a740 1 client.15837 shutdown 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 msgr2=0x55b534bff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff330 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd3540096c0 tx=0x7fd354005730 comp rx=0 tx=0).stop 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 shutdown_connections 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c121c0 0x55b534c145b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534c01e10 0x55b534bff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd3437fe640 0 client.15837 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff330 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1647618649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534bf88e0 0x55b534bfedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 >> 172.21.15.79:0/1647618649 conn(0x55b534bf2fb0 msgr2=0x55b534bf4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 shutdown_connections 2024-04-22T06:45:04.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 1 -- 172.21.15.79:0/1647618649 wait complete. 2024-04-22T06:45:04.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.270+0000 7fd36d47a740 20 client.15837 trim_cache size 0 max 16384 2024-04-22T06:45:04.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 Processor -- start 2024-04-22T06:45:04.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 -- start start 2024-04-22T06:45:04.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf88e0 0x55b534c0dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c19b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641b25c0 con 0x55b534bf9290 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641afe90 con 0x55b534bf88e0 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd36d47a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641adce0 con 0x55b534c01e10 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45890/0 (socket says 172.21.15.79:45890) 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 -- 172.21.15.79:0/1810228010 learned_addr learned my addr 172.21.15.79:0/1810228010 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd3637fe640 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c19b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 -- 172.21.15.79:0/1810228010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 msgr2=0x55b534c19b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c19b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 -- 172.21.15.79:0/1810228010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf88e0 msgr2=0x55b534c0dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf88e0 0x55b534c0dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 -- 172.21.15.79:0/1810228010 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b534bf9030 con 0x55b534bf9290 2024-04-22T06:45:04.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.287+0000 7fd363fff640 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff690 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd354000c00 tx=0x7fd354001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd48932b1a4d408c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd349ffb640 1 -- 172.21.15.79:0/1810228010 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540035f0 con 0x55b534bf9290 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd349ffb640 1 -- 172.21.15.79:0/1810228010 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd354003790 con 0x55b534bf9290 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd349ffb640 1 -- 172.21.15.79:0/1810228010 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35403fdd0 con 0x55b534bf9290 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 -- 172.21.15.79:0/1810228010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 msgr2=0x55b534bff690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff690 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd354000c00 tx=0x7fd354001e70 comp rx=0 tx=0).stop 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 -- 172.21.15.79:0/1810228010 shutdown_connections 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c19b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf9290 0x55b534bff690 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 --2- 172.21.15.79:0/1810228010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf88e0 0x55b534c0dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 -- 172.21.15.79:0/1810228010 >> 172.21.15.79:0/1810228010 conn(0x55b534bf2fb0 msgr2=0x55b534bf2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 -- 172.21.15.79:0/1810228010 shutdown_connections 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 -- 172.21.15.79:0/1810228010 wait complete. 2024-04-22T06:45:04.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.288+0000 7fd36d47a740 1 -- start start 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 Processor -- start 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 0x55b534c0a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf9290 0x55b534c05930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c05ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641ae350 con 0x55b534bf88e0 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641af7c0 con 0x55b534bf9290 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641aea20 con 0x55b534c01e10 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd36d47a740 10 client.0 _ll_register_callbacks cb 0x7fd35800b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 0x55b534c0a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 0x55b534c0a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45898/0 (socket says 172.21.15.79:45898) 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 -- 172.21.15.79:0/2644224947 learned_addr learned my addr 172.21.15.79:0/2644224947 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 -- 172.21.15.79:0/2644224947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 msgr2=0x55b534c05ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c05ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 -- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf9290 msgr2=0x55b534c05930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd363fff640 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf9290 0x55b534c05930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf9290 0x55b534c05930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 -- 172.21.15.79:0/2644224947 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b534c06720 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3637fe640 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 0x55b534c0a5c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd35800c010 tx=0x7fd3580039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82e3521428efed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.289+0000 7fd3437fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd3437fe640 1 -- 172.21.15.79:0/2644224947 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358028060 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd3437fe640 1 -- 172.21.15.79:0/2644224947 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd358023250 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd354003790 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd3437fe640 1 -- 172.21.15.79:0/2644224947 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd35801b040 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 10 client.15843 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b534c78560 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd3437fe640 1 -- 172.21.15.79:0/2644224947 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd354003790 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd3437fe640 1 -- 172.21.15.79:0/2644224947 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd35802db20 con 0x55b534bf88e0 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 10 client.15843 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:04.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 20 client.15843 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c78c90 0x55b534c7b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 4 client.15843 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b534b64790 con 0x55b534c78c90 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.290+0000 7fd363fff640 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c78c90 0x55b534c7b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.291+0000 7fd3437fe640 10 client.15843 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.291+0000 7fd363fff640 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c78c90 0x55b534c7b0b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e448d10e400a5458 server_cookie=41bcce059e01a81e in_seq=0 out_seq=0 2024-04-22T06:45:04.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd3437fe640 1 -- 172.21.15.79:0/2644224947 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b534b64790 con 0x55b534c78c90 2024-04-22T06:45:04.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd3437fe640 10 client.15843 handle_command_reply: tid=0 2024-04-22T06:45:04.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 client.15843 shutdown 2024-04-22T06:45:04.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 msgr2=0x55b534c0a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 0x55b534c0a5c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd35800c010 tx=0x7fd3580039c0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 shutdown_connections 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b534c78c90 0x55b534c7b0b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b534bf9290 0x55b534c05930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b534c01e10 0x55b534c05ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 --2- 172.21.15.79:0/2644224947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b534bf88e0 0x55b534c0a5c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 >> 172.21.15.79:0/2644224947 conn(0x55b534bf2fb0 msgr2=0x55b534bf3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd3437fe640 0 client.15843 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 shutdown_connections 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 1 -- 172.21.15.79:0/2644224947 wait complete. 2024-04-22T06:45:04.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.292+0000 7fd36d47a740 20 client.15843 trim_cache size 0 max 16384 2024-04-22T06:45:04.297 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:04.297 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:45:04.297 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "718a5681-b4cd-4cb2-a984-7a8f112edb3b", 2024-04-22T06:45:04.297 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:45:04.297 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:04.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd34c0a3ed0 msgr2=0x7fd34c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd34c0a3ed0 0x7fd34c0a6390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0ce140 tx=0x7fd35c087040 comp rx=0 tx=0).stop 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 msgr2=0x7fd3641018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 0x7fd3641018a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00eaa0 tx=0x7fd35c00ef60 comp rx=0 tx=0).stop 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 shutdown_connections 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd36410c510 0x7fd3641018a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364103af0 0x7fd364101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd34c0a3ed0 0x7fd34c0a6390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 --2- 172.21.15.79:0/861040615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3641031d0 0x7fd364100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 >> 172.21.15.79:0/861040615 conn(0x7fd364077f50 msgr2=0x7fd3640fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 shutdown_connections 2024-04-22T06:45:04.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.293+0000 7fd36a881640 1 -- 172.21.15.79:0/861040615 wait complete. 2024-04-22T06:45:04.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:04.302 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:04.322 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:04 smithi079 ceph-mon[27946]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:04.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 -- 172.21.15.79:0/3300099709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c103ae0 msgr2=0x7fd61c10bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3300099709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c103ae0 0x7fd61c10bfc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd610009900 tx=0x7fd61002bd60 comp rx=0 tx=0).stop 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 -- 172.21.15.79:0/3300099709 shutdown_connections 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3300099709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c10c500 0x7fd61c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3300099709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c103ae0 0x7fd61c10bfc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3300099709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61c1031c0 0x7fd61c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 -- 172.21.15.79:0/3300099709 >> 172.21.15.79:0/3300099709 conn(0x7fd61c077fd0 msgr2=0x7fd61c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 -- 172.21.15.79:0/3300099709 shutdown_connections 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.403+0000 7fd6214c7640 1 -- 172.21.15.79:0/3300099709 wait complete. 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 Processor -- start 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 -- start start 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c1031c0 0x7fd61c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61c103ae0 0x7fd61c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 0x7fd61c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c112270 con 0x7fd61c10c500 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1120f0 con 0x7fd61c1031c0 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd6214c7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1123f0 con 0x7fd61c103ae0 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 0x7fd61c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 0x7fd61c101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45916/0 (socket says 172.21.15.79:45916) 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 -- 172.21.15.79:0/3847815018 learned_addr learned my addr 172.21.15.79:0/3847815018 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61affd640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c1031c0 0x7fd61c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 -- 172.21.15.79:0/3847815018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61c103ae0 msgr2=0x7fd61c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61c103ae0 0x7fd61c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61c103ae0 0x7fd61c101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 -- 172.21.15.79:0/3847815018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c1031c0 msgr2=0x7fd61c100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c1031c0 0x7fd61c100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61b7fe640 1 -- 172.21.15.79:0/3847815018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61c1a6e00 con 0x7fd61c10c500 2024-04-22T06:45:04.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.404+0000 7fd61affd640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c1031c0 0x7fd61c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:04.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.405+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 0x7fd61c101890 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd61400b9e0 tx=0x7fd61400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.405+0000 7fd603fff640 1 -- 172.21.15.79:0/3847815018 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614019070 con 0x7fd61c10c500 2024-04-22T06:45:04.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.405+0000 7fd603fff640 1 -- 172.21.15.79:0/3847815018 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd614014440 con 0x7fd61c10c500 2024-04-22T06:45:04.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.405+0000 7fd603fff640 1 -- 172.21.15.79:0/3847815018 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140075d0 con 0x7fd61c10c500 2024-04-22T06:45:04.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.405+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61c1a70f0 con 0x7fd61c10c500 2024-04-22T06:45:04.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.405+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd61c1a75f0 con 0x7fd61c10c500 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd603fff640 1 -- 172.21.15.79:0/3847815018 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fd61401c070 con 0x7fd61c10c500 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 Processor -- start 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 -- start start 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027db6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd603fff640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6040a3ed0 0x7fd6040a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 0x55a027dbf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027dbfa80 0x55a027dc1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a027d1b1e0 con 0x55a027db6ea0 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a027cb8390 con 0x55a027dbfa80 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd62418b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a027d171a0 con 0x55a027db6580 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.407+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6040a3ed0 0x7fd6040a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 0x55a027dbf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd603fff640 1 -- 172.21.15.79:0/3847815018 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd6140d0030 con 0x7fd61c10c500 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 0x55a027dbf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45930/0 (socket says 172.21.15.79:45930) 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 -- 172.21.15.79:0/1860313774 learned_addr learned my addr 172.21.15.79:0/1860313774 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027dbfa80 0x55a027dc1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61affd640 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027db6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 -- 172.21.15.79:0/1860313774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 msgr2=0x55a027db6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027db6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 -- 172.21.15.79:0/1860313774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027dbfa80 msgr2=0x55a027dc1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027dbfa80 0x55a027dc1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 -- 172.21.15.79:0/1860313774 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a027af77d0 con 0x55a027db6ea0 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6040a3ed0 0x7fd6040a6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd610009900 tx=0x7fd61002c780 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 0x55a027dbf540 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd610005770 tx=0x7fd610045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f5e6873c0c0b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd6017fa640 1 -- 172.21.15.79:0/1860313774 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610052070 con 0x55a027db6ea0 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd6017fa640 1 -- 172.21.15.79:0/1860313774 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6100054e0 con 0x55a027db6ea0 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.408+0000 7fd6017fa640 1 -- 172.21.15.79:0/1860313774 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610003910 con 0x55a027db6ea0 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 -- 172.21.15.79:0/1860313774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 msgr2=0x55a027dbf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 0x55a027dbf540 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd610005770 tx=0x7fd610045000 comp rx=0 tx=0).stop 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 -- 172.21.15.79:0/1860313774 shutdown_connections 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027dbfa80 0x55a027dc1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6ea0 0x55a027dbf540 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 --2- 172.21.15.79:0/1860313774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027db6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 -- 172.21.15.79:0/1860313774 >> 172.21.15.79:0/1860313774 conn(0x55a027db0cb0 msgr2=0x55a027db1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 -- 172.21.15.79:0/1860313774 shutdown_connections 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 -- 172.21.15.79:0/1860313774 wait complete. 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 -- start start 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.409+0000 7fd62418b740 1 Processor -- start 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 0x55a027dbcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dbd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027dbfa80 0x55a027dbd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a027dc2810 con 0x55a027db6580 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a027cd42f0 con 0x55a027db6ea0 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a027dc2990 con 0x55a027dbfa80 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 10 client.0 _ll_register_callbacks cb 0x55a027ce1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 0x55a027dbcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 0x55a027dbcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45934/0 (socket says 172.21.15.79:45934) 2024-04-22T06:45:04.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 -- 172.21.15.79:0/114882425 learned_addr learned my addr 172.21.15.79:0/114882425 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dbd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027dbfa80 0x55a027dbd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 -- 172.21.15.79:0/114882425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027dbfa80 msgr2=0x55a027dbd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027dbfa80 0x55a027dbd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 -- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 msgr2=0x55a027dbd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dbd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 -- 172.21.15.79:0/114882425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a027a2d3f0 con 0x55a027db6580 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dbd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd61affd640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 0x55a027dbcaf0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00e500 tx=0x7fd60c00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de4614241caee1a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd5feff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd5feff5640 1 -- 172.21.15.79:0/114882425 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c012600 con 0x55a027db6580 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd5feff5640 1 -- 172.21.15.79:0/114882425 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd60c0127a0 con 0x55a027db6580 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6100054e0 con 0x55a027db6580 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.410+0000 7fd5feff5640 1 -- 172.21.15.79:0/114882425 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd60c022050 con 0x55a027db6580 2024-04-22T06:45:04.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 10 client.15855 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a0279ed0f0 con 0x55a027db6580 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd5feff5640 1 -- 172.21.15.79:0/114882425 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6100054e0 con 0x55a027db6580 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd5feff5640 1 -- 172.21.15.79:0/114882425 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd60c014b50 con 0x55a027db6580 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 10 client.15855 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 20 client.15855 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027dcfb10 0x55a027dd1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 4 client.15855 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a027d28c00 con 0x55a027dcfb10 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027dcfb10 0x55a027dd1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd5feff5640 10 client.15855 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.411+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027dcfb10 0x55a027dd1f80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf00b63213735336 server_cookie=9c801d904c2be958 in_seq=0 out_seq=0 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd5feff5640 1 -- 172.21.15.79:0/114882425 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a027d28c00 con 0x55a027dcfb10 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd5feff5640 10 client.15855 handle_command_reply: tid=0 2024-04-22T06:45:04.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 client.15855 shutdown 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 msgr2=0x55a027dbcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 0x55a027dbcaf0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00e500 tx=0x7fd60c00e9c0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 shutdown_connections 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027dcfb10 0x55a027dd1f80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dbd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027dbfa80 0x55a027dbd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 --2- 172.21.15.79:0/114882425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027db6580 0x55a027dbcaf0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd5feff5640 0 client.15855 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 >> 172.21.15.79:0/114882425 conn(0x55a027db0cb0 msgr2=0x55a027db2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 shutdown_connections 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 1 -- 172.21.15.79:0/114882425 wait complete. 2024-04-22T06:45:04.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.412+0000 7fd62418b740 20 client.15855 trim_cache size 0 max 16384 2024-04-22T06:45:04.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 Processor -- start 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 -- start start 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6580 0x55a027dbf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6ea0 0x55a027dcb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027dd7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1b26a0 con 0x55a027dbfa80 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1aff70 con 0x55a027db6580 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd62418b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1adde0 con 0x55a027db6ea0 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027dd7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027dd7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45942/0 (socket says 172.21.15.79:45942) 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 -- 172.21.15.79:0/4200958710 learned_addr learned my addr 172.21.15.79:0/4200958710 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61affd640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6580 0x55a027dbf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6ea0 0x55a027dcb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 -- 172.21.15.79:0/4200958710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6ea0 msgr2=0x55a027dcb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6ea0 0x55a027dcb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 -- 172.21.15.79:0/4200958710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6580 msgr2=0x55a027dbf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6580 0x55a027dbf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 -- 172.21.15.79:0/4200958710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a027db86f0 con 0x55a027dbfa80 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6ea0 0x55a027dcb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61affd640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6580 0x55a027dbf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027dd7850 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd614098200 tx=0x7fd6140bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bfd24b461235ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd600ff9640 1 -- 172.21.15.79:0/4200958710 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61409c4f0 con 0x55a027dbfa80 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd600ff9640 1 -- 172.21.15.79:0/4200958710 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd61409bdf0 con 0x55a027dbfa80 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.430+0000 7fd600ff9640 1 -- 172.21.15.79:0/4200958710 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b5930 con 0x55a027dbfa80 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 -- 172.21.15.79:0/4200958710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 msgr2=0x55a027dd7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027dd7850 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd614098200 tx=0x7fd6140bc040 comp rx=0 tx=0).stop 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 -- 172.21.15.79:0/4200958710 shutdown_connections 2024-04-22T06:45:04.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027dd7850 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6ea0 0x55a027dcb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 --2- 172.21.15.79:0/4200958710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6580 0x55a027dbf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 -- 172.21.15.79:0/4200958710 >> 172.21.15.79:0/4200958710 conn(0x55a027db0cb0 msgr2=0x55a027db72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 -- 172.21.15.79:0/4200958710 shutdown_connections 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 -- 172.21.15.79:0/4200958710 wait complete. 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 -- start start 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.431+0000 7fd62418b740 1 Processor -- start 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027dc7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dc8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027db8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1ae430 con 0x55a027dbfa80 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1af8a0 con 0x55a027db6ea0 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c1aeb00 con 0x55a027db6580 2024-04-22T06:45:04.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 10 client.0 _ll_register_callbacks cb 0x7fd60c00c1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027db8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027db8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45958/0 (socket says 172.21.15.79:45958) 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 -- 172.21.15.79:0/710312846 learned_addr learned my addr 172.21.15.79:0/710312846 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61affd640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027dc7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dc8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 -- 172.21.15.79:0/710312846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 msgr2=0x55a027dc7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027dc7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 -- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 msgr2=0x55a027dc8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dc8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 -- 172.21.15.79:0/710312846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a027db9740 con 0x55a027dbfa80 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61affd640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027dc7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61a7fc640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dc8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd61b7fe640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027db8f40 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd61400cfd0 tx=0x7fd61400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0cce262734d053 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:04.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd5feff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd5feff5640 1 -- 172.21.15.79:0/710312846 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140a4a60 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd5feff5640 1 -- 172.21.15.79:0/710312846 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fd61409c050 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd61409bdf0 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd5feff5640 1 -- 172.21.15.79:0/710312846 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd614081050 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 10 client.15861 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a027e35ae0 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.432+0000 7fd5feff5640 1 -- 172.21.15.79:0/710312846 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd61409bdf0 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd5feff5640 1 -- 172.21.15.79:0/710312846 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd61409b2d0 con 0x55a027dbfa80 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd62418b740 10 client.15861 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd62418b740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd62418b740 20 client.15861 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd62418b740 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027e36100 0x55a027e38520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd62418b740 4 client.15861 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a027d22a70 con 0x55a027e36100 2024-04-22T06:45:04.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd61affd640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027e36100 0x55a027e38520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd5feff5640 10 client.15861 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.433+0000 7fd61affd640 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027e36100 0x55a027e38520 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef63beedbd8c80d7 server_cookie=d7852b1cf191baf1 in_seq=0 out_seq=0 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd5feff5640 1 -- 172.21.15.79:0/710312846 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a027d22a70 con 0x55a027e36100 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd5feff5640 10 client.15861 handle_command_reply: tid=0 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 client.15861 shutdown 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 msgr2=0x55a027db8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027db8f40 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd61400cfd0 tx=0x7fd61400c490 comp rx=0 tx=0).stop 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 shutdown_connections 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a027e36100 0x55a027e38520 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a027db6ea0 0x55a027dc8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a027dbfa80 0x55a027db8f40 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd5feff5640 0 client.15861 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 --2- 172.21.15.79:0/710312846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a027db6580 0x55a027dc7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 >> 172.21.15.79:0/710312846 conn(0x55a027db0cb0 msgr2=0x55a027dbb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 shutdown_connections 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 1 -- 172.21.15.79:0/710312846 wait complete. 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd62418b740 20 client.15861 trim_cache size 0 max 16384 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: "718a5681-b4cd-4cb2-a984-7a8f112edb3b": { 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "718a5681-b4cd-4cb2-a984-7a8f112edb3b", 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:45:04.440 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:04.441 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:04.441 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:04.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6040a3ed0 msgr2=0x7fd6040a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6040a3ed0 0x7fd6040a6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd610009900 tx=0x7fd61002c780 comp rx=0 tx=0).stop 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 msgr2=0x7fd61c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 0x7fd61c101890 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd61400b9e0 tx=0x7fd61400bea0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 shutdown_connections 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd61c10c500 0x7fd61c101890 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61c103ae0 0x7fd61c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6040a3ed0 0x7fd6040a6390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 --2- 172.21.15.79:0/3847815018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd61c1031c0 0x7fd61c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.434+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 >> 172.21.15.79:0/3847815018 conn(0x7fd61c077fd0 msgr2=0x7fd61c0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.435+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 shutdown_connections 2024-04-22T06:45:04.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:04.435+0000 7fd6214c7640 1 -- 172.21.15.79:0/3847815018 wait complete. 2024-04-22T06:45:04.443 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:04 smithi193 ceph-mon[31815]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:04.443 INFO:tasks.cephfs.filesystem:scrub status for tag:718a5681-b4cd-4cb2-a984-7a8f112edb3b - {'path': '/', 'tag': '718a5681-b4cd-4cb2-a984-7a8f112edb3b', 'options': 'recursive,force'} 2024-04-22T06:45:04.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:04 smithi190 ceph-mon[31421]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:05 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:45:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:05 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:05 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:45:05.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:05 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:05.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:05 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:45:05.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:05 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:05.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:05 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:45:05.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:05 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:05 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:45:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:05 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:05 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:45:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:05 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:06.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:06 smithi193 ceph-mon[31815]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-22T06:45:06.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:06 smithi190 ceph-mon[31421]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-22T06:45:06.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:06 smithi079 ceph-mon[27946]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-22T06:45:08.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:08 smithi193 ceph-mon[31815]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-04-22T06:45:08.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:08 smithi190 ceph-mon[31421]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-04-22T06:45:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:08 smithi079 ceph-mon[27946]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-04-22T06:45:09.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:09 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:45:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:09 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:45:09.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:09 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:45:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:10 smithi193 ceph-mon[31815]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:45:10.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:10 smithi190 ceph-mon[31421]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:45:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:10 smithi079 ceph-mon[27946]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:45:12.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:12 smithi193 ceph-mon[31815]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-22T06:45:12.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:12 smithi190 ceph-mon[31421]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-22T06:45:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:12 smithi079 ceph-mon[27946]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-22T06:45:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 -- 172.21.15.79:0/2467616683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006b3c0 msgr2=0x7f6680106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 --2- 172.21.15.79:0/2467616683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006b3c0 0x7f6680106b80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6674009900 tx=0x7f667402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 -- 172.21.15.79:0/2467616683 shutdown_connections 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 --2- 172.21.15.79:0/2467616683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 0x7f668010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 --2- 172.21.15.79:0/2467616683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006b3c0 0x7f6680106b80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 --2- 172.21.15.79:0/2467616683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f668006aa10 0x7f668006adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 -- 172.21.15.79:0/2467616683 >> 172.21.15.79:0/2467616683 conn(0x7f6680076c50 msgr2=0x7f6680077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 -- 172.21.15.79:0/2467616683 shutdown_connections 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.910+0000 7f66868c8640 1 -- 172.21.15.79:0/2467616683 wait complete. 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.911+0000 7f66868c8640 1 Processor -- start 2024-04-22T06:45:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.911+0000 7f66868c8640 1 -- start start 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.911+0000 7f66868c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 0x7f6680198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.911+0000 7f66868c8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 0x7f6680198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f66868c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f668019cfd0 0x7f668019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f66868c8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f66801121a0 con 0x7f668006aa10 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f66868c8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6680112020 con 0x7f6680107260 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f66868c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6680112320 con 0x7f668019cfd0 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 0x7f6680198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 0x7f6680198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43718/0 (socket says 172.21.15.79:43718) 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 -- 172.21.15.79:0/549757861 learned_addr learned my addr 172.21.15.79:0/549757861 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667f7fe640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 0x7f6680198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 -- 172.21.15.79:0/549757861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f668019cfd0 msgr2=0x7f668019d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f668019cfd0 0x7f668019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 -- 172.21.15.79:0/549757861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 msgr2=0x7f6680198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 0x7f6680198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 -- 172.21.15.79:0/549757861 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668019dac0 con 0x7f668006aa10 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667f7fe640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 0x7f6680198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667ffff640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 0x7f6680198760 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f667000d770 tx=0x7f667000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667d7fa640 1 -- 172.21.15.79:0/549757861 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670017070 con 0x7f668006aa10 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667d7fa640 1 -- 172.21.15.79:0/549757861 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66700054d0 con 0x7f668006aa10 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668019ddb0 con 0x7f668006aa10 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f667d7fa640 1 -- 172.21.15.79:0/549757861 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670004c30 con 0x7f668006aa10 2024-04-22T06:45:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.912+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66801a5550 con 0x7f668006aa10 2024-04-22T06:45:12.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.915+0000 7f6689597740 1 Processor -- start 2024-04-22T06:45:12.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.915+0000 7f667d7fa640 1 -- 172.21.15.79:0/549757861 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f6670004e10 con 0x7f668006aa10 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.915+0000 7f6689597740 1 -- start start 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6689597740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e9058d0 0x564d9e904620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6689597740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e904fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6689597740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e90e0e0 0x564d9e9104d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6689597740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564d9e8698d0 con 0x564d9e904b60 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6689597740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564d9e806e80 con 0x564d9e90e0e0 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6689597740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d9e865890 con 0x564d9e9058d0 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e904fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e904fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43732/0 (socket says 172.21.15.79:43732) 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 -- 172.21.15.79:0/1976869921 learned_addr learned my addr 172.21.15.79:0/1976869921 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6684e65640 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e9058d0 0x564d9e904620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667f7fe640 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e90e0e0 0x564d9e9104d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 -- 172.21.15.79:0/1976869921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e9058d0 msgr2=0x564d9e904620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667d7fa640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f66680a3f70 0x7f66680a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e9058d0 0x564d9e904620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 -- 172.21.15.79:0/1976869921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e90e0e0 msgr2=0x564d9e9104d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e90e0e0 0x564d9e9104d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 -- 172.21.15.79:0/1976869921 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d9e828320 con 0x564d9e904b60 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6684e65640 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e9058d0 0x564d9e904620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f6684e65640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f66680a3f70 0x7f66680a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667ffff640 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e904fe0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f66700cdbf0 tx=0x7f66700d6f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad0a61d855280c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.916+0000 7f667d7fa640 1 -- 172.21.15.79:0/549757861 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f66700d0710 con 0x7f668006aa10 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f66667fc640 1 -- 172.21.15.79:0/1976869921 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670020030 con 0x564d9e904b60 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f66667fc640 1 -- 172.21.15.79:0/1976869921 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6670005680 con 0x564d9e904b60 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f66667fc640 1 -- 172.21.15.79:0/1976869921 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700b0e60 con 0x564d9e904b60 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 -- 172.21.15.79:0/1976869921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 msgr2=0x564d9e904fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6684e65640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f66680a3f70 0x7f66680a6430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6678005fd0 tx=0x7f66780096a0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e904fe0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f66700cdbf0 tx=0x7f66700d6f60 comp rx=0 tx=0).stop 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 -- 172.21.15.79:0/1976869921 shutdown_connections 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e90e0e0 0x564d9e9104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e904fe0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 --2- 172.21.15.79:0/1976869921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e9058d0 0x564d9e904620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 -- 172.21.15.79:0/1976869921 >> 172.21.15.79:0/1976869921 conn(0x564d9e8cbc90 msgr2=0x564d9e8ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 -- 172.21.15.79:0/1976869921 shutdown_connections 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.917+0000 7f6689597740 1 -- 172.21.15.79:0/1976869921 wait complete. 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 -- start start 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 Processor -- start 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e904b60 0x564d9e90db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 0x564d9e90ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564d9e910db0 con 0x564d9e9058d0 2024-04-22T06:45:12.923 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564d9e7f1eb0 con 0x564d9e904b60 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d9e910f30 con 0x564d9e90e0e0 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f6689597740 10 client.0 _ll_register_callbacks cb 0x564d9e81eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f667f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 0x564d9e90ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f667f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 0x564d9e90ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43742/0 (socket says 172.21.15.79:43742) 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f667f7fe640 1 -- 172.21.15.79:0/3313298097 learned_addr learned my addr 172.21.15.79:0/3313298097 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.918+0000 7f667ffff640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e904b60 0x564d9e90db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667f7fe640 1 -- 172.21.15.79:0/3313298097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 msgr2=0x564d9e90b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f6684e65640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667f7fe640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667f7fe640 1 -- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e904b60 msgr2=0x564d9e90db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667f7fe640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e904b60 0x564d9e90db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667f7fe640 1 -- 172.21.15.79:0/3313298097 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d9e57c210 con 0x564d9e9058d0 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667ffff640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e904b60 0x564d9e90db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f667f7fe640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 0x564d9e90ae90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6674005ab0 tx=0x7f667402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fb3b34ebed4c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f665f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f665f7fe640 1 -- 172.21.15.79:0/3313298097 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674005d50 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f665f7fe640 1 -- 172.21.15.79:0/3313298097 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f6674005220 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6670005680 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f665f7fe640 1 -- 172.21.15.79:0/3313298097 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f667404c050 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f6689597740 10 client.15873 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564d9e648620 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.919+0000 7f665f7fe640 1 -- 172.21.15.79:0/3313298097 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6670005680 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f665f7fe640 1 -- 172.21.15.79:0/3313298097 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f66740035f0 con 0x564d9e9058d0 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f6689597740 10 client.15873 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f6689597740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f6689597740 20 client.15873 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f6689597740 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e91e160 0x564d9e920590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f6689597740 4 client.15873 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d9e87d880 con 0x564d9e91e160 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f667ffff640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e91e160 0x564d9e920590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f665f7fe640 10 client.15873 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.920+0000 7f667ffff640 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e91e160 0x564d9e920590 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9d4835df6d5359 server_cookie=a1038b93db75857c in_seq=0 out_seq=0 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f665f7fe640 1 -- 172.21.15.79:0/3313298097 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d9e87d880 con 0x564d9e91e160 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f665f7fe640 10 client.15873 handle_command_reply: tid=0 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 client.15873 shutdown 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 msgr2=0x564d9e90ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 0x564d9e90ae90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6674005ab0 tx=0x7f667402ca10 comp rx=0 tx=0).stop 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 shutdown_connections 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e91e160 0x564d9e920590 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e9058d0 0x564d9e90ae90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f665f7fe640 0 client.15873 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 --2- 172.21.15.79:0/3313298097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e904b60 0x564d9e90db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 >> 172.21.15.79:0/3313298097 conn(0x564d9e8cbc90 msgr2=0x564d9e9015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 shutdown_connections 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.921+0000 7f6689597740 1 -- 172.21.15.79:0/3313298097 wait complete. 2024-04-22T06:45:12.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.922+0000 7f6689597740 20 client.15873 trim_cache size 0 max 16384 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 Processor -- start 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 -- start start 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e919cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f66801b0400 con 0x564d9e904b60 2024-04-22T06:45:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f66801adcd0 con 0x564d9e9058d0 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f6689597740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66801abb20 con 0x564d9e90e0e0 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f667ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e919cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f667ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e919cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43758/0 (socket says 172.21.15.79:43758) 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.936+0000 7f667ffff640 1 -- 172.21.15.79:0/47653311 learned_addr learned my addr 172.21.15.79:0/47653311 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667f7fe640 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667ffff640 1 -- 172.21.15.79:0/47653311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 msgr2=0x564d9e90db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667ffff640 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667ffff640 1 -- 172.21.15.79:0/47653311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 msgr2=0x564d9e90d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667ffff640 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667ffff640 1 -- 172.21.15.79:0/47653311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d9e9093e0 con 0x564d9e904b60 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667f7fe640 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f667ffff640 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e919cb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f66700cdd90 tx=0x7f66700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b27bfb1b0baca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f6665ffb640 1 -- 172.21.15.79:0/47653311 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700d6020 con 0x564d9e904b60 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f6665ffb640 1 -- 172.21.15.79:0/47653311 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6670005680 con 0x564d9e904b60 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.937+0000 7f6665ffb640 1 -- 172.21.15.79:0/47653311 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670020040 con 0x564d9e904b60 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- 172.21.15.79:0/47653311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 msgr2=0x564d9e919cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e919cb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f66700cdd90 tx=0x7f66700046c0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- 172.21.15.79:0/47653311 shutdown_connections 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e90e0e0 0x564d9e90db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- 172.21.15.79:0/47653311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e904b60 0x564d9e919cb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- 172.21.15.79:0/47653311 >> 172.21.15.79:0/47653311 conn(0x564d9e8cbc90 msgr2=0x564d9e906030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- 172.21.15.79:0/47653311 shutdown_connections 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- 172.21.15.79:0/47653311 wait complete. 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- start start 2024-04-22T06:45:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 Processor -- start 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e904b60 0x564d9e90eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 0x564d9e9151c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f66801ac190 con 0x564d9e90e0e0 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f66801ad600 con 0x564d9e9058d0 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.938+0000 7f6689597740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66801ac860 con 0x564d9e904b60 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6689597740 10 client.0 _ll_register_callbacks cb 0x7f6674009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 0x564d9e9151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 0x564d9e9151c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43760/0 (socket says 172.21.15.79:43760) 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 -- 172.21.15.79:0/1885884038 learned_addr learned my addr 172.21.15.79:0/1885884038 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f667ffff640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e904b60 0x564d9e90eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f667f7fe640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 -- 172.21.15.79:0/1885884038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e904b60 msgr2=0x564d9e90eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e904b60 0x564d9e90eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 -- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 msgr2=0x564d9e90efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 -- 172.21.15.79:0/1885884038 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d9e915800 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f667ffff640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e904b60 0x564d9e90eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6684e65640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 0x564d9e9151c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f667801e6e0 tx=0x7f667800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc936b53728b8f42 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f665f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f665f7fe640 1 -- 172.21.15.79:0/1885884038 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66780129c0 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f665f7fe640 1 -- 172.21.15.79:0/1885884038 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f6678012b60 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6670005680 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f665f7fe640 1 -- 172.21.15.79:0/1885884038 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6678012d60 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6689597740 10 client.15879 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.939+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564d9e984710 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f665f7fe640 1 -- 172.21.15.79:0/1885884038 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6670005680 con 0x564d9e90e0e0 2024-04-22T06:45:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f665f7fe640 1 -- 172.21.15.79:0/1885884038 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6678025020 con 0x564d9e90e0e0 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f6689597740 10 client.15879 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f6689597740 10 client.15879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f6689597740 20 client.15879 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f6689597740 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e984d20 0x564d9e987140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f6689597740 4 client.15879 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d9e8093d0 con 0x564d9e984d20 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f667ffff640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e984d20 0x564d9e987140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f665f7fe640 10 client.15879 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.940+0000 7f667ffff640 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e984d20 0x564d9e987140 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd5b8d1c95e02b server_cookie=b0c1529805286a6f in_seq=0 out_seq=0 2024-04-22T06:45:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.946+0000 7f665f7fe640 1 -- 172.21.15.79:0/1885884038 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d9e8093d0 con 0x564d9e984d20 2024-04-22T06:45:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.946+0000 7f665f7fe640 10 client.15879 handle_command_reply: tid=0 2024-04-22T06:45:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.946+0000 7f6689597740 1 client.15879 shutdown 2024-04-22T06:45:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 msgr2=0x564d9e9151c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 0x564d9e9151c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f667801e6e0 tx=0x7f667800cf90 comp rx=0 tx=0).stop 2024-04-22T06:45:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 shutdown_connections 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564d9e984d20 0x564d9e987140 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564d9e9058d0 0x564d9e90efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564d9e90e0e0 0x564d9e9151c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f665f7fe640 0 client.15879 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 --2- 172.21.15.79:0/1885884038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d9e904b60 0x564d9e90eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 >> 172.21.15.79:0/1885884038 conn(0x564d9e8cbc90 msgr2=0x564d9e909670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 shutdown_connections 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 1 -- 172.21.15.79:0/1885884038 wait complete. 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f6689597740 20 client.15879 trim_cache size 0 max 16384 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:45:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:45:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:45:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:45:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f66680a3f70 msgr2=0x7f66680a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f66868c8640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f66680a3f70 0x7f66680a6430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6678005fd0 tx=0x7f66780096a0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 msgr2=0x7f6680198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.947+0000 7f66868c8640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 0x7f6680198760 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f667000d770 tx=0x7f667000dc30 comp rx=0 tx=0).stop 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 shutdown_connections 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f66680a3f70 0x7f66680a6430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6680107260 0x7f6680198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f668019cfd0 0x7f668019d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 --2- 172.21.15.79:0/549757861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f668006aa10 0x7f6680198760 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 >> 172.21.15.79:0/549757861 conn(0x7f6680076c50 msgr2=0x7f66800ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 shutdown_connections 2024-04-22T06:45:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:12.948+0000 7f66868c8640 1 -- 172.21.15.79:0/549757861 wait complete. 2024-04-22T06:45:14.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:14 smithi193 ceph-mon[31815]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-22T06:45:14.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:14 smithi190 ceph-mon[31421]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-22T06:45:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:14 smithi079 ceph-mon[27946]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-22T06:45:16.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:16 smithi193 ceph-mon[31815]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:45:16.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:16 smithi190 ceph-mon[31421]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:45:16.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:16 smithi079 ceph-mon[27946]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:45:18.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:18 smithi193 ceph-mon[31815]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-22T06:45:18.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:18 smithi190 ceph-mon[31421]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-22T06:45:18.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:18 smithi079 ceph-mon[27946]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-22T06:45:19.780 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:19.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 -- 172.21.15.79:0/3231201692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b0102ef0 msgr2=0x7fe6b006a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 --2- 172.21.15.79:0/3231201692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b0102ef0 0x7fe6b006a640 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009900 tx=0x7fe6a402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 -- 172.21.15.79:0/3231201692 shutdown_connections 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 --2- 172.21.15.79:0/3231201692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6b010e830 0x7fe6b0110c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 --2- 172.21.15.79:0/3231201692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 0x7fe6b006b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 --2- 172.21.15.79:0/3231201692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b0102ef0 0x7fe6b006a640 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 -- 172.21.15.79:0/3231201692 >> 172.21.15.79:0/3231201692 conn(0x7fe6b0077e30 msgr2=0x7fe6b0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 -- 172.21.15.79:0/3231201692 shutdown_connections 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.890+0000 7fe6b7031640 1 -- 172.21.15.79:0/3231201692 wait complete. 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 Processor -- start 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 -- start start 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 0x7fe6b01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6b0102ef0 0x7fe6b01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:19.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 0x7fe6b019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b0114440 con 0x7fe6b010e830 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b01142c0 con 0x7fe6b0102ef0 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b7031640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b01145c0 con 0x7fe6b006ac10 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 0x7fe6b019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 0x7fe6b019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43784/0 (socket says 172.21.15.79:43784) 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 -- 172.21.15.79:0/486870378 learned_addr learned my addr 172.21.15.79:0/486870378 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6affff640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6b0102ef0 0x7fe6b01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b4dcd640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 0x7fe6b01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 -- 172.21.15.79:0/486870378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 msgr2=0x7fe6b01a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 0x7fe6b01a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 -- 172.21.15.79:0/486870378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6b0102ef0 msgr2=0x7fe6b01a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6b0102ef0 0x7fe6b01a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:19.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 -- 172.21.15.79:0/486870378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b019afd0 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b4dcd640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 0x7fe6b01a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.891+0000 7fe6b55ce640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 0x7fe6b019a800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800e9d0 tx=0x7fe6a800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.892+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a800cd80 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.892+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6a8005020 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.892+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b019b2c0 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.892+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a80116d0 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.892+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b01a74e0 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.893+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b01014e0 con 0x7fe6b010e830 2024-04-22T06:45:19.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.894+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fe6a80051d0 con 0x7fe6b010e830 2024-04-22T06:45:19.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.895+0000 7fe6adffb640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe6a00a3ed0 0x7fe6a00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:19.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.895+0000 7fe6b4dcd640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe6a00a3ed0 0x7fe6a00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:19.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.895+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fe6a80d0c70 con 0x7fe6b010e830 2024-04-22T06:45:19.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.895+0000 7fe6b4dcd640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe6a00a3ed0 0x7fe6a00a6390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4036040 tx=0x7fe6a4000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:19.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:19.900+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6a8016030 con 0x7fe6b010e830 2024-04-22T06:45:20.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.131+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6b01a7710 con 0x7fe6b010e830 2024-04-22T06:45:20.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.131+0000 7fe6adffb640 1 -- 172.21.15.79:0/486870378 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11878 (secure 0 0 0) 0x7fe6a809d100 con 0x7fe6b010e830 2024-04-22T06:45:20.130 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:45:20.130 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:44:23.666906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24440,"mds_2":24436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":2,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.193:6839/746968688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":746968688},{"type":"v1","addr":"172.21.15.193:6839","nonce":746968688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.190:6837/2879949884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2879949884},{"type":"v1","addr":"172.21.15.190:6837","nonce":2879949884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24436,24440]},"id":1}]} 2024-04-22T06:45:20.130 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 22 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe6a00a3ed0 msgr2=0x7fe6a00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe6a00a3ed0 0x7fe6a00a6390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4036040 tx=0x7fe6a4000bc0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 msgr2=0x7fe6b019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 0x7fe6b019a800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800e9d0 tx=0x7fe6a800ee90 comp rx=0 tx=0).stop 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 shutdown_connections 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe6a00a3ed0 0x7fe6a00a6390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6b0102ef0 0x7fe6b01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6b010e830 0x7fe6b019a800 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 --2- 172.21.15.79:0/486870378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6b006ac10 0x7fe6b01a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.133+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 >> 172.21.15.79:0/486870378 conn(0x7fe6b0077e30 msgr2=0x7fe6b010f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.134+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 shutdown_connections 2024-04-22T06:45:20.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.134+0000 7fe6b7031640 1 -- 172.21.15.79:0/486870378 wait complete. 2024-04-22T06:45:20.144 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id c807895f --release --await' 2024-04-22T06:45:20.145 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id c807895f --release --await 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 -- 172.21.15.79:0/204512882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c1016d0 msgr2=0x7f505c10be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 --2- 172.21.15.79:0/204512882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c1016d0 0x7f505c10be30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f504c009900 tx=0x7f504c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 -- 172.21.15.79:0/204512882 shutdown_connections 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 --2- 172.21.15.79:0/204512882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f505c10c4f0 0x7f505c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 --2- 172.21.15.79:0/204512882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c1016d0 0x7f505c10be30 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 --2- 172.21.15.79:0/204512882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c100d20 0x7f505c101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 -- 172.21.15.79:0/204512882 >> 172.21.15.79:0/204512882 conn(0x7f505c076150 msgr2=0x7f505c076560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.257 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 -- 172.21.15.79:0/204512882 shutdown_connections 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 -- 172.21.15.79:0/204512882 wait complete. 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.256+0000 7f50612ba640 1 Processor -- start 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 -- start start 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f505c100d20 0x7f505c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 0x7f505c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c19d000 0x7f505c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f505c112180 con 0x7f505c10c4f0 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f505c112000 con 0x7f505c19d000 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f50612ba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f505c112300 con 0x7f505c100d20 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 0x7f505c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 0x7f505c198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43794/0 (socket says 172.21.15.79:43794) 2024-04-22T06:45:20.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 -- 172.21.15.79:0/780472466 learned_addr learned my addr 172.21.15.79:0/780472466 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505affd640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f505c100d20 0x7f505c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505b7fe640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c19d000 0x7f505c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 -- 172.21.15.79:0/780472466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f505c100d20 msgr2=0x7f505c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f505c100d20 0x7f505c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 -- 172.21.15.79:0/780472466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c19d000 msgr2=0x7f505c19d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c19d000 0x7f505c19d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 -- 172.21.15.79:0/780472466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c19daf0 con 0x7f505c10c4f0 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505b7fe640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c19d000 0x7f505c19d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.257+0000 7f505a7fc640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 0x7f505c198c80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f504c036040 tx=0x7f504c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.258+0000 7f5053fff640 1 -- 172.21.15.79:0/780472466 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c043020 con 0x7f505c10c4f0 2024-04-22T06:45:20.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.258+0000 7f5053fff640 1 -- 172.21.15.79:0/780472466 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c02cbe0 con 0x7f505c10c4f0 2024-04-22T06:45:20.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.258+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505c19dd80 con 0x7f505c10c4f0 2024-04-22T06:45:20.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.258+0000 7f5053fff640 1 -- 172.21.15.79:0/780472466 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c003c30 con 0x7f505c10c4f0 2024-04-22T06:45:20.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.258+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f505c1a55a0 con 0x7f505c10c4f0 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5053fff640 1 -- 172.21.15.79:0/780472466 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f504c046020 con 0x7f505c10c4f0 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 Processor -- start 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 -- start start 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331418420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418960 0x5633314256e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 0x563331428010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56333137d500 con 0x563331425c20 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5633313236a0 con 0x563331418040 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5063f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5633313794c0 con 0x563331418960 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 0x563331428010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 0x563331428010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43808/0 (socket says 172.21.15.79:43808) 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f505a7fc640 1 -- 172.21.15.79:0/4234767892 learned_addr learned my addr 172.21.15.79:0/4234767892 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f5053fff640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f50480a3ed0 0x7f50480a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.261+0000 7f505b7fe640 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331418420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505b7fe640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f50480a3ed0 0x7f50480a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505affd640 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418960 0x5633314256e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f5053fff640 1 -- 172.21.15.79:0/780472466 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f504c0f2c60 con 0x7f505c10c4f0 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505a7fc640 1 -- 172.21.15.79:0/4234767892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418960 msgr2=0x5633314256e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505a7fc640 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418960 0x5633314256e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505a7fc640 1 -- 172.21.15.79:0/4234767892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 msgr2=0x563331418420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505a7fc640 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331418420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505a7fc640 1 -- 172.21.15.79:0/4234767892 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633313425e0 con 0x563331425c20 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505a7fc640 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 0x563331428010 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f504c0be800 tx=0x7f504c003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4c7930e4218915 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:20.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f505b7fe640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f50480a3ed0 0x7f50480a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5054009fd0 tx=0x7f5054004a40 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f50517fa640 1 -- 172.21.15.79:0/4234767892 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c0be460 con 0x563331425c20 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f50517fa640 1 -- 172.21.15.79:0/4234767892 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c0bf200 con 0x563331425c20 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.262+0000 7f50517fa640 1 -- 172.21.15.79:0/4234767892 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c050d50 con 0x563331425c20 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 -- 172.21.15.79:0/4234767892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 msgr2=0x563331428010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 0x563331428010 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f504c0be800 tx=0x7f504c003760 comp rx=0 tx=0).stop 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 -- 172.21.15.79:0/4234767892 shutdown_connections 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331425c20 0x563331428010 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418960 0x5633314256e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 --2- 172.21.15.79:0/4234767892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331418420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 -- 172.21.15.79:0/4234767892 >> 172.21.15.79:0/4234767892 conn(0x563331412a40 msgr2=0x563331412e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 -- 172.21.15.79:0/4234767892 shutdown_connections 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 -- 172.21.15.79:0/4234767892 wait complete. 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 -- start start 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 Processor -- start 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.263+0000 7f5063f85740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331422b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331423050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x563331423590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5633314287a0 con 0x563331418960 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563331336a60 con 0x563331418040 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563331428920 con 0x563331425c20 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 10 client.0 _ll_register_callbacks cb 0x56333108f110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:20.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331423050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331423050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43822/0 (socket says 172.21.15.79:43822) 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 -- 172.21.15.79:0/204944207 learned_addr learned my addr 172.21.15.79:0/204944207 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 -- 172.21.15.79:0/204944207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 msgr2=0x563331423590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x563331423590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 -- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 msgr2=0x563331422b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331422b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 -- 172.21.15.79:0/204944207 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56333108f920 con 0x563331418960 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f505a7fc640 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331423050 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f504c0be800 tx=0x7f504c0be530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d99aaead0e4684 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f50427fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f50427fc640 1 -- 172.21.15.79:0/204944207 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c0beb00 con 0x563331418960 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f50427fc640 1 -- 172.21.15.79:0/204944207 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f504c0df120 con 0x563331418960 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f504c0bf200 con 0x563331418960 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f50427fc640 1 -- 172.21.15.79:0/204944207 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f504c0bb170 con 0x563331418960 2024-04-22T06:45:20.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 10 client.15897 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56333115bd30 con 0x563331418960 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.264+0000 7f50427fc640 1 -- 172.21.15.79:0/204944207 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c0bf200 con 0x563331418960 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f50427fc640 1 -- 172.21.15.79:0/204944207 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f504c0fa040 con 0x563331418960 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f5063f85740 10 client.15897 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f5063f85740 10 client.15897 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f5063f85740 20 client.15897 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f5063f85740 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5633314359f0 0x563331437de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f5063f85740 4 client.15897 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563331350a10 con 0x5633314359f0 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.265+0000 7f505affd640 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5633314359f0 0x563331437de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f50427fc640 10 client.15897 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f505affd640 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5633314359f0 0x563331437de0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9dcce8457849db4 server_cookie=4a727743181f1554 in_seq=0 out_seq=0 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f50427fc640 1 -- 172.21.15.79:0/204944207 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563331350a10 con 0x5633314359f0 2024-04-22T06:45:20.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f50427fc640 10 client.15897 handle_command_reply: tid=0 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 client.15897 shutdown 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 msgr2=0x563331423050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331423050 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f504c0be800 tx=0x7f504c0be530 comp rx=0 tx=0).stop 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 shutdown_connections 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5633314359f0 0x563331437de0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x563331423590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331423050 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f50427fc640 0 client.15897 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 --2- 172.21.15.79:0/204944207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331422b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.266+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 >> 172.21.15.79:0/204944207 conn(0x563331412a40 msgr2=0x563331414790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.267+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 shutdown_connections 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.267+0000 7f5063f85740 1 -- 172.21.15.79:0/204944207 wait complete. 2024-04-22T06:45:20.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.267+0000 7f5063f85740 20 client.15897 trim_cache size 0 max 16384 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 Processor -- start 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 -- start start 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418040 0x563331425240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x5633314311e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331425c20 0x5633314213e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f505c1ae190 con 0x563331418960 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f505c1aba60 con 0x563331425c20 2024-04-22T06:45:20.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.288+0000 7f5063f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f505c1a98b0 con 0x563331418040 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x5633314311e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x5633314311e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43826/0 (socket says 172.21.15.79:43826) 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 -- 172.21.15.79:0/1732236044 learned_addr learned my addr 172.21.15.79:0/1732236044 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505affd640 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418040 0x563331425240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 -- 172.21.15.79:0/1732236044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418040 msgr2=0x563331425240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418040 0x563331425240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 -- 172.21.15.79:0/1732236044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331425c20 msgr2=0x5633314213e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331425c20 0x5633314213e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 -- 172.21.15.79:0/1732236044 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563331417c80 con 0x563331418960 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f505a7fc640 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x5633314311e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f504c0bea90 tx=0x7f504c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a4375509ca3e7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f5050ff9640 1 -- 172.21.15.79:0/1732236044 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c0bb310 con 0x563331418960 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f5050ff9640 1 -- 172.21.15.79:0/1732236044 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c0bf200 con 0x563331418960 2024-04-22T06:45:20.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.289+0000 7f5050ff9640 1 -- 172.21.15.79:0/1732236044 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c0ba970 con 0x563331418960 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- 172.21.15.79:0/1732236044 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 msgr2=0x5633314311e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x5633314311e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f504c0bea90 tx=0x7f504c009680 comp rx=0 tx=0).stop 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- 172.21.15.79:0/1732236044 shutdown_connections 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331425c20 0x5633314213e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x5633314311e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- 172.21.15.79:0/1732236044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331418040 0x563331425240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- 172.21.15.79:0/1732236044 >> 172.21.15.79:0/1732236044 conn(0x563331412a40 msgr2=0x563331414720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- 172.21.15.79:0/1732236044 shutdown_connections 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- 172.21.15.79:0/1732236044 wait complete. 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- start start 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 Processor -- start 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331439010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331439550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x56333149abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f505c1a9f20 con 0x563331418960 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f505c1ab390 con 0x563331418040 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.290+0000 7f5063f85740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f505c1aa5f0 con 0x563331425c20 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f5063f85740 10 client.0 _ll_register_callbacks cb 0x7f5054009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331439550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331439550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43830/0 (socket says 172.21.15.79:43830) 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 -- 172.21.15.79:0/3411098541 learned_addr learned my addr 172.21.15.79:0/3411098541 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505affd640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331439010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505b7fe640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x56333149abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 -- 172.21.15.79:0/3411098541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 msgr2=0x56333149abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x56333149abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 -- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 msgr2=0x563331439010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331439010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 -- 172.21.15.79:0/3411098541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633314319c0 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505affd640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331439010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f505a7fc640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331439550 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f504c0be800 tx=0x7f504c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35035279062a7ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f50427fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f50427fc640 1 -- 172.21.15.79:0/3411098541 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c0bb310 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f50427fc640 1 -- 172.21.15.79:0/3411098541 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7f504c0bab30 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f504c0bf200 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f50427fc640 1 -- 172.21.15.79:0/3411098541 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f504c0beb00 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f5063f85740 10 client.15903 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5633314319c0 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.291+0000 7f50427fc640 1 -- 172.21.15.79:0/3411098541 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c0bf200 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f50427fc640 1 -- 172.21.15.79:0/3411098541 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f504c0f0540 con 0x563331418960 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f5063f85740 10 client.15903 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f5063f85740 10 client.15903 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f5063f85740 20 client.15903 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:20.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f5063f85740 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56333149c5f0 0x56333149ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:20.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f5063f85740 4 client.15903 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c807895f", "release": true, "await": true}] 2024-04-22T06:45:20.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c807895f", "release": true, "await": true}) -- 0x56333138af20 con 0x56333149c5f0 2024-04-22T06:45:20.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f505affd640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56333149c5f0 0x56333149ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:20.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f50427fc640 10 client.15903 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:20.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.292+0000 7f505affd640 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56333149c5f0 0x56333149ea10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9365634a7d2035c5 server_cookie=5dbd231ce1a56422 in_seq=0 out_seq=0 2024-04-22T06:45:20.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f50427fc640 1 -- 172.21.15.79:0/3411098541 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x56333138af20 con 0x56333149c5f0 2024-04-22T06:45:20.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f50427fc640 10 client.15903 handle_command_reply: tid=0 2024-04-22T06:45:20.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 client.15903 shutdown 2024-04-22T06:45:20.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 msgr2=0x563331439550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331439550 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f504c0be800 tx=0x7f504c009260 comp rx=0 tx=0).stop 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 shutdown_connections 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56333149c5f0 0x56333149ea10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563331425c20 0x56333149abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f50427fc640 0 client.15903 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563331418960 0x563331439550 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 --2- 172.21.15.79:0/3411098541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563331418040 0x563331439010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 >> 172.21.15.79:0/3411098541 conn(0x563331412a40 msgr2=0x563331413d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 shutdown_connections 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 1 -- 172.21.15.79:0/3411098541 wait complete. 2024-04-22T06:45:20.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.294+0000 7f5063f85740 20 client.15903 trim_cache size 0 max 16384 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 22, 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 4, 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "c807895f": { 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 4, 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:45:20.299 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 84.2, 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:20.300 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f50480a3ed0 msgr2=0x7f50480a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f50480a3ed0 0x7f50480a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5054009fd0 tx=0x7f5054004a40 comp rx=0 tx=0).stop 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 msgr2=0x7f505c198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 0x7f505c198c80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f504c036040 tx=0x7f504c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 shutdown_connections 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f50480a3ed0 0x7f50480a6390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c19d000 0x7f505c19d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f505c10c4f0 0x7f505c198c80 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 --2- 172.21.15.79:0/780472466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f505c100d20 0x7f505c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:20.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 >> 172.21.15.79:0/780472466 conn(0x7f505c076150 msgr2=0x7f505c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:20.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 shutdown_connections 2024-04-22T06:45:20.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:20.295+0000 7f50612ba640 1 -- 172.21.15.79:0/780472466 wait complete. 2024-04-22T06:45:20.304 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: c807895f, seconds elapsed: 25.2 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-mon[31815]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fa384066700 time 2024-04-22T06:45:20.294460+0000 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa39669f3dd] 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.351 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: clone() 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.294+0000 7fa384066700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fa384066700 time 2024-04-22T06:45:20.294460+0000 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa39669f3dd] 2024-04-22T06:45:20.352 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-mon[31421]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:45:20.354 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.354 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: clone() 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: *** Caught signal (Aborted) ** 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: in thread 7fa384066700 thread_name:quiesce.agt 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa395427d20] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: gsignal() 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: abort() 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa39669f437] 2024-04-22T06:45:20.355 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 13: clone() 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.296+0000 7fa384066700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: in thread 7fa384066700 thread_name:quiesce.agt 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa395427d20] 2024-04-22T06:45:20.356 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: gsignal() 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: abort() 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa39669f437] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 13: clone() 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: -1> 2024-04-22T06:45:20.294+0000 7fa384066700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fa384066700 time 2024-04-22T06:45:20.294460+0000 2024-04-22T06:45:20.357 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa39669f3dd] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: clone() 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 0> 2024-04-22T06:45:20.296+0000 7fa384066700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: in thread 7fa384066700 thread_name:quiesce.agt 2024-04-22T06:45:20.358 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa395427d20] 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: gsignal() 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: abort() 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa39669f437] 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.359 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 13: clone() 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: -1> 2024-04-22T06:45:20.294+0000 7fa384066700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fa384066700 time 2024-04-22T06:45:20.294460+0000 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa39669f3dd] 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.360 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: clone() 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 0> 2024-04-22T06:45:20.296+0000 7fa384066700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: in thread 7fa384066700 thread_name:quiesce.agt 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa395427d20] 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2: gsignal() 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 3: abort() 2024-04-22T06:45:20.361 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa39669f437] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fa39669f5a3] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557511ce5e9c] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557511ce5f9f] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557511c140d3] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557511c1403c] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557511e93d3f] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557511efc560] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa39541d1ca] 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 13: clone() 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[63259]: 2024-04-22T06:45:20.362 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:20 smithi193 podman[66140]: 2024-04-22 06:45:20.353261916 +0000 UTC m=+0.022949412 container died 3bd07a4cc9decb0fd6d11a94fd700138099d527de1e364656eb3a385ed99a888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f404ed5e700 time 2024-04-22T06:45:20.294237+0000 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40613973dd] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.363 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: clone() 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.295+0000 7f404ed5e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f404ed5e700 time 2024-04-22T06:45:20.294237+0000 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40613973dd] 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.364 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: clone() 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: *** Caught signal (Aborted) ** 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: in thread 7f404ed5e700 thread_name:quiesce.agt 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f406011fd20] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: gsignal() 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: abort() 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061397437] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.365 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 13: clone() 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.296+0000 7f404ed5e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: in thread 7f404ed5e700 thread_name:quiesce.agt 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f406011fd20] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: gsignal() 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: abort() 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061397437] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.366 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 13: clone() 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: -1> 2024-04-22T06:45:20.295+0000 7f404ed5e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f404ed5e700 time 2024-04-22T06:45:20.294237+0000 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40613973dd] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.367 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: clone() 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 0> 2024-04-22T06:45:20.296+0000 7f404ed5e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: in thread 7f404ed5e700 thread_name:quiesce.agt 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f406011fd20] 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: gsignal() 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: abort() 2024-04-22T06:45:20.368 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061397437] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 13: clone() 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: -1> 2024-04-22T06:45:20.295+0000 7f404ed5e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f404ed5e700 time 2024-04-22T06:45:20.294237+0000 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.369 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40613973dd] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: clone() 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 0> 2024-04-22T06:45:20.296+0000 7f404ed5e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: in thread 7f404ed5e700 thread_name:quiesce.agt 2024-04-22T06:45:20.370 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f406011fd20] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2: gsignal() 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 3: abort() 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f4061397437] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f40613975a3] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x559232cbfe9c] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x559232cbff9f] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x559232bee0d3] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x559232bee03c] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x559232e6dd3f] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x559232ed6560] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f40601151ca] 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 13: clone() 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:20.371 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[62027]: 2024-04-22T06:45:20.372 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:20 smithi190 podman[65211]: 2024-04-22 06:45:20.353650804 +0000 UTC m=+0.022554306 container died 443444e919816eb05088a7336453e5d206b4ff4935db23a0bf4fd3a0a904a366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:45:20.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:20 smithi079 ceph-mon[27946]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:45:21.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:21 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/486870378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:21.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:21 smithi193 podman[66140]: 2024-04-22 06:45:21.071286867 +0000 UTC m=+0.740974383 container remove 3bd07a4cc9decb0fd6d11a94fd700138099d527de1e364656eb3a385ed99a888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T06:45:21.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:21 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:45:21.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:21 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-22T06:45:21.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:21 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/486870378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:21.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:21 smithi190 podman[65211]: 2024-04-22 06:45:21.054637403 +0000 UTC m=+0.723540932 container remove 443444e919816eb05088a7336453e5d206b4ff4935db23a0bf4fd3a0a904a366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:45:21.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:21 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:45:21.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:21 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:45:21.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:21 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/486870378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:22.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:22 smithi193 ceph-mon[31815]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:45:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:22 smithi190 ceph-mon[31421]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:45:22.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:22 smithi079 ceph-mon[27946]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:45:24.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:24 smithi079 ceph-mon[27946]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:45:24.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:24 smithi193 ceph-mon[31815]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:45:24.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:24 smithi190 ceph-mon[31421]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:45:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/1761309053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a8101120 msgr2=0x7fb2a8101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/1761309053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a8101120 0x7fb2a8101500 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb298009900 tx=0x7fb29802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/1761309053 shutdown_connections 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/1761309053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a81061a0 0x7fb2a810dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/1761309053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2a8101ad0 0x7fb2a8105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/1761309053 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a8101120 0x7fb2a8101500 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/1761309053 >> 172.21.15.79:0/1761309053 conn(0x7fb2a8078060 msgr2=0x7fb2a8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/1761309053 shutdown_connections 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.035+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/1761309053 wait complete. 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 Processor -- start 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 -- start start 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a8101120 0x7fb2a8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2a81061a0 0x7fb2a8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 0x7fb2a819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8111100 con 0x7fb2a819cf80 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8110f80 con 0x7fb2a8101120 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2ae4c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8111280 con 0x7fb2a81061a0 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2aca5e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 0x7fb2a819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2aca5e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 0x7fb2a819d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36450/0 (socket says 172.21.15.79:36450) 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2aca5e640 1 -- 172.21.15.79:0/2876330526 learned_addr learned my addr 172.21.15.79:0/2876330526 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.036+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2a81061a0 0x7fb2a8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a8101120 0x7fb2a8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2aca5e640 1 -- 172.21.15.79:0/2876330526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2a81061a0 msgr2=0x7fb2a8198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2aca5e640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2a81061a0 0x7fb2a8198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2aca5e640 1 -- 172.21.15.79:0/2876330526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a8101120 msgr2=0x7fb2a8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2aca5e640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a8101120 0x7fb2a8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2aca5e640 1 -- 172.21.15.79:0/2876330526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a819da70 con 0x7fb2a819cf80 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a8101120 0x7fb2a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2aca5e640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 0x7fb2a819d430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000cab0 tx=0x7fb2a000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2a57fa640 1 -- 172.21.15.79:0/2876330526 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a000ae20 con 0x7fb2a819cf80 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2a57fa640 1 -- 172.21.15.79:0/2876330526 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a000f900 con 0x7fb2a819cf80 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a819dd60 con 0x7fb2a819cf80 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2a57fa640 1 -- 172.21.15.79:0/2876330526 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a00189f0 con 0x7fb2a819cf80 2024-04-22T06:45:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.037+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a81a5500 con 0x7fb2a819cf80 2024-04-22T06:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.039+0000 7fb2a57fa640 1 -- 172.21.15.79:0/2876330526 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fb2a000fab0 con 0x7fb2a819cf80 2024-04-22T06:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.039+0000 7fb2b1192740 1 Processor -- start 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 -- start start 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8608d0 0x556cbd85f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd85fb60 0x556cbd85ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd86b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x556cbd7c48d0 con 0x556cbd8690e0 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556cbd761e80 con 0x556cbd8608d0 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2b1192740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556cbd7c0890 con 0x556cbd85fb60 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a57fa640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb2900a3f70 0x7fb2900a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd86b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd86b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36460/0 (socket says 172.21.15.79:36460) 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2931250032 learned_addr learned my addr 172.21.15.79:0/2931250032 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2aca5e640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb2900a3f70 0x7fb2900a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd85fb60 0x556cbd85ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8608d0 0x556cbd85f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a57fa640 1 -- 172.21.15.79:0/2876330526 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fb2a00d11d0 con 0x7fb2a819cf80 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2931250032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd85fb60 msgr2=0x556cbd85ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd85fb60 0x556cbd85ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2931250032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8608d0 msgr2=0x556cbd85f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8608d0 0x556cbd85f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2931250032 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cbd783320 con 0x556cbd8690e0 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.040+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd86b4d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00bbb0 tx=0x7fb29c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37606acfd8880c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2aca5e640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb2900a3f70 0x7fb2900a6430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00ce7b0 tx=0x7fb2a006e9e0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb28e7fc640 1 -- 172.21.15.79:0/2931250032 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c005020 con 0x556cbd8690e0 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb28e7fc640 1 -- 172.21.15.79:0/2931250032 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb29c0047a0 con 0x556cbd8690e0 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb28e7fc640 1 -- 172.21.15.79:0/2931250032 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c0106c0 con 0x556cbd8690e0 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 -- 172.21.15.79:0/2931250032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 msgr2=0x556cbd86b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd86b4d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00bbb0 tx=0x7fb29c0041d0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 -- 172.21.15.79:0/2931250032 shutdown_connections 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd86b4d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd85fb60 0x556cbd85ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2931250032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8608d0 0x556cbd85f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 -- 172.21.15.79:0/2931250032 >> 172.21.15.79:0/2931250032 conn(0x556cbd826c90 msgr2=0x556cbd85a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 -- 172.21.15.79:0/2931250032 shutdown_connections 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.041+0000 7fb2b1192740 1 -- 172.21.15.79:0/2931250032 wait complete. 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 -- start start 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 Processor -- start 2024-04-22T06:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd868b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd865ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd866620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x556cbd86be10 con 0x556cbd85fb60 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x556cbd74ceb0 con 0x556cbd8690e0 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556cbd86bf90 con 0x556cbd8608d0 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2b1192740 10 client.0 _ll_register_callbacks cb 0x556cbd779b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd868b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd868b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36462/0 (socket says 172.21.15.79:36462) 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2393737571 learned_addr learned my addr 172.21.15.79:0/2393737571 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd866620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2393737571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 msgr2=0x556cbd865ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd865ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 msgr2=0x556cbd866620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd866620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.042+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2393737571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cbd4d7210 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd868b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb29c004770 tx=0x7fb29c00b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874f7bcd49560698 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2877fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2877fe640 1 -- 172.21.15.79:0/2393737571 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c005200 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2877fe640 1 -- 172.21.15.79:0/2393737571 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fb29c0053a0 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29c0047a0 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2877fe640 1 -- 172.21.15.79:0/2393737571 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb29c0055a0 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 10 client.15915 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556cbd5a3620 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2877fe640 1 -- 172.21.15.79:0/2393737571 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb29c0047a0 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2877fe640 1 -- 172.21.15.79:0/2393737571 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb29c01d3f0 con 0x556cbd85fb60 2024-04-22T06:45:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 10 client.15915 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 20 client.15915 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8790c0 0x556cbd87b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 4 client.15915 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.043+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cbd7d8880 con 0x556cbd8790c0 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.044+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8790c0 0x556cbd87b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.044+0000 7fb2877fe640 10 client.15915 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.044+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8790c0 0x556cbd87b4b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=706e3e30e9b2c8bc server_cookie=cae8f6955d892857 in_seq=0 out_seq=0 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2877fe640 1 -- 172.21.15.79:0/2393737571 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cbd7d8880 con 0x556cbd8790c0 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2877fe640 10 client.15915 handle_command_reply: tid=0 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 client.15915 shutdown 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 msgr2=0x556cbd868b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd868b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb29c004770 tx=0x7fb29c00b420 comp rx=0 tx=0).stop 2024-04-22T06:45:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 shutdown_connections 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8790c0 0x556cbd87b4b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd866620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2877fe640 0 client.15915 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd865ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2393737571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd868b50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 >> 172.21.15.79:0/2393737571 conn(0x556cbd826c90 msgr2=0x556cbd85c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 shutdown_connections 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 1 -- 172.21.15.79:0/2393737571 wait complete. 2024-04-22T06:45:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.045+0000 7fb2b1192740 20 client.15915 trim_cache size 0 max 16384 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 Processor -- start 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 -- start start 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd85fb60 0x556cbd8657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd85c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd868610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81b02c0 con 0x556cbd8690e0 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81adb90 con 0x556cbd85fb60 2024-04-22T06:45:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2b1192740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81ab9e0 con 0x556cbd8608d0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd868610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd868610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36468/0 (socket says 172.21.15.79:36468) 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2861833253 learned_addr learned my addr 172.21.15.79:0/2861833253 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.060+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd85c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd85fb60 0x556cbd8657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2861833253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 msgr2=0x556cbd85c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd85c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2861833253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd85fb60 msgr2=0x556cbd8657c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd85fb60 0x556cbd8657c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2861833253 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cbd85aa60 con 0x556cbd8690e0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd868610 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb29c006010 tx=0x7fb29c004e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749533500c6484f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb28dffb640 1 -- 172.21.15.79:0/2861833253 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c037070 con 0x556cbd8690e0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb28dffb640 1 -- 172.21.15.79:0/2861833253 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb29c0047a0 con 0x556cbd8690e0 2024-04-22T06:45:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.061+0000 7fb28dffb640 1 -- 172.21.15.79:0/2861833253 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c004210 con 0x556cbd8690e0 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- 172.21.15.79:0/2861833253 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 msgr2=0x556cbd868610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd868610 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb29c006010 tx=0x7fb29c004e50 comp rx=0 tx=0).stop 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- 172.21.15.79:0/2861833253 shutdown_connections 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd8690e0 0x556cbd868610 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd85c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2861833253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd85fb60 0x556cbd8657c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- 172.21.15.79:0/2861833253 >> 172.21.15.79:0/2861833253 conn(0x556cbd826c90 msgr2=0x556cbd85bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- 172.21.15.79:0/2861833253 shutdown_connections 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- 172.21.15.79:0/2861833253 wait complete. 2024-04-22T06:45:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- start start 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 Processor -- start 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd86acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd871220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd863160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81ac050 con 0x556cbd85fb60 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81ad4c0 con 0x556cbd8690e0 2024-04-22T06:45:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81ac720 con 0x556cbd8608d0 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.062+0000 7fb2b1192740 10 client.0 _ll_register_callbacks cb 0x7fb298009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd86acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd86acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36472/0 (socket says 172.21.15.79:36472) 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2737294192 learned_addr learned my addr 172.21.15.79:0/2737294192 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd871220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2737294192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 msgr2=0x556cbd871220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd871220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 msgr2=0x556cbd863160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd863160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 -- 172.21.15.79:0/2737294192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cbd871ab0 con 0x556cbd85fb60 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd871220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2a77fe640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd86acf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb29c005d00 tx=0x7fb29c00b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ba3aed53fe5228 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2877fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2877fe640 1 -- 172.21.15.79:0/2737294192 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c037070 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29c0047a0 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2877fe640 1 -- 172.21.15.79:0/2737294192 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8469+0+0 (secure 0 0 0) 0x7fb29c004120 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2877fe640 1 -- 172.21.15.79:0/2737294192 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb29c01d3f0 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2b1192740 10 client.15921 fetch_fsmap learned FSMap version 22 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.063+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556cbd869af0 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2877fe640 1 -- 172.21.15.79:0/2737294192 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb29c0047a0 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2877fe640 1 -- 172.21.15.79:0/2737294192 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb29c03a020 con 0x556cbd85fb60 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2b1192740 10 client.15921 fetch_fsmap finished waiting for FSMap version 22 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2b1192740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2b1192740 20 client.15921 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8df2c0 0x556cbd8e16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2b1192740 4 client.15921 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556cbd7643d0 con 0x556cbd8df2c0 2024-04-22T06:45:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8df2c0 0x556cbd8e16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2877fe640 10 client.15921 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.064+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8df2c0 0x556cbd8e16b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b01d7baffeed379 server_cookie=fc42bf05930f1ea1 in_seq=0 out_seq=0 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2877fe640 1 -- 172.21.15.79:0/2737294192 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556cbd7643d0 con 0x556cbd8df2c0 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2877fe640 10 client.15921 handle_command_reply: tid=0 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 client.15921 shutdown 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 msgr2=0x556cbd86acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd86acf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb29c005d00 tx=0x7fb29c00b420 comp rx=0 tx=0).stop 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 shutdown_connections 2024-04-22T06:45:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x556cbd8df2c0 0x556cbd8e16b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556cbd8690e0 0x556cbd863160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cbd8608d0 0x556cbd871220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2877fe640 0 client.15921 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 --2- 172.21.15.79:0/2737294192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x556cbd85fb60 0x556cbd86acf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 >> 172.21.15.79:0/2737294192 conn(0x556cbd826c90 msgr2=0x556cbd85c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 shutdown_connections 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.068+0000 7fb2b1192740 1 -- 172.21.15.79:0/2737294192 wait complete. 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2b1192740 20 client.15921 trim_cache size 0 max 16384 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:45:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:45:26.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb2900a3f70 msgr2=0x7fb2900a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb2900a3f70 0x7fb2900a6430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00ce7b0 tx=0x7fb2a006e9e0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 msgr2=0x7fb2a819d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 0x7fb2a819d430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000cab0 tx=0x7fb2a000cf70 comp rx=0 tx=0).stop 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 shutdown_connections 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2a819cf80 0x7fb2a819d430 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2a81061a0 0x7fb2a8198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb2900a3f70 0x7fb2900a6430 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 --2- 172.21.15.79:0/2876330526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2a8101120 0x7fb2a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 >> 172.21.15.79:0/2876330526 conn(0x7fb2a8078060 msgr2=0x7fb2a8104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 shutdown_connections 2024-04-22T06:45:26.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:26.069+0000 7fb2ae4c1640 1 -- 172.21.15.79:0/2876330526 wait complete. 2024-04-22T06:45:26.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:26 smithi193 ceph-mon[31815]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:45:26.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:26 smithi190 ceph-mon[31421]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:45:26.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:26 smithi079 ceph-mon[27946]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:45:28.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:28 smithi193 ceph-mon[31815]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:45:28.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:28 smithi190 ceph-mon[31421]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:45:28.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:28 smithi079 ceph-mon[27946]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:45:30.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:30 smithi193 ceph-mon[31815]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T06:45:30.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:30 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:30 smithi190 ceph-mon[31421]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T06:45:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:30 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:30.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:30 smithi079 ceph-mon[27946]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-22T06:45:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:30 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:31.522 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:31 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:31 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:31 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:31.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:31 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:31.523 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:31 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.678 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:45:31.678 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 systemd[1]: Stopped Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:31.678 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 systemd[1]: Starting Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:45:31.679 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:31 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.679 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:31 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.679 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:31 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:31.679 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:31 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:31.679 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:31 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 systemd[1]: Stopped Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:31.680 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 systemd[1]: Starting Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:45:31.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 podman[66369]: 2024-04-22 06:45:31.680550485 +0000 UTC m=+0.114200403 container create 3de8b91a108f4e85ea2257de0be1348b3df44cd4faef4252d4c63b5ff26928b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:45:31.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 podman[66369]: 2024-04-22 06:45:31.584615782 +0000 UTC m=+0.018265664 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:45:31.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 podman[66369]: 2024-04-22 06:45:31.925150157 +0000 UTC m=+0.358800029 container init 3de8b91a108f4e85ea2257de0be1348b3df44cd4faef4252d4c63b5ff26928b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T06:45:31.940 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 podman[66369]: 2024-04-22 06:45:31.928501385 +0000 UTC m=+0.362151265 container start 3de8b91a108f4e85ea2257de0be1348b3df44cd4faef4252d4c63b5ff26928b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T06:45:31.947 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 podman[65445]: 2024-04-22 06:45:31.584763558 +0000 UTC m=+0.017554831 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:45:31.947 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 podman[65445]: 2024-04-22 06:45:31.707802407 +0000 UTC m=+0.140593659 container create 5436cf65340ec100dfc76cefc7948e0edf921cbeb5b89cb3759591b34ce9ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T06:45:32.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 podman[65445]: 2024-04-22 06:45:31.949835492 +0000 UTC m=+0.382626750 container init 5436cf65340ec100dfc76cefc7948e0edf921cbeb5b89cb3759591b34ce9ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:45:32.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 podman[65445]: 2024-04-22 06:45:31.95347682 +0000 UTC m=+0.386268074 container start 5436cf65340ec100dfc76cefc7948e0edf921cbeb5b89cb3759591b34ce9ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:45:32.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-mds[65464]: -- 172.21.15.190:0/628773855 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56199bf41a40 con 0x56199bf14800 2024-04-22T06:45:32.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: starting mds.h at 2024-04-22T06:45:32.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:32 smithi190 bash[65445]: 5436cf65340ec100dfc76cefc7948e0edf921cbeb5b89cb3759591b34ce9ec94 2024-04-22T06:45:32.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:45:32 smithi190 systemd[1]: Started Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:32.266 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: starting mds.i at 2024-04-22T06:45:32.266 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:32 smithi193 bash[66369]: 3de8b91a108f4e85ea2257de0be1348b3df44cd4faef4252d4c63b5ff26928b2 2024-04-22T06:45:32.266 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:45:32 smithi193 systemd[1]: Started Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: daemon mds.i restarted 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:boot 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: daemon mds.h restarted 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:boot 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,2=f=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:45:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:32 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=f=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: daemon mds.i restarted 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:boot 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:45:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: daemon mds.h restarted 2024-04-22T06:45:32.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:45:32.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:boot 2024-04-22T06:45:32.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-22T06:45:32.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,2=f=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:45:32.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:45:32.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:32 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=f=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: daemon mds.i restarted 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:boot 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:45:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: daemon mds.h restarted 2024-04-22T06:45:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:45:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:boot 2024-04-22T06:45:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-22T06:45:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,2=f=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:45:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:45:32.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:32 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=f=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:34.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:34 smithi193 ceph-mon[31815]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-22T06:45:34.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:34 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:resolve 2024-04-22T06:45:34.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:34 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:resolve 2024-04-22T06:45:34.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:34 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=f=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:34.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:34.444 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:34 smithi190 ceph-mon[31421]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-22T06:45:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:34 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:resolve 2024-04-22T06:45:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:34 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:resolve 2024-04-22T06:45:34.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:34 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=f=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:34.464 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:34 smithi079 ceph-mon[27946]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-22T06:45:34.464 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:34 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:resolve 2024-04-22T06:45:34.464 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:34 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:resolve 2024-04-22T06:45:34.464 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:34 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=f=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:34.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/2823147488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98101a30 msgr2=0x7f4c98105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/2823147488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98101a30 0x7f4c98105a70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:34.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/2823147488 shutdown_connections 2024-04-22T06:45:34.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/2823147488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98106150 0x7f4c9810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/2823147488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c98101a30 0x7f4c98105a70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/2823147488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98101080 0x7f4c98101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/2823147488 >> 172.21.15.79:0/2823147488 conn(0x7f4c98078000 msgr2=0x7f4c98078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/2823147488 shutdown_connections 2024-04-22T06:45:34.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.550+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/2823147488 wait complete. 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 Processor -- start 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 -- start start 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98101080 0x7f4c981986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98106150 0x7f4c98198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 0x7f4c9819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981121d0 con 0x7f4c9819cfb0 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98112050 con 0x7f4c98106150 2024-04-22T06:45:34.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c9dfdb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c98112350 con 0x7f4c98101080 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 0x7f4c9819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 0x7f4c9819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49854/0 (socket says 172.21.15.79:49854) 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c97fff640 1 -- 172.21.15.79:0/3641840632 learned_addr learned my addr 172.21.15.79:0/3641840632 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.551+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98106150 0x7f4c98198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c977fe640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98101080 0x7f4c981986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c97fff640 1 -- 172.21.15.79:0/3641840632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98101080 msgr2=0x7f4c981986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c97fff640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98101080 0x7f4c981986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c97fff640 1 -- 172.21.15.79:0/3641840632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98106150 msgr2=0x7f4c98198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c97fff640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98106150 0x7f4c98198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c97fff640 1 -- 172.21.15.79:0/3641840632 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9819daa0 con 0x7f4c9819cfb0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c97fff640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 0x7f4c9819d460 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000eb10 tx=0x7f4c9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c94ff9640 1 -- 172.21.15.79:0/3641840632 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90004b20 con 0x7f4c9819cfb0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9819dd90 con 0x7f4c9819cfb0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c981a55a0 con 0x7f4c9819cfb0 2024-04-22T06:45:34.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c94ff9640 1 -- 172.21.15.79:0/3641840632 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c90004cc0 con 0x7f4c9819cfb0 2024-04-22T06:45:34.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.552+0000 7f4c94ff9640 1 -- 172.21.15.79:0/3641840632 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90010690 con 0x7f4c9819cfb0 2024-04-22T06:45:34.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.554+0000 7f4c94ff9640 1 -- 172.21.15.79:0/3641840632 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4c90005020 con 0x7f4c9819cfb0 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.555+0000 7f4ca0bc5740 1 Processor -- start 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.555+0000 7f4c94ff9640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c800a3ed0 0x7f4c800a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.555+0000 7f4ca0bc5740 1 -- start start 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c800a3ed0 0x7f4c800a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267a4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55da267091e0 con 0x55da267a4ea0 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55da266a6390 con 0x55da267ada80 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da267051a0 con 0x55da267a4580 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c94ff9640 1 -- 172.21.15.79:0/3641840632 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f4c900d10e0 con 0x7f4c9819cfb0 2024-04-22T06:45:34.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49856/0 (socket says 172.21.15.79:49856) 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 -- 172.21.15.79:0/1491262861 learned_addr learned my addr 172.21.15.79:0/1491262861 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c977fe640 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c97fff640 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c800a3ed0 0x7f4c800a6390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c005ec0 tx=0x7f4c8c004310 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 -- 172.21.15.79:0/1491262861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 msgr2=0x55da267a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 -- 172.21.15.79:0/1491262861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 msgr2=0x55da267afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 -- 172.21.15.79:0/1491262861 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da264e57d0 con 0x55da267a4ea0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.556+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267ad540 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c04a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74d61637536ca10 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4c7dffb640 1 -- 172.21.15.79:0/1491262861 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c049070 con 0x55da267a4ea0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4c7dffb640 1 -- 172.21.15.79:0/1491262861 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c052070 con 0x55da267a4ea0 2024-04-22T06:45:34.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4c7dffb640 1 -- 172.21.15.79:0/1491262861 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c004cb0 con 0x55da267a4ea0 2024-04-22T06:45:34.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/1491262861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 msgr2=0x55da267ad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267ad540 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c04a000 comp rx=0 tx=0).stop 2024-04-22T06:45:34.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/1491262861 shutdown_connections 2024-04-22T06:45:34.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267ad540 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/1491262861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/1491262861 >> 172.21.15.79:0/1491262861 conn(0x55da2679ecb0 msgr2=0x55da2679f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/1491262861 shutdown_connections 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.557+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/1491262861 wait complete. 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 -- start start 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 Processor -- start 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 0x55da267aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267ab020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267ab560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55da267b0810 con 0x55da267a4580 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55da266c22f0 con 0x55da267ada80 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da267b0990 con 0x55da267a4ea0 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.558+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 0x55da267aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 0x55da267aaae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49864/0 (socket says 172.21.15.79:49864) 2024-04-22T06:45:34.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 -- 172.21.15.79:0/2163243667 learned_addr learned my addr 172.21.15.79:0/2163243667 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c97fff640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4ca0bc5740 10 client.0 _ll_register_callbacks cb 0x55da266cf580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 -- 172.21.15.79:0/2163243667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 msgr2=0x55da267ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 -- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 msgr2=0x55da267ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 -- 172.21.15.79:0/2163243667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2641b3f0 con 0x55da267a4580 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c977fe640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 0x55da267aaae0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4c88013710 tx=0x7f4c88013b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bac5e11142c0bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c76ffd640 1 -- 172.21.15.79:0/2163243667 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88020060 con 0x55da267a4580 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c76ffd640 1 -- 172.21.15.79:0/2163243667 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f4c88011dd0 con 0x55da267a4580 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c8c052070 con 0x55da267a4580 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4c76ffd640 1 -- 172.21.15.79:0/2163243667 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4c8800f450 con 0x55da267a4580 2024-04-22T06:45:34.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4ca0bc5740 10 client.15945 fetch_fsmap learned FSMap version 28 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.559+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da263db0f0 con 0x55da267a4580 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4c76ffd640 1 -- 172.21.15.79:0/2163243667 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c052070 con 0x55da267a4580 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4c76ffd640 1 -- 172.21.15.79:0/2163243667 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4c8801a030 con 0x55da267a4580 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4ca0bc5740 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4ca0bc5740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4ca0bc5740 20 client.15945 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da267bdab0 0x55da267bff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4ca0bc5740 4 client.15945 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.560+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da26716c00 con 0x55da267bdab0 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.561+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da267bdab0 0x55da267bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.561+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da267bdab0 0x55da267bff70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18751ec000977e3d server_cookie=2fd6f14a519b9368 in_seq=0 out_seq=0 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.561+0000 7f4c76ffd640 10 client.15945 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4c76ffd640 1 -- 172.21.15.79:0/2163243667 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da26716c00 con 0x55da267bdab0 2024-04-22T06:45:34.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4c76ffd640 10 client.15945 handle_command_reply: tid=0 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 client.15945 shutdown 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 msgr2=0x55da267aaae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 0x55da267aaae0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4c88013710 tx=0x7f4c88013b10 comp rx=0 tx=0).stop 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 shutdown_connections 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da267bdab0 0x55da267bff70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/2163243667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4580 0x55da267aaae0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 >> 172.21.15.79:0/2163243667 conn(0x55da2679ecb0 msgr2=0x55da267a0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4c76ffd640 0 client.15945 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 shutdown_connections 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/2163243667 wait complete. 2024-04-22T06:45:34.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.562+0000 7f4ca0bc5740 20 client.15945 trim_cache size 0 max 16384 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.579+0000 7f4ca0bc5740 1 Processor -- start 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.579+0000 7f4ca0bc5740 1 -- start start 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267ad210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981addf0 con 0x55da267a4ea0 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981ae4c0 con 0x55da267ada80 2024-04-22T06:45:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981abc60 con 0x55da267a4580 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267b96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49874/0 (socket says 172.21.15.79:49874) 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 -- 172.21.15.79:0/979828579 learned_addr learned my addr 172.21.15.79:0/979828579 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c977fe640 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 -- 172.21.15.79:0/979828579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 msgr2=0x55da267ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c97fff640 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 -- 172.21.15.79:0/979828579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 msgr2=0x55da267c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 -- 172.21.15.79:0/979828579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da267a4cd0 con 0x55da267a4ea0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c97fff640 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.580+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267b96d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c004860 tx=0x7f4c8c0047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75637b5bd6e37d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4c7d7fa640 1 -- 172.21.15.79:0/979828579 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c060070 con 0x55da267a4ea0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4c7d7fa640 1 -- 172.21.15.79:0/979828579 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c05b030 con 0x55da267a4ea0 2024-04-22T06:45:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4c7d7fa640 1 -- 172.21.15.79:0/979828579 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c049040 con 0x55da267a4ea0 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/979828579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 msgr2=0x55da267b96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267b96d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c004860 tx=0x7f4c8c0047f0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/979828579 shutdown_connections 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267ada80 0x55da267c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267a4ea0 0x55da267b96d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/979828579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4580 0x55da267ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/979828579 >> 172.21.15.79:0/979828579 conn(0x55da2679ecb0 msgr2=0x55da267a93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/979828579 shutdown_connections 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.581+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/979828579 wait complete. 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 -- start start 2024-04-22T06:45:34.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 Processor -- start 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267a4580 0x55da267b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267b6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 0x55da267a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981ac2b0 con 0x55da267ada80 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981ad720 con 0x55da267a4580 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c981ac980 con 0x55da267a4ea0 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.582+0000 7f4ca0bc5740 10 client.0 _ll_register_callbacks cb 0x7f4c88009730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 0x55da267a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 0x55da267a6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49888/0 (socket says 172.21.15.79:49888) 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 -- 172.21.15.79:0/3955577505 learned_addr learned my addr 172.21.15.79:0/3955577505 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c96ffd640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c977fe640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267a4580 0x55da267b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 -- 172.21.15.79:0/3955577505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 msgr2=0x55da267b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 -- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267a4580 msgr2=0x55da267b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267a4580 0x55da267b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 -- 172.21.15.79:0/3955577505 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da267a76d0 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c97fff640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 0x55da267a6ed0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4c9009cac0 tx=0x7f4c9009cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99a4a49baa07507 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c76ffd640 1 -- 172.21.15.79:0/3955577505 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c900bb060 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c76ffd640 1 -- 172.21.15.79:0/3955577505 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f4c9009d230 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c8c05b030 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.583+0000 7f4c76ffd640 1 -- 172.21.15.79:0/3955577505 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4c900b5030 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 10 client.15951 fetch_fsmap learned FSMap version 28 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da26823ab0 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4c76ffd640 1 -- 172.21.15.79:0/3955577505 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8c05b030 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4c76ffd640 1 -- 172.21.15.79:0/3955577505 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4c9007bbe0 con 0x55da267ada80 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:34.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 20 client.15951 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da26824180 0x55da268265a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 4 client.15951 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da26710a70 con 0x55da26824180 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.584+0000 7f4c977fe640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da26824180 0x55da268265a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4c76ffd640 10 client.15951 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4c977fe640 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da26824180 0x55da268265a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74abc2669e22650c server_cookie=e86f9dd84e0ce8fe in_seq=0 out_seq=0 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4c76ffd640 1 -- 172.21.15.79:0/3955577505 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55da26710a70 con 0x55da26824180 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4c76ffd640 10 client.15951 handle_command_reply: tid=0 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 client.15951 shutdown 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 msgr2=0x55da267a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 0x55da267a6ed0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4c9009cac0 tx=0x7f4c9009cc50 comp rx=0 tx=0).stop 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 shutdown_connections 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55da26824180 0x55da268265a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4c76ffd640 0 client.15951 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:34.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55da267ada80 0x55da267a6ed0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da267a4ea0 0x55da267b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 --2- 172.21.15.79:0/3955577505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55da267a4580 0x55da267b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 >> 172.21.15.79:0/3955577505 conn(0x55da2679ecb0 msgr2=0x55da267aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 shutdown_connections 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 1 -- 172.21.15.79:0/3955577505 wait complete. 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.585+0000 7f4ca0bc5740 20 client.15951 trim_cache size 0 max 16384 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:45:34.589 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:34.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c800a3ed0 msgr2=0x7f4c800a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c800a3ed0 0x7f4c800a6390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c005ec0 tx=0x7f4c8c004310 comp rx=0 tx=0).stop 2024-04-22T06:45:34.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 msgr2=0x7f4c9819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 0x7f4c9819d460 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000eb10 tx=0x7f4c9000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 shutdown_connections 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4c800a3ed0 0x7f4c800a6390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c98106150 0x7f4c98198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4c9819cfb0 0x7f4c9819d460 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 --2- 172.21.15.79:0/3641840632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c98101080 0x7f4c981986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 >> 172.21.15.79:0/3641840632 conn(0x7f4c98078000 msgr2=0x7f4c980fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 shutdown_connections 2024-04-22T06:45:34.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.586+0000 7f4c9dfdb640 1 -- 172.21.15.79:0/3641840632 wait complete. 2024-04-22T06:45:34.595 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:45:34.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:34.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 -- 172.21.15.79:0/2206551447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 msgr2=0x7f14901083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 --2- 172.21.15.79:0/2206551447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f14901083f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1480009900 tx=0x7f148002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 -- 172.21.15.79:0/2206551447 shutdown_connections 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 --2- 172.21.15.79:0/2206551447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14901028f0 0x7f149010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 --2- 172.21.15.79:0/2206551447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f14901023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 --2- 172.21.15.79:0/2206551447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f14901083f0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 -- 172.21.15.79:0/2206551447 >> 172.21.15.79:0/2206551447 conn(0x7f14900793e0 msgr2=0x7f14900797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 -- 172.21.15.79:0/2206551447 shutdown_connections 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 -- 172.21.15.79:0/2206551447 wait complete. 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 Processor -- start 2024-04-22T06:45:34.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.771+0000 7f149597e640 1 -- start start 2024-04-22T06:45:34.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f149597e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f1490198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f149597e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14901028f0 0x7f1490198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f149597e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f149019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f149597e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1490112180 con 0x7f1490108010 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f149597e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1490112000 con 0x7f14901028f0 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f149597e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1490112300 con 0x7f1490101f50 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f1490198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f1490198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:48996/0 (socket says 172.21.15.79:48996) 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 -- 172.21.15.79:0/3305280430 learned_addr learned my addr 172.21.15.79:0/3305280430 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148f7fe640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f149019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148e7fc640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14901028f0 0x7f1490198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 -- 172.21.15.79:0/3305280430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14901028f0 msgr2=0x7f1490198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14901028f0 0x7f1490198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 -- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 msgr2=0x7f149019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f149019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148effd640 1 -- 172.21.15.79:0/3305280430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149019d650 con 0x7f1490101f50 2024-04-22T06:45:34.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.772+0000 7f148f7fe640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f149019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.773+0000 7f148effd640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f1490198660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1480005940 tx=0x7f14800045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.773+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480043020 con 0x7f1490101f50 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.773+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148002dba0 con 0x7f1490101f50 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.773+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149019d8e0 con 0x7f1490101f50 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.773+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480003930 con 0x7f1490101f50 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.773+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1490074180 con 0x7f1490101f50 2024-04-22T06:45:34.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.774+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1490068de0 con 0x7f1490101f50 2024-04-22T06:45:34.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.775+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f1480046020 con 0x7f1490101f50 2024-04-22T06:45:34.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.776+0000 7f149497c640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14780a3ed0 0x7f14780a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:34.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.776+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f14800f30e0 con 0x7f1490101f50 2024-04-22T06:45:34.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.776+0000 7f148e7fc640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14780a3ed0 0x7f14780a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:34.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.776+0000 7f148e7fc640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14780a3ed0 0x7f14780a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1490100270 tx=0x7f148400f820 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:34.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:34.780+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f148004d020 con 0x7f1490101f50 2024-04-22T06:45:35.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.104+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f149010a400 con 0x7f1490101f50 2024-04-22T06:45:35.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.105+0000 7f149497c640 1 -- 172.21.15.79:0/3305280430 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11858 (secure 0 0 0) 0x7f14800bf5d0 con 0x7f1490101f50 2024-04-22T06:45:35.103 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:45:35.104 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":25261,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":25265,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:45:33.998961+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24413,"mds_2":24406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":24,"state":"up:rejoin","state_seq":56,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":26,"state":"up:rejoin","state_seq":57,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467]},"id":1}]} 2024-04-22T06:45:35.104 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 29 2024-04-22T06:45:35.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14780a3ed0 msgr2=0x7f14780a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14780a3ed0 0x7f14780a6390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1490100270 tx=0x7f148400f820 comp rx=0 tx=0).stop 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 msgr2=0x7f1490198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f1490198660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1480005940 tx=0x7f14800045e0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 shutdown_connections 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14780a3ed0 0x7f14780a6390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f14901028f0 0x7f1490198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1490108010 0x7f149019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 --2- 172.21.15.79:0/3305280430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1490101f50 0x7f1490198660 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 >> 172.21.15.79:0/3305280430 conn(0x7f14900793e0 msgr2=0x7f1490100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 shutdown_connections 2024-04-22T06:45:35.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.108+0000 7f149597e640 1 -- 172.21.15.79:0/3305280430 wait complete. 2024-04-22T06:45:35.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:45:35.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 -- 172.21.15.79:0/3509679875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc10c510 msgr2=0x7f79cc10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 --2- 172.21.15.79:0/3509679875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc10c510 0x7f79cc10e9b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009ca0 tx=0x7f79c402c230 comp rx=0 tx=0).stop 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 -- 172.21.15.79:0/3509679875 shutdown_connections 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 --2- 172.21.15.79:0/3509679875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc10c510 0x7f79cc10e9b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 --2- 172.21.15.79:0/3509679875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 0x7f79cc10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 --2- 172.21.15.79:0/3509679875 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc1031d0 0x7f79cc1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 -- 172.21.15.79:0/3509679875 >> 172.21.15.79:0/3509679875 conn(0x7f79cc077f50 msgr2=0x7f79cc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 -- 172.21.15.79:0/3509679875 shutdown_connections 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.225+0000 7f79d3dca640 1 -- 172.21.15.79:0/3509679875 wait complete. 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 Processor -- start 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 -- start start 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 0x7f79cc1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 0x7f79cc1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc10c510 0x7f79cc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc112180 con 0x7f79cc1031d0 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc112000 con 0x7f79cc10c510 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d3dca640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc112300 con 0x7f79cc103af0 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d1b66640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 0x7f79cc1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d1b66640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 0x7f79cc1a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49924/0 (socket says 172.21.15.79:49924) 2024-04-22T06:45:35.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.226+0000 7f79d1b66640 1 -- 172.21.15.79:0/1516029755 learned_addr learned my addr 172.21.15.79:0/1516029755 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1365640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 0x7f79cc1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1b66640 1 -- 172.21.15.79:0/1516029755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 msgr2=0x7f79cc1a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d2367640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc10c510 0x7f79cc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1b66640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 0x7f79cc1a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1b66640 1 -- 172.21.15.79:0/1516029755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc10c510 msgr2=0x7f79cc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1b66640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc10c510 0x7f79cc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1b66640 1 -- 172.21.15.79:0/1516029755 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc19af20 con 0x7f79cc1031d0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1365640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 0x7f79cc1a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d2367640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc10c510 0x7f79cc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d1b66640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 0x7f79cc1a05a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f79c800d950 tx=0x7f79c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79c2ffd640 1 -- 172.21.15.79:0/1516029755 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c8017070 con 0x7f79cc1031d0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79c2ffd640 1 -- 172.21.15.79:0/1516029755 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c8009c30 con 0x7f79cc1031d0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79cc19b210 con 0x7f79cc1031d0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.227+0000 7f79c2ffd640 1 -- 172.21.15.79:0/1516029755 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c800f620 con 0x7f79cc1031d0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.228+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79cc1a7520 con 0x7f79cc1031d0 2024-04-22T06:45:35.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.230+0000 7f79c2ffd640 1 -- 172.21.15.79:0/1516029755 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f79c8010460 con 0x7f79cc1031d0 2024-04-22T06:45:35.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.231+0000 7f79d69b6740 1 Processor -- start 2024-04-22T06:45:35.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79c2ffd640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f79b80a3ed0 0x7f79b80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 -- start start 2024-04-22T06:45:35.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc61d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc626430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d2367640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f79b80a3ed0 0x7f79b80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc628e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5639cc5821f0 con 0x5639cc61de80 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5639cc51f4b0 con 0x5639cc61d4d0 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d69b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5639cc57e1b0 con 0x5639cc626a00 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79c2ffd640 1 -- 172.21.15.79:0/1516029755 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f79c80d01b0 con 0x7f79cc1031d0 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc626430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc626430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49930/0 (socket says 172.21.15.79:49930) 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 -- 172.21.15.79:0/615980489 learned_addr learned my addr 172.21.15.79:0/615980489 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1b66640 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc61d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d2367640 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc628e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 -- 172.21.15.79:0/615980489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 msgr2=0x5639cc628e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc628e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 -- 172.21.15.79:0/615980489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 msgr2=0x5639cc61d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc61d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d1365640 1 -- 172.21.15.79:0/615980489 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639cc5111b0 con 0x5639cc61de80 2024-04-22T06:45:35.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.232+0000 7f79d2367640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f79b80a3ed0 0x7f79b80a6390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f79c4037040 tx=0x7f79c4032340 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.233+0000 7f79d1365640 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc626430 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00bb60 tx=0x7f79bc00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a626987dc73659e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.233+0000 7f79b7fff640 1 -- 172.21.15.79:0/615980489 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00be60 con 0x5639cc61de80 2024-04-22T06:45:35.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.233+0000 7f79b7fff640 1 -- 172.21.15.79:0/615980489 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79bc017ad0 con 0x5639cc61de80 2024-04-22T06:45:35.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.233+0000 7f79b7fff640 1 -- 172.21.15.79:0/615980489 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc00fba0 con 0x5639cc61de80 2024-04-22T06:45:35.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 -- 172.21.15.79:0/615980489 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 msgr2=0x5639cc626430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc626430 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00bb60 tx=0x7f79bc00d710 comp rx=0 tx=0).stop 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 -- 172.21.15.79:0/615980489 shutdown_connections 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc628e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc626430 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 --2- 172.21.15.79:0/615980489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc61d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 -- 172.21.15.79:0/615980489 >> 172.21.15.79:0/615980489 conn(0x5639cc617ba0 msgr2=0x5639cc617f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 -- 172.21.15.79:0/615980489 shutdown_connections 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 -- 172.21.15.79:0/615980489 wait complete. 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.234+0000 7f79d69b6740 1 -- start start 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 Processor -- start 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc6262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61de80 0x5639cc623670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 0x5639cc623bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5639cc629970 con 0x5639cc626a00 2024-04-22T06:45:35.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5639cc53b500 con 0x5639cc61d4d0 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5639cc629af0 con 0x5639cc61de80 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d69b6740 10 client.0 _ll_register_callbacks cb 0x5639cc509b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d2367640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 0x5639cc623bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d1b66640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc6262e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d2367640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 0x5639cc623bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49940/0 (socket says 172.21.15.79:49940) 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d1b66640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc6262e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:45992/0 (socket says 172.21.15.79:45992) 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d2367640 1 -- 172.21.15.79:0/2220132883 learned_addr learned my addr 172.21.15.79:0/2220132883 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.235+0000 7f79d1365640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61de80 0x5639cc623670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d2367640 1 -- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61de80 msgr2=0x5639cc623670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d2367640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61de80 0x5639cc623670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d2367640 1 -- 172.21.15.79:0/2220132883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 msgr2=0x5639cc6262e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d2367640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc6262e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d2367640 1 -- 172.21.15.79:0/2220132883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639cc2943f0 con 0x5639cc626a00 2024-04-22T06:45:35.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d1b66640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc6262e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d2367640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 0x5639cc623bd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f79c4003f70 tx=0x7f79c402c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2a2905c12fb65e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79b57fa640 1 -- 172.21.15.79:0/2220132883 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4046070 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79bc017ad0 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79b57fa640 1 -- 172.21.15.79:0/2220132883 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f79c4032840 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79b57fa640 1 -- 172.21.15.79:0/2220132883 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f79c4059050 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d69b6740 10 client.15969 fetch_fsmap learned FSMap version 29 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.236+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5639cc35e7d0 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79b57fa640 1 -- 172.21.15.79:0/2220132883 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79bc017ad0 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79b57fa640 1 -- 172.21.15.79:0/2220132883 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f79c4042cd0 con 0x5639cc626a00 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d69b6740 10 client.15969 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d69b6740 10 client.15969 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d69b6740 20 client.15969 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d69b6740 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc6370b0 0x5639cc639510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d69b6740 4 client.15969 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 --> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639cc521930 con 0x5639cc6370b0 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.237+0000 7f79d1b66640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc6370b0 0x5639cc639510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.238+0000 7f79b57fa640 10 client.15969 ms_handle_connect on v2:172.21.15.193:6834/3775212644 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.238+0000 7f79d1b66640 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc6370b0 0x5639cc639510 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c955238dc3d10e68 server_cookie=e0df1815a9d3af83 in_seq=0 out_seq=0 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.238+0000 7f79b57fa640 1 -- 172.21.15.79:0/2220132883 <== mds.2 v2:172.21.15.193:6834/3775212644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639cc521930 con 0x5639cc6370b0 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.238+0000 7f79b57fa640 10 client.15969 handle_command_reply: tid=0 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 client.15969 shutdown 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 msgr2=0x5639cc623bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 0x5639cc623bd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f79c4003f70 tx=0x7f79c402c230 comp rx=0 tx=0).stop 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 shutdown_connections 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc6370b0 0x5639cc639510 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc626a00 0x5639cc623bd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79b57fa640 0 client.15969 ms_handle_reset on v2:172.21.15.193:6834/3775212644 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61de80 0x5639cc623670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 --2- 172.21.15.79:0/2220132883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61d4d0 0x5639cc6262e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 >> 172.21.15.79:0/2220132883 conn(0x5639cc617ba0 msgr2=0x5639cc6198f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 shutdown_connections 2024-04-22T06:45:35.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 1 -- 172.21.15.79:0/2220132883 wait complete. 2024-04-22T06:45:35.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.239+0000 7f79d69b6740 20 client.15969 trim_cache size 0 max 16384 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.264+0000 7f79d69b6740 1 Processor -- start 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.264+0000 7f79d69b6740 1 -- start start 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 0x5639cc632860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61de80 0x5639cc625da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc6262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d69b6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1aff70 con 0x5639cc61d4d0 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d69b6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1b0640 con 0x5639cc61de80 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d69b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1addc0 con 0x5639cc626a00 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 0x5639cc632860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 0x5639cc632860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49942/0 (socket says 172.21.15.79:49942) 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 -- 172.21.15.79:0/936630025 learned_addr learned my addr 172.21.15.79:0/936630025 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d2367640 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc6262e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1365640 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61de80 0x5639cc625da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 -- 172.21.15.79:0/936630025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 msgr2=0x5639cc6262e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc6262e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 -- 172.21.15.79:0/936630025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61de80 msgr2=0x5639cc625da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61de80 0x5639cc625da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 -- 172.21.15.79:0/936630025 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639cc629a80 con 0x5639cc61d4d0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d2367640 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc6262e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.265+0000 7f79d1b66640 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 0x5639cc632860 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f79c809bde0 tx=0x7f79c8098220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127367cae3793642 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.266+0000 7f79b77fe640 1 -- 172.21.15.79:0/936630025 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c8020040 con 0x5639cc61d4d0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.266+0000 7f79b77fe640 1 -- 172.21.15.79:0/936630025 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c809e070 con 0x5639cc61d4d0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.266+0000 7f79b77fe640 1 -- 172.21.15.79:0/936630025 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c80cd9f0 con 0x5639cc61d4d0 2024-04-22T06:45:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 -- 172.21.15.79:0/936630025 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 msgr2=0x5639cc632860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 0x5639cc632860 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f79c809bde0 tx=0x7f79c8098220 comp rx=0 tx=0).stop 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 -- 172.21.15.79:0/936630025 shutdown_connections 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc61de80 0x5639cc625da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc626a00 0x5639cc6262e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 --2- 172.21.15.79:0/936630025 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61d4d0 0x5639cc632860 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 -- 172.21.15.79:0/936630025 >> 172.21.15.79:0/936630025 conn(0x5639cc617ba0 msgr2=0x5639cc623630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 -- 172.21.15.79:0/936630025 shutdown_connections 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 -- 172.21.15.79:0/936630025 wait complete. 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 -- start start 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.267+0000 7f79d69b6740 1 Processor -- start 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61d4d0 0x5639cc62ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc62a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 0x5639cc62a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1ae430 con 0x5639cc61de80 2024-04-22T06:45:35.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1af8a0 con 0x5639cc626a00 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1aeb00 con 0x5639cc61d4d0 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d69b6740 10 client.0 _ll_register_callbacks cb 0x7f79c4003e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d1365640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc62a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d2367640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 0x5639cc62a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d2367640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 0x5639cc62a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:45998/0 (socket says 172.21.15.79:45998) 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d2367640 1 -- 172.21.15.79:0/513662030 learned_addr learned my addr 172.21.15.79:0/513662030 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.268+0000 7f79d1b66640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61d4d0 0x5639cc62ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d2367640 1 -- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61d4d0 msgr2=0x5639cc62ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d2367640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61d4d0 0x5639cc62ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d2367640 1 -- 172.21.15.79:0/513662030 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 msgr2=0x5639cc62a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d2367640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc62a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d2367640 1 -- 172.21.15.79:0/513662030 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639cc62ae30 con 0x5639cc626a00 2024-04-22T06:45:35.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d1b66640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61d4d0 0x5639cc62ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d1365640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc62a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d2367640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 0x5639cc62a870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79c4044040 tx=0x7f79c4034650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a8ec8975e069b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79b57fa640 1 -- 172.21.15.79:0/513662030 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4046040 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79b57fa640 1 -- 172.21.15.79:0/513662030 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f79c4032840 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.269+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79c809e070 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79b57fa640 1 -- 172.21.15.79:0/513662030 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f79c4032450 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 10 client.25301 fetch_fsmap learned FSMap version 29 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5639cc69d460 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79b57fa640 1 -- 172.21.15.79:0/513662030 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c809e070 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79b57fa640 1 -- 172.21.15.79:0/513662030 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f79c4042910 con 0x5639cc626a00 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 10 client.25301 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 10 client.25301 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-22T06:45:35.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 20 client.25301 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc69dbe0 0x5639cc6a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 4 client.25301 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 --> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5639cc58fc10 con 0x5639cc69dbe0 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.270+0000 7f79d1b66640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc69dbe0 0x5639cc6a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79b57fa640 10 client.25301 ms_handle_connect on v2:172.21.15.193:6834/3775212644 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d1b66640 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc69dbe0 0x5639cc6a0000 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=655b9d9fae25ded1 server_cookie=c684938fb9d87219 in_seq=0 out_seq=0 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79b57fa640 1 -- 172.21.15.79:0/513662030 <== mds.2 v2:172.21.15.193:6834/3775212644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5639cc58fc10 con 0x5639cc69dbe0 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79b57fa640 10 client.25301 handle_command_reply: tid=0 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 client.25301 shutdown 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 msgr2=0x5639cc62a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 0x5639cc62a870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79c4044040 tx=0x7f79c4034650 comp rx=0 tx=0).stop 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 shutdown_connections 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5639cc626a00 0x5639cc62a870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x5639cc69dbe0 0x5639cc6a0000 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5639cc61de80 0x5639cc62a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79b57fa640 0 client.25301 ms_handle_reset on v2:172.21.15.193:6834/3775212644 2024-04-22T06:45:35.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 --2- 172.21.15.79:0/513662030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639cc61d4d0 0x5639cc62ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 >> 172.21.15.79:0/513662030 conn(0x5639cc617ba0 msgr2=0x5639cc61e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 shutdown_connections 2024-04-22T06:45:35.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 1 -- 172.21.15.79:0/513662030 wait complete. 2024-04-22T06:45:35.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.271+0000 7f79d69b6740 20 client.25301 trim_cache size 0 max 16384 2024-04-22T06:45:35.277 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f79b80a3ed0 msgr2=0x7f79b80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f79b80a3ed0 0x7f79b80a6390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f79c4037040 tx=0x7f79c4032340 comp rx=0 tx=0).stop 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 msgr2=0x7f79cc1a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 0x7f79cc1a05a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f79c800d950 tx=0x7f79c800de10 comp rx=0 tx=0).stop 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 shutdown_connections 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f79b80a3ed0 0x7f79b80a6390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79cc10c510 0x7f79cc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79cc103af0 0x7f79cc1a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 --2- 172.21.15.79:0/1516029755 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f79cc1031d0 0x7f79cc1a05a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 >> 172.21.15.79:0/1516029755 conn(0x7f79cc077f50 msgr2=0x7f79cc1083e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 shutdown_connections 2024-04-22T06:45:35.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.272+0000 7f79d3dca640 1 -- 172.21.15.79:0/1516029755 wait complete. 2024-04-22T06:45:35.284 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.000999972 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15330 172.21.15.79:0/1421414208 after 0 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15330 172.21.15.79:0/1421414208 after 0.000999972 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15333 172.21.15.79:0/2601682053 after 0.000999972 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15339 172.21.15.79:0/3376003914 after 0 2024-04-22T06:45:35.284 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999972 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: reconnect by client.15339 172.21.15.79:0/3376003914 after 0.000999972 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:reconnect 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:reconnect 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=f=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:rejoin 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:rejoin 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=f=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-22T06:45:35.285 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:35 smithi079 ceph-mon[27946]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 -- 172.21.15.79:0/944271588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c06b170 msgr2=0x7fea3c106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 --2- 172.21.15.79:0/944271588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c06b170 0x7fea3c106ba0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fea30009900 tx=0x7fea3002bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 -- 172.21.15.79:0/944271588 shutdown_connections 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 --2- 172.21.15.79:0/944271588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea3c107280 0x7fea3c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 --2- 172.21.15.79:0/944271588 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c06b170 0x7fea3c106ba0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 --2- 172.21.15.79:0/944271588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea3c06a7c0 0x7fea3c06aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 -- 172.21.15.79:0/944271588 >> 172.21.15.79:0/944271588 conn(0x7fea3c076c50 msgr2=0x7fea3c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 -- 172.21.15.79:0/944271588 shutdown_connections 2024-04-22T06:45:35.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.392+0000 7fea418b3640 1 -- 172.21.15.79:0/944271588 wait complete. 2024-04-22T06:45:35.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 Processor -- start 2024-04-22T06:45:35.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 -- start start 2024-04-22T06:45:35.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea3c06a7c0 0x7fea3c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 0x7fea3c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea3c19cf20 0x7fea3c19d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c112180 con 0x7fea3c107280 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c112000 con 0x7fea3c19cf20 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea418b3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c112300 con 0x7fea3c06a7c0 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.393+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 0x7fea3c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 0x7fea3c198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49950/0 (socket says 172.21.15.79:49950) 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 -- 172.21.15.79:0/2006399242 learned_addr learned my addr 172.21.15.79:0/2006399242 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3affd640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea3c06a7c0 0x7fea3c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea3c19cf20 0x7fea3c19d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 -- 172.21.15.79:0/2006399242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea3c06a7c0 msgr2=0x7fea3c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea3c06a7c0 0x7fea3c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 -- 172.21.15.79:0/2006399242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea3c19cf20 msgr2=0x7fea3c19d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea3c19cf20 0x7fea3c19d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 -- 172.21.15.79:0/2006399242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3c19da10 con 0x7fea3c107280 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 0x7fea3c198ba0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fea30036040 tx=0x7fea3002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea408b1640 1 -- 172.21.15.79:0/2006399242 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30043020 con 0x7fea3c107280 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea408b1640 1 -- 172.21.15.79:0/2006399242 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea3002cbe0 con 0x7fea3c107280 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3c19dca0 con 0x7fea3c107280 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.394+0000 7fea408b1640 1 -- 172.21.15.79:0/2006399242 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30003c30 con 0x7fea3c107280 2024-04-22T06:45:35.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.395+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea3c1014c0 con 0x7fea3c107280 2024-04-22T06:45:35.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.397+0000 7fea408b1640 1 -- 172.21.15.79:0/2006399242 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fea30046020 con 0x7fea3c107280 2024-04-22T06:45:35.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.397+0000 7fea408b1640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fea240a3f50 0x7fea240a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea408b1640 1 -- 172.21.15.79:0/2006399242 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fea300f2910 con 0x7fea3c107280 2024-04-22T06:45:35.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea3affd640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fea240a3f50 0x7fea240a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 Processor -- start 2024-04-22T06:45:35.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 -- start start 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a3094d0 0x55969a3098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a309e80 0x55969a312430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 0x55969a314e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55969a26e1f0 con 0x55969a312a00 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55969a20b4b0 con 0x55969a3094d0 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea4457f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55969a26a1b0 con 0x55969a309e80 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea3b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 0x55969a314e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea3b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 0x55969a314e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49958/0 (socket says 172.21.15.79:49958) 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea3b7fe640 1 -- 172.21.15.79:0/223550614 learned_addr learned my addr 172.21.15.79:0/223550614 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea3affd640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fea240a3f50 0x7fea240a6410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00bbe0 tx=0x7fea2c00b390 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.398+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a309e80 0x55969a312430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3affd640 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a3094d0 0x55969a3098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3b7fe640 1 -- 172.21.15.79:0/223550614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a309e80 msgr2=0x55969a312430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a309e80 0x55969a312430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3b7fe640 1 -- 172.21.15.79:0/223550614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a3094d0 msgr2=0x55969a3098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a3094d0 0x55969a3098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3b7fe640 1 -- 172.21.15.79:0/223550614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55969a1fd1b0 con 0x55969a312a00 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3affd640 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a3094d0 0x55969a3098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 0x55969a314e50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fea34007e50 tx=0x7fea3400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856e9529284d6d43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea21ffb640 1 -- 172.21.15.79:0/223550614 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea34014070 con 0x55969a312a00 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea21ffb640 1 -- 172.21.15.79:0/223550614 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea3400ee90 con 0x55969a312a00 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.399+0000 7fea21ffb640 1 -- 172.21.15.79:0/223550614 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea34005380 con 0x55969a312a00 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 -- 172.21.15.79:0/223550614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 msgr2=0x55969a314e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 0x55969a314e50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fea34007e50 tx=0x7fea3400e720 comp rx=0 tx=0).stop 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 -- 172.21.15.79:0/223550614 shutdown_connections 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a312a00 0x55969a314e50 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a309e80 0x55969a312430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 --2- 172.21.15.79:0/223550614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a3094d0 0x55969a3098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 -- 172.21.15.79:0/223550614 >> 172.21.15.79:0/223550614 conn(0x55969a303ba0 msgr2=0x55969a303f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 -- 172.21.15.79:0/223550614 shutdown_connections 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 -- 172.21.15.79:0/223550614 wait complete. 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 -- start start 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.400+0000 7fea4457f740 1 Processor -- start 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a312340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a30f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a30fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55969a315970 con 0x55969a309e80 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a30f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a30f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49974/0 (socket says 172.21.15.79:49974) 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3682723640 learned_addr learned my addr 172.21.15.79:0/3682723640 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3affd640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a312340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55969a227500 con 0x55969a312a00 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55969a315af0 con 0x55969a3094d0 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea4457f740 10 client.0 _ll_register_callbacks cb 0x55969a1f5b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a30fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3682723640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 msgr2=0x55969a312340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a312340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 msgr2=0x55969a30fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a30fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3682723640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559699f803f0 con 0x55969a309e80 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a30f6d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fea300be400 tx=0x7fea30009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6973176960417568 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.401+0000 7fea1affd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea1affd640 1 -- 172.21.15.79:0/3682723640 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea300be700 con 0x55969a309e80 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea1affd640 1 -- 172.21.15.79:0/3682723640 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fea300c6ee0 con 0x55969a309e80 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea3400ee90 con 0x55969a309e80 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea1affd640 1 -- 172.21.15.79:0/3682723640 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea300c2ab0 con 0x55969a309e80 2024-04-22T06:45:35.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 10 client.15987 fetch_fsmap learned FSMap version 29 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55969a04a7d0 con 0x55969a309e80 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea1affd640 1 -- 172.21.15.79:0/3682723640 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea3400ee90 con 0x55969a309e80 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea1affd640 1 -- 172.21.15.79:0/3682723640 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fea300dd070 con 0x55969a309e80 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 10 client.15987 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 10 client.15987 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.e 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 20 client.15987 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a322e10 0x55969a325250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 4 client.15987 mds_command: new command op to 24413 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.402+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 --> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55969a20d930 con 0x55969a322e10 2024-04-22T06:45:35.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.403+0000 7fea3affd640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a322e10 0x55969a325250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.403+0000 7fea1affd640 10 client.15987 ms_handle_connect on v2:172.21.15.190:6834/2543246858 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.403+0000 7fea3affd640 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a322e10 0x55969a325250 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4736927482b942d2 server_cookie=d31ea0bc331e1363 in_seq=0 out_seq=0 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea1affd640 1 -- 172.21.15.79:0/3682723640 <== mds.1 v2:172.21.15.190:6834/2543246858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55969a20d930 con 0x55969a322e10 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea1affd640 10 client.15987 handle_command_reply: tid=0 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 client.15987 shutdown 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 msgr2=0x55969a30f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a30f6d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fea300be400 tx=0x7fea30009260 comp rx=0 tx=0).stop 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 shutdown_connections 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a322e10 0x55969a325250 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a30fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a30f6d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 --2- 172.21.15.79:0/3682723640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a312340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 >> 172.21.15.79:0/3682723640 conn(0x55969a303ba0 msgr2=0x55969a3058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea1affd640 0 client.15987 ms_handle_reset on v2:172.21.15.190:6834/2543246858 2024-04-22T06:45:35.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 shutdown_connections 2024-04-22T06:45:35.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 1 -- 172.21.15.79:0/3682723640 wait complete. 2024-04-22T06:45:35.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.404+0000 7fea4457f740 20 client.15987 trim_cache size 0 max 16384 2024-04-22T06:45:35.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.429+0000 7fea4457f740 1 Processor -- start 2024-04-22T06:45:35.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.429+0000 7fea4457f740 1 -- start start 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.429+0000 7fea4457f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a31e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.429+0000 7fea4457f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a311e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.429+0000 7fea4457f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a312340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.429+0000 7fea4457f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c1afee0 con 0x55969a309e80 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea4457f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c1b05b0 con 0x55969a312a00 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea4457f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c1add30 con 0x55969a3094d0 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a311e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a311e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49976/0 (socket says 172.21.15.79:49976) 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3200183768 learned_addr learned my addr 172.21.15.79:0/3200183768 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a312340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3200183768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 msgr2=0x55969a31e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a31e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3200183768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 msgr2=0x55969a312340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a312340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 -- 172.21.15.79:0/3200183768 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55969a2ff470 con 0x55969a309e80 2024-04-22T06:45:35.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a311e00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fea30005ec0 tx=0x7fea30003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f360d04ce4b0cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea217fa640 1 -- 172.21.15.79:0/3200183768 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea300be060 con 0x55969a309e80 2024-04-22T06:45:35.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea217fa640 1 -- 172.21.15.79:0/3200183768 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea300bdb50 con 0x55969a309e80 2024-04-22T06:45:35.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.430+0000 7fea217fa640 1 -- 172.21.15.79:0/3200183768 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea300f0020 con 0x55969a309e80 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 -- 172.21.15.79:0/3200183768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 msgr2=0x55969a311e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a311e00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fea30005ec0 tx=0x7fea30003760 comp rx=0 tx=0).stop 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 -- 172.21.15.79:0/3200183768 shutdown_connections 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a312a00 0x55969a312340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a309e80 0x55969a311e00 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 --2- 172.21.15.79:0/3200183768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a3094d0 0x55969a31e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 -- 172.21.15.79:0/3200183768 >> 172.21.15.79:0/3200183768 conn(0x55969a303ba0 msgr2=0x55969a302c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 -- 172.21.15.79:0/3200183768 shutdown_connections 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 -- 172.21.15.79:0/3200183768 wait complete. 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.431+0000 7fea4457f740 1 -- start start 2024-04-22T06:45:35.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 Processor -- start 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 0x55969a31b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a309e80 0x55969a3164d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a312a00 0x55969a316a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c1ae3a0 con 0x55969a3094d0 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 0x55969a31b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3a7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a309e80 0x55969a3164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 0x55969a31b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49992/0 (socket says 172.21.15.79:49992) 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 -- 172.21.15.79:0/124796483 learned_addr learned my addr 172.21.15.79:0/124796483 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c1af810 con 0x55969a309e80 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea3c1aea70 con 0x55969a312a00 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea4457f740 10 client.0 _ll_register_callbacks cb 0x7fea300be5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3b7fe640 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a312a00 0x55969a316a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 -- 172.21.15.79:0/124796483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a312a00 msgr2=0x55969a316a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a312a00 0x55969a316a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 -- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a309e80 msgr2=0x55969a3164d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a309e80 0x55969a3164d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.432+0000 7fea3affd640 1 -- 172.21.15.79:0/124796483 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55969a3172b0 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea3affd640 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 0x55969a31b160 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fea2c010860 tx=0x7fea2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fba15e244fe935a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea1affd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea1affd640 1 -- 172.21.15.79:0/124796483 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c02a070 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea1affd640 1 -- 172.21.15.79:0/124796483 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fea2c02a210 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55969a27bc10 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea1affd640 1 -- 172.21.15.79:0/124796483 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea2c039050 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea4457f740 10 client.15993 fetch_fsmap learned FSMap version 29 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55969a389480 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea1affd640 1 -- 172.21.15.79:0/124796483 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea2c02a8c0 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.433+0000 7fea1affd640 1 -- 172.21.15.79:0/124796483 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fea2c029a80 con 0x55969a3094d0 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 10 client.15993 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T06:45:35.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 10 client.15993 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.e 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 20 client.15993 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a389bf0 0x55969a38c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 4 client.15993 mds_command: new command op to 24413 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 --> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55969a27bc10 con 0x55969a389bf0 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a389bf0 0x55969a38c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea1affd640 10 client.15993 ms_handle_connect on v2:172.21.15.190:6834/2543246858 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea3a7fc640 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a389bf0 0x55969a38c010 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61d958510106a707 server_cookie=53e2decc57986942 in_seq=0 out_seq=0 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea1affd640 1 -- 172.21.15.79:0/124796483 <== mds.1 v2:172.21.15.190:6834/2543246858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55969a27bc10 con 0x55969a389bf0 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea1affd640 10 client.15993 handle_command_reply: tid=0 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.434+0000 7fea4457f740 1 client.15993 shutdown 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 msgr2=0x55969a31b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 0x55969a31b160 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fea2c010860 tx=0x7fea2c0046c0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 shutdown_connections 2024-04-22T06:45:35.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55969a389bf0 0x55969a38c010 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55969a309e80 0x55969a3164d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55969a312a00 0x55969a316a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 --2- 172.21.15.79:0/124796483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55969a3094d0 0x55969a31b160 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 >> 172.21.15.79:0/124796483 conn(0x55969a303ba0 msgr2=0x55969a30a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea1affd640 0 client.15993 ms_handle_reset on v2:172.21.15.190:6834/2543246858 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 shutdown_connections 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 1 -- 172.21.15.79:0/124796483 wait complete. 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.435+0000 7fea4457f740 20 client.15993 trim_cache size 0 max 16384 2024-04-22T06:45:35.438 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:45:35.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fea240a3f50 msgr2=0x7fea240a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fea240a3f50 0x7fea240a6410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00bbe0 tx=0x7fea2c00b390 comp rx=0 tx=0).stop 2024-04-22T06:45:35.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 msgr2=0x7fea3c198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 0x7fea3c198ba0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fea30036040 tx=0x7fea3002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 shutdown_connections 2024-04-22T06:45:35.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fea240a3f50 0x7fea240a6410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea3c19cf20 0x7fea3c19d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c107280 0x7fea3c198ba0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 --2- 172.21.15.79:0/2006399242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea3c06a7c0 0x7fea3c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 >> 172.21.15.79:0/2006399242 conn(0x7fea3c076c50 msgr2=0x7fea3c0ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 shutdown_connections 2024-04-22T06:45:35.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.436+0000 7fea418b3640 1 -- 172.21.15.79:0/2006399242 wait complete. 2024-04-22T06:45:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.000999972 2024-04-22T06:45:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15330 172.21.15.79:0/1421414208 after 0 2024-04-22T06:45:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15330 172.21.15.79:0/1421414208 after 0.000999972 2024-04-22T06:45:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:45:35.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15333 172.21.15.79:0/2601682053 after 0.000999972 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15339 172.21.15.79:0/3376003914 after 0 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999972 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: reconnect by client.15339 172.21.15.79:0/3376003914 after 0.000999972 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:reconnect 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:reconnect 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=f=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:rejoin 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:rejoin 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=f=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-22T06:45:35.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:35 smithi193 ceph-mon[31815]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:45:35.445 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:45:35.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.000999972 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15330 172.21.15.79:0/1421414208 after 0 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15330 172.21.15.79:0/1421414208 after 0.000999972 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15333 172.21.15.79:0/2601682053 after 0.000999972 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15339 172.21.15.79:0/3376003914 after 0 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999972 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: reconnect by client.15339 172.21.15.79:0/3376003914 after 0.000999972 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:reconnect 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:reconnect 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=f=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:rejoin 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:rejoin 2024-04-22T06:45:35.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=f=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:35.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-22T06:45:35.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:35 smithi190 ceph-mon[31421]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.551+0000 7f7d20359640 1 -- 172.21.15.79:0/2087045080 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 msgr2=0x7f7d1810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.551+0000 7f7d20359640 1 --2- 172.21.15.79:0/2087045080 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d1810dd30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009ca0 tx=0x7f7d1002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 -- 172.21.15.79:0/2087045080 shutdown_connections 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 --2- 172.21.15.79:0/2087045080 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d1810dd30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 --2- 172.21.15.79:0/2087045080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d18101a40 0x7f7d18105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 --2- 172.21.15.79:0/2087045080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d18101090 0x7f7d18101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 -- 172.21.15.79:0/2087045080 >> 172.21.15.79:0/2087045080 conn(0x7f7d18077f50 msgr2=0x7f7d18078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 -- 172.21.15.79:0/2087045080 shutdown_connections 2024-04-22T06:45:35.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 -- 172.21.15.79:0/2087045080 wait complete. 2024-04-22T06:45:35.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 Processor -- start 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 -- start start 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d18101090 0x7f7d18197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d18197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d1819bef0 0x7f7d1819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.552+0000 7f7d20359640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181110b0 con 0x7f7d18106160 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d20359640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18110f30 con 0x7f7d18101090 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d20359640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18111230 con 0x7f7d1819bef0 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d18197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d18197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50022/0 (socket says 172.21.15.79:50022) 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 -- 172.21.15.79:0/1760338827 learned_addr learned my addr 172.21.15.79:0/1760338827 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d18101090 0x7f7d18197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d1819bef0 0x7f7d1819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 -- 172.21.15.79:0/1760338827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d1819bef0 msgr2=0x7f7d1819c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d1819bef0 0x7f7d1819c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 -- 172.21.15.79:0/1760338827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d18101090 msgr2=0x7f7d18197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d18101090 0x7f7d18197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 -- 172.21.15.79:0/1760338827 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1819c9e0 con 0x7f7d18106160 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d1819bef0 0x7f7d1819c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d18197b70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7d08005fd0 tx=0x7f7d0800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d0f7fe640 1 -- 172.21.15.79:0/1760338827 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08017070 con 0x7f7d18106160 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d0f7fe640 1 -- 172.21.15.79:0/1760338827 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d08009c30 con 0x7f7d18106160 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d0f7fe640 1 -- 172.21.15.79:0/1760338827 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08003d30 con 0x7f7d18106160 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1819ccd0 con 0x7f7d18106160 2024-04-22T06:45:35.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.553+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d181a4470 con 0x7f7d18106160 2024-04-22T06:45:35.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.556+0000 7f7d0f7fe640 1 -- 172.21.15.79:0/1760338827 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f7d08003ed0 con 0x7f7d18106160 2024-04-22T06:45:35.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.557+0000 7f7d2301f740 1 Processor -- start 2024-04-22T06:45:35.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.557+0000 7f7d0f7fe640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7d040a3ed0 0x7f7d040a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.557+0000 7f7d2301f740 1 -- start start 2024-04-22T06:45:35.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.557+0000 7f7d0f7fe640 1 -- 172.21.15.79:0/1760338827 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f7d080d0810 con 0x7f7d18106160 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.557+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 0x55bbc13698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.557+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7d040a3ed0 0x7f7d040a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc1372430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc1372a00 0x55bbc1374e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d2301f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bbc12ce1f0 con 0x55bbc1372a00 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d2301f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bbc126b4b0 con 0x55bbc13694d0 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d2301f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bbc12ca1b0 con 0x55bbc1369e80 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e8f6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc1372a00 0x55bbc1374e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 0x55bbc13698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e8f6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc1372a00 0x55bbc1374e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50026/0 (socket says 172.21.15.79:50026) 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e8f6640 1 -- 172.21.15.79:0/759095028 learned_addr learned my addr 172.21.15.79:0/759095028 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 0x55bbc13698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:46092/0 (socket says 172.21.15.79:46092) 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc1372430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/759095028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 msgr2=0x55bbc1372430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc1372430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7d040a3ed0 0x7f7d040a6390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009a60 tx=0x7f7d1002cf90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/759095028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc1372a00 msgr2=0x55bbc1374e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc1372a00 0x55bbc1374e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/759095028 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbc125d1b0 con 0x55bbc13694d0 2024-04-22T06:45:35.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc1372430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:35.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.558+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 0x55bbc13698b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400bbe0 tx=0x7f7d1400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=789d5bc06371cb1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d0cbfe640 1 -- 172.21.15.79:0/759095028 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14015070 con 0x55bbc13694d0 2024-04-22T06:45:35.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d0cbfe640 1 -- 172.21.15.79:0/759095028 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d140035f0 con 0x55bbc13694d0 2024-04-22T06:45:35.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d0cbfe640 1 -- 172.21.15.79:0/759095028 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d140116a0 con 0x55bbc13694d0 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 -- 172.21.15.79:0/759095028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 msgr2=0x55bbc13698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 0x55bbc13698b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400bbe0 tx=0x7f7d1400f730 comp rx=0 tx=0).stop 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 -- 172.21.15.79:0/759095028 shutdown_connections 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc1372a00 0x55bbc1374e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc1372430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 --2- 172.21.15.79:0/759095028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc13694d0 0x55bbc13698b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.559+0000 7f7d2301f740 1 -- 172.21.15.79:0/759095028 >> 172.21.15.79:0/759095028 conn(0x55bbc1363ba0 msgr2=0x55bbc1363f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 -- 172.21.15.79:0/759095028 shutdown_connections 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 -- 172.21.15.79:0/759095028 wait complete. 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 -- start start 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 Processor -- start 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1372350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc136f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc136fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bbc1377b80 con 0x55bbc13694d0 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bbc1287500 con 0x55bbc1372a00 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bbc1377d00 con 0x55bbc1369e80 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1372350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d2301f740 10 client.0 _ll_register_callbacks cb 0x55bbc1255b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1372350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50040/0 (socket says 172.21.15.79:50040) 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.560+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/4025475150 learned_addr learned my addr 172.21.15.79:0/4025475150 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc136fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc136f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/4025475150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 msgr2=0x55bbc136f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc136f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 msgr2=0x55bbc136fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc136fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/4025475150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbc0fe03f0 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc136f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1372350 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7d140041c0 tx=0x7f7d140041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500e9ff12013e761 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d01ffb640 1 -- 172.21.15.79:0/4025475150 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14028060 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d01ffb640 1 -- 172.21.15.79:0/4025475150 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f7d1400bcd0 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d140035f0 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d01ffb640 1 -- 172.21.15.79:0/4025475150 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d14015070 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d2301f740 10 client.16005 fetch_fsmap learned FSMap version 29 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.561+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bbc10aa7d0 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d01ffb640 1 -- 172.21.15.79:0/4025475150 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d140035f0 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d01ffb640 1 -- 172.21.15.79:0/4025475150 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f7d140118d0 con 0x55bbc13694d0 2024-04-22T06:45:35.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d2301f740 10 client.16005 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d2301f740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d2301f740 20 client.16005 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d2301f740 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc1384cf0 0x55bbc13870e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d2301f740 4 client.16005 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbc126d930 con 0x55bbc1384cf0 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.562+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc1384cf0 0x55bbc13870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc1384cf0 0x55bbc13870e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f14eef4fa54aa523 server_cookie=1da5629bb344b528 in_seq=0 out_seq=0 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d01ffb640 10 client.16005 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d01ffb640 1 -- 172.21.15.79:0/4025475150 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbc126d930 con 0x55bbc1384cf0 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d01ffb640 10 client.16005 handle_command_reply: tid=0 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d2301f740 1 client.16005 shutdown 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 msgr2=0x55bbc1372350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.563+0000 7f7d2301f740 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1372350 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7d140041c0 tx=0x7f7d140041f0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 shutdown_connections 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc1384cf0 0x55bbc13870e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc136fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc136f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d01ffb640 0 client.16005 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 --2- 172.21.15.79:0/4025475150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1372350 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 >> 172.21.15.79:0/4025475150 conn(0x55bbc1363ba0 msgr2=0x55bbc13658f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 shutdown_connections 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 1 -- 172.21.15.79:0/4025475150 wait complete. 2024-04-22T06:45:35.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.564+0000 7f7d2301f740 20 client.16005 trim_cache size 0 max 16384 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 Processor -- start 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 -- start start 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1380a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc13657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc1371e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181abd90 con 0x55bbc13694d0 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181ac460 con 0x55bbc1372a00 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d2301f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181a9be0 con 0x55bbc1369e80 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1380a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1380a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50054/0 (socket says 172.21.15.79:50054) 2024-04-22T06:45:35.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.588+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/2311546877 learned_addr learned my addr 172.21.15.79:0/2311546877 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc1371e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/2311546877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 msgr2=0x55bbc13657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc13657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/2311546877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 msgr2=0x55bbc1371e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc1371e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/2311546877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbc1377cb0 con 0x55bbc13694d0 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1380a40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7d14000c00 tx=0x7f7d140059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4db6053e9b84f85 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d03fff640 1 -- 172.21.15.79:0/2311546877 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14028070 con 0x55bbc13694d0 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d03fff640 1 -- 172.21.15.79:0/2311546877 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d14004260 con 0x55bbc13694d0 2024-04-22T06:45:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.589+0000 7f7d03fff640 1 -- 172.21.15.79:0/2311546877 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14005550 con 0x55bbc13694d0 2024-04-22T06:45:35.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 -- 172.21.15.79:0/2311546877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 msgr2=0x55bbc1380a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1380a40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7d14000c00 tx=0x7f7d140059b0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 -- 172.21.15.79:0/2311546877 shutdown_connections 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1372a00 0x55bbc1371e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1369e80 0x55bbc13657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 --2- 172.21.15.79:0/2311546877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc1380a40 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 -- 172.21.15.79:0/2311546877 >> 172.21.15.79:0/2311546877 conn(0x55bbc1363ba0 msgr2=0x55bbc1362b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 -- 172.21.15.79:0/2311546877 shutdown_connections 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 -- 172.21.15.79:0/2311546877 wait complete. 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.590+0000 7f7d2301f740 1 -- start start 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 Processor -- start 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc137d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1369e80 0x55bbc136c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1372a00 0x55bbc136c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181aa250 con 0x55bbc13694d0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181ab6c0 con 0x55bbc1369e80 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181aa920 con 0x55bbc1372a00 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d2301f740 10 client.0 _ll_register_callbacks cb 0x7f7d1400fee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc137d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc137d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50056/0 (socket says 172.21.15.79:50056) 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/1068785176 learned_addr learned my addr 172.21.15.79:0/1068785176 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1369e80 0x55bbc136c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e8f6640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1372a00 0x55bbc136c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/1068785176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1372a00 msgr2=0x55bbc136c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1372a00 0x55bbc136c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1369e80 msgr2=0x55bbc136c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1369e80 0x55bbc136c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.591+0000 7f7d1e0f5640 1 -- 172.21.15.79:0/1068785176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbc13e9f20 con 0x55bbc13694d0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d1e0f5640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc137d4a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7d140053d0 tx=0x7f7d1400b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba14e0ce7928aef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:35.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d01ffb640 1 -- 172.21.15.79:0/1068785176 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14039060 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d01ffb640 1 -- 172.21.15.79:0/1068785176 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f7d14011640 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d14004260 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d01ffb640 1 -- 172.21.15.79:0/1068785176 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d14028040 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 10 client.16011 fetch_fsmap learned FSMap version 29 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bbc13eb040 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d01ffb640 1 -- 172.21.15.79:0/1068785176 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d14004260 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d01ffb640 1 -- 172.21.15.79:0/1068785176 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f7d14032020 con 0x55bbc13694d0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 10 client.16011 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 20 client.16011 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc13eb6a0 0x55bbc13edac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 4 client.16011 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.592+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbc12dbc10 con 0x55bbc13eb6a0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.593+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc13eb6a0 0x55bbc13edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:35.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.593+0000 7f7d01ffb640 10 client.16011 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.593+0000 7f7d1d8f4640 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc13eb6a0 0x55bbc13edac0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a4d8ce4c9865304 server_cookie=83b7d115232281b9 in_seq=0 out_seq=0 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.593+0000 7f7d01ffb640 1 -- 172.21.15.79:0/1068785176 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbc12dbc10 con 0x55bbc13eb6a0 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.593+0000 7f7d01ffb640 10 client.16011 handle_command_reply: tid=0 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.593+0000 7f7d2301f740 1 client.16011 shutdown 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 msgr2=0x55bbc137d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc137d4a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7d140053d0 tx=0x7f7d1400b420 comp rx=0 tx=0).stop 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 shutdown_connections 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55bbc13eb6a0 0x55bbc13edac0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d01ffb640 0 client.16011 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bbc1369e80 0x55bbc136c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bbc1372a00 0x55bbc136c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 --2- 172.21.15.79:0/1068785176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bbc13694d0 0x55bbc137d4a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 >> 172.21.15.79:0/1068785176 conn(0x55bbc1363ba0 msgr2=0x55bbc13657b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 shutdown_connections 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 1 -- 172.21.15.79:0/1068785176 wait complete. 2024-04-22T06:45:35.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d2301f740 20 client.16011 trim_cache size 0 max 16384 2024-04-22T06:45:35.598 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7d040a3ed0 msgr2=0x7f7d040a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7d040a3ed0 0x7f7d040a6390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009a60 tx=0x7f7d1002cf90 comp rx=0 tx=0).stop 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 msgr2=0x7f7d18197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d18197b70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7d08005fd0 tx=0x7f7d0800dde0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 shutdown_connections 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d1819bef0 0x7f7d1819c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d18106160 0x7f7d18197b70 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.594+0000 7f7d20359640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7d040a3ed0 0x7f7d040a6390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.595+0000 7f7d20359640 1 --2- 172.21.15.79:0/1760338827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d18101090 0x7f7d18197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.595+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 >> 172.21.15.79:0/1760338827 conn(0x7f7d18077f50 msgr2=0x7f7d180fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.595+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 shutdown_connections 2024-04-22T06:45:35.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:35.595+0000 7f7d20359640 1 -- 172.21.15.79:0/1760338827 wait complete. 2024-04-22T06:45:36.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3305280430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:active 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:active 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:36.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:36 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:45:36.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3305280430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:active 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:active 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:36.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:36 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3305280430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] up:active 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] up:active 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:36 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:45:36.605 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:45:36.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:45:36.606 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:45:36.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 -- 172.21.15.79:0/2260071856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c103af0 msgr2=0x7fb76c10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 --2- 172.21.15.79:0/2260071856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c103af0 0x7fb76c10bfd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb760009900 tx=0x7fb76002bd60 comp rx=0 tx=0).stop 2024-04-22T06:45:36.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 -- 172.21.15.79:0/2260071856 shutdown_connections 2024-04-22T06:45:36.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 --2- 172.21.15.79:0/2260071856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c10c510 0x7fb76c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 --2- 172.21.15.79:0/2260071856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c103af0 0x7fb76c10bfd0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 --2- 172.21.15.79:0/2260071856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb76c1031d0 0x7fb76c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 -- 172.21.15.79:0/2260071856 >> 172.21.15.79:0/2260071856 conn(0x7fb76c077f50 msgr2=0x7fb76c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 -- 172.21.15.79:0/2260071856 shutdown_connections 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 -- 172.21.15.79:0/2260071856 wait complete. 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 Processor -- start 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.721+0000 7fb77292e640 1 -- start start 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb77292e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1031d0 0x7fb76c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb77292e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb76c103af0 0x7fb76c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb77292e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 0x7fb76c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb77292e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1121f0 con 0x7fb76c10c510 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb77292e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c112070 con 0x7fb76c103af0 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb77292e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c112370 con 0x7fb76c1031d0 2024-04-22T06:45:36.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 0x7fb76c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1031d0 0x7fb76c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 0x7fb76c19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50080/0 (socket says 172.21.15.79:50080) 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 -- 172.21.15.79:0/3441829831 learned_addr learned my addr 172.21.15.79:0/3441829831 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb76c103af0 0x7fb76c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 -- 172.21.15.79:0/3441829831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1031d0 msgr2=0x7fb76c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1031d0 0x7fb76c198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 -- 172.21.15.79:0/3441829831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb76c103af0 msgr2=0x7fb76c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb76c103af0 0x7fb76c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 -- 172.21.15.79:0/3441829831 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c19d700 con 0x7fb76c10c510 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb770ecb640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 0x7fb76c19cfd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb76400eaa0 tx=0x7fb76400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.722+0000 7fb7697fa640 1 -- 172.21.15.79:0/3441829831 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76400cdb0 con 0x7fb76c10c510 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.723+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76c19d9f0 con 0x7fb76c10c510 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.723+0000 7fb7697fa640 1 -- 172.21.15.79:0/3441829831 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb764005020 con 0x7fb76c10c510 2024-04-22T06:45:36.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.723+0000 7fb7697fa640 1 -- 172.21.15.79:0/3441829831 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7640116d0 con 0x7fb76c10c510 2024-04-22T06:45:36.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.723+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76c1a5280 con 0x7fb76c10c510 2024-04-22T06:45:36.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7697fa640 1 -- 172.21.15.79:0/3441829831 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fb7640051d0 con 0x7fb76c10c510 2024-04-22T06:45:36.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7755f4740 1 Processor -- start 2024-04-22T06:45:36.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7755f4740 1 -- start start 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7697fa640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7540a3ed0 0x7fb7540a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 0x560971482cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x56097148b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.726+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56097148be10 0x56097148e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb7755f4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5609713e6f00 con 0x5609714828e0 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb7755f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560971384590 con 0x560971483290 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb7755f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5609713e2ec0 con 0x56097148be10 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7540a3ed0 0x7fb7540a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 0x560971482cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb7697fa640 1 -- 172.21.15.79:0/3441829831 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fb7640d08e0 con 0x7fb76c10c510 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 0x560971482cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50084/0 (socket says 172.21.15.79:50084) 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 -- 172.21.15.79:0/2047324773 learned_addr learned my addr 172.21.15.79:0/2047324773 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb770ecb640 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56097148be10 0x56097148e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x56097148b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 -- 172.21.15.79:0/2047324773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56097148be10 msgr2=0x56097148e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56097148be10 0x56097148e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 -- 172.21.15.79:0/2047324773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 msgr2=0x56097148b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x56097148b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 -- 172.21.15.79:0/2047324773 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609711c17e0 con 0x5609714828e0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76bfff640 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 0x560971482cc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb75c00bc50 tx=0x7fb75c012b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9e43ba366b2631 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7540a3ed0 0x7fb7540a6390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb760005ab0 tx=0x7fb760004310 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.727+0000 7fb7527fc640 1 -- 172.21.15.79:0/2047324773 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c020070 con 0x5609714828e0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7527fc640 1 -- 172.21.15.79:0/2047324773 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb75c00ddd0 con 0x5609714828e0 2024-04-22T06:45:36.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7527fc640 1 -- 172.21.15.79:0/2047324773 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c01bc10 con 0x5609714828e0 2024-04-22T06:45:36.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 -- 172.21.15.79:0/2047324773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 msgr2=0x560971482cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 0x560971482cc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb75c00bc50 tx=0x7fb75c012b30 comp rx=0 tx=0).stop 2024-04-22T06:45:36.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 -- 172.21.15.79:0/2047324773 shutdown_connections 2024-04-22T06:45:36.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x56097148b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56097148be10 0x56097148e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2047324773 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714828e0 0x560971482cc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 -- 172.21.15.79:0/2047324773 >> 172.21.15.79:0/2047324773 conn(0x56097147cfb0 msgr2=0x56097147d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 -- 172.21.15.79:0/2047324773 shutdown_connections 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.728+0000 7fb7755f4740 1 -- 172.21.15.79:0/2047324773 wait complete. 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 -- start start 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 Processor -- start 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560971483290 0x560971488e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x560971489360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 0x5609714981f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56097148ed80 con 0x5609714898a0 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5609713a0580 con 0x56097148be10 2024-04-22T06:45:36.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56097148ef00 con 0x560971483290 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb7755f4740 10 client.0 _ll_register_callbacks cb 0x5609713a6650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb770ecb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 0x5609714981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb770ecb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 0x5609714981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50090/0 (socket says 172.21.15.79:50090) 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb770ecb640 1 -- 172.21.15.79:0/3116645192 learned_addr learned my addr 172.21.15.79:0/3116645192 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x560971489360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.729+0000 7fb76bfff640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560971483290 0x560971488e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb770ecb640 1 -- 172.21.15.79:0/3116645192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560971483290 msgr2=0x560971488e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb770ecb640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560971483290 0x560971488e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb770ecb640 1 -- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 msgr2=0x560971489360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb770ecb640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x560971489360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb770ecb640 1 -- 172.21.15.79:0/3116645192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560971221f70 con 0x5609714898a0 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb76bfff640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560971483290 0x560971488e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb770ecb640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 0x5609714981f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb7640cdeb0 tx=0x7fb764087050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8df59963b6ab3d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb74b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb74b7fe640 1 -- 172.21.15.79:0/3116645192 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76401f040 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb74b7fe640 1 -- 172.21.15.79:0/3116645192 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fb764098ce0 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb75c00ddd0 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb74b7fe640 1 -- 172.21.15.79:0/3116645192 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb7640a5230 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb7755f4740 10 client.16023 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5609710b9170 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.730+0000 7fb74b7fe640 1 -- 172.21.15.79:0/3116645192 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb764080530 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb74b7fe640 1 -- 172.21.15.79:0/3116645192 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fb764022060 con 0x5609714898a0 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb7755f4740 10 client.16023 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb7755f4740 10 client.16023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb7755f4740 20 client.16023 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb7755f4740 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56097149c7b0 0x56097149ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb7755f4740 4 client.16023 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:36.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb75c00ddd0 con 0x56097149c7b0 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb76bfff640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56097149c7b0 0x56097149ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb76bfff640 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56097149c7b0 0x56097149ebe0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2353f0fe2a964af5 server_cookie=335713416606cd65 in_seq=0 out_seq=0 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.731+0000 7fb74b7fe640 10 client.16023 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb74b7fe640 1 -- 172.21.15.79:0/3116645192 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fb75c00ddd0 con 0x56097149c7b0 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb74b7fe640 10 client.16023 handle_command_reply: tid=0 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb7755f4740 1 client.16023 shutdown 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 msgr2=0x5609714981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb7755f4740 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 0x5609714981f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb7640cdeb0 tx=0x7fb764087050 comp rx=0 tx=0).stop 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 shutdown_connections 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.732+0000 7fb7755f4740 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56097149c7b0 0x56097149ebe0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x560971489360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5609714898a0 0x5609714981f0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 1 --2- 172.21.15.79:0/3116645192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560971483290 0x560971488e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb74b7fe640 0 client.16023 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 >> 172.21.15.79:0/3116645192 conn(0x56097147cfb0 msgr2=0x56097147ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 shutdown_connections 2024-04-22T06:45:36.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 1 -- 172.21.15.79:0/3116645192 wait complete. 2024-04-22T06:45:36.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.733+0000 7fb7755f4740 20 client.16023 trim_cache size 0 max 16384 2024-04-22T06:45:36.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.763+0000 7fb7755f4740 1 Processor -- start 2024-04-22T06:45:36.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 -- start start 2024-04-22T06:45:36.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 0x56097147be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x5609714a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1ae2f0 con 0x560971483290 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1abbc0 con 0x56097148be10 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb7755f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1a9a10 con 0x5609714898a0 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 0x56097147be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 0x56097147be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50094/0 (socket says 172.21.15.79:50094) 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 -- 172.21.15.79:0/4107509103 learned_addr learned my addr 172.21.15.79:0/4107509103 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb770ecb640 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x5609714a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 -- 172.21.15.79:0/4107509103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 msgr2=0x5609714a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 -- 172.21.15.79:0/4107509103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 msgr2=0x5609714a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x5609714a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 -- 172.21.15.79:0/4107509103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097147cae0 con 0x560971483290 2024-04-22T06:45:36.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.764+0000 7fb76bfff640 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 0x56097147be90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb75c01e040 tx=0x7fb75c01b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a683ae3c043bc956 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.765+0000 7fb751ffb640 1 -- 172.21.15.79:0/4107509103 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c020030 con 0x560971483290 2024-04-22T06:45:36.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.765+0000 7fb751ffb640 1 -- 172.21.15.79:0/4107509103 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb75c02e070 con 0x560971483290 2024-04-22T06:45:36.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.765+0000 7fb751ffb640 1 -- 172.21.15.79:0/4107509103 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c0054a0 con 0x560971483290 2024-04-22T06:45:36.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 -- 172.21.15.79:0/4107509103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 msgr2=0x56097147be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 0x56097147be90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb75c01e040 tx=0x7fb75c01b5a0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 -- 172.21.15.79:0/4107509103 shutdown_connections 2024-04-22T06:45:36.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56097148be10 0x5609714a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 --2- 172.21.15.79:0/4107509103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560971483290 0x56097147be90 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 -- 172.21.15.79:0/4107509103 >> 172.21.15.79:0/4107509103 conn(0x56097147cfb0 msgr2=0x56097147b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 -- 172.21.15.79:0/4107509103 shutdown_connections 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 -- 172.21.15.79:0/4107509103 wait complete. 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 -- start start 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.766+0000 7fb7755f4740 1 Processor -- start 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x560971490280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714907c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 0x560971501d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1aa080 con 0x56097148be10 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1ab4f0 con 0x560971483290 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1aa750 con 0x5609714898a0 2024-04-22T06:45:36.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb7755f4740 10 client.0 _ll_register_callbacks cb 0x7fb76409c480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 0x560971501d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 0x560971501d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50110/0 (socket says 172.21.15.79:50110) 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 -- 172.21.15.79:0/2413949983 learned_addr learned my addr 172.21.15.79:0/2413949983 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb76b7fe640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714907c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb76bfff640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x560971490280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 -- 172.21.15.79:0/2413949983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 msgr2=0x5609714907c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714907c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 -- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 msgr2=0x560971490280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x560971490280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.767+0000 7fb770ecb640 1 -- 172.21.15.79:0/2413949983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560971502280 con 0x56097148be10 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb770ecb640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 0x560971501d40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb764098ad0 tx=0x7fb764085de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e553beec274a2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb74b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb74b7fe640 1 -- 172.21.15.79:0/2413949983 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76401f040 con 0x56097148be10 2024-04-22T06:45:36.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb74b7fe640 1 -- 172.21.15.79:0/2413949983 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fb7640983c0 con 0x56097148be10 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb75c02e070 con 0x56097148be10 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb74b7fe640 1 -- 172.21.15.79:0/2413949983 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb76409bfd0 con 0x56097148be10 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 10 client.16029 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560971503590 con 0x56097148be10 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb74b7fe640 1 -- 172.21.15.79:0/2413949983 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb75c02e070 con 0x56097148be10 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb74b7fe640 1 -- 172.21.15.79:0/2413949983 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fb76400f040 con 0x56097148be10 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 10 client.16029 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 20 client.16029 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560971503ac0 0x560971505eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 4 client.16029 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.768+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5609713f4920 con 0x560971503ac0 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.769+0000 7fb76bfff640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560971503ac0 0x560971505eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.769+0000 7fb76bfff640 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560971503ac0 0x560971505eb0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3385ae1a4ce6eb5 server_cookie=61147b9f0b2fc4c9 in_seq=0 out_seq=0 2024-04-22T06:45:36.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.769+0000 7fb74b7fe640 10 client.16029 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.770+0000 7fb74b7fe640 1 -- 172.21.15.79:0/2413949983 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5609713f4920 con 0x560971503ac0 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.770+0000 7fb74b7fe640 10 client.16029 handle_command_reply: tid=0 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.770+0000 7fb7755f4740 1 client.16029 shutdown 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.770+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 msgr2=0x560971501d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.770+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 0x560971501d40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb764098ad0 tx=0x7fb764085de0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 shutdown_connections 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560971503ac0 0x560971505eb0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56097148be10 0x560971501d40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5609714898a0 0x5609714907c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 --2- 172.21.15.79:0/2413949983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560971483290 0x560971490280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 >> 172.21.15.79:0/2413949983 conn(0x56097147cfb0 msgr2=0x56097147b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb74b7fe640 0 client.16029 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 shutdown_connections 2024-04-22T06:45:36.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 1 -- 172.21.15.79:0/2413949983 wait complete. 2024-04-22T06:45:36.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.771+0000 7fb7755f4740 20 client.16029 trim_cache size 0 max 16384 2024-04-22T06:45:36.775 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:36.775 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:45:36.775 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "051b9dd4-b366-4a88-be39-1c6338699b4c", 2024-04-22T06:45:36.775 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:45:36.775 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:36.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7540a3ed0 msgr2=0x7fb7540a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7540a3ed0 0x7fb7540a6390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb760005ab0 tx=0x7fb760004310 comp rx=0 tx=0).stop 2024-04-22T06:45:36.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 msgr2=0x7fb76c19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 0x7fb76c19cfd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb76400eaa0 tx=0x7fb76400ef60 comp rx=0 tx=0).stop 2024-04-22T06:45:36.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 shutdown_connections 2024-04-22T06:45:36.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7540a3ed0 0x7fb7540a6390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb76c103af0 0x7fb76c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb76c10c510 0x7fb76c19cfd0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 --2- 172.21.15.79:0/3441829831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1031d0 0x7fb76c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 >> 172.21.15.79:0/3441829831 conn(0x7fb76c077f50 msgr2=0x7fb76c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 shutdown_connections 2024-04-22T06:45:36.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.772+0000 7fb77292e640 1 -- 172.21.15.79:0/3441829831 wait complete. 2024-04-22T06:45:36.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:36.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 -- 172.21.15.79:0/1830294950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01021f0 msgr2=0x7f1be01025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 --2- 172.21.15.79:0/1830294950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01021f0 0x7f1be01025d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009900 tx=0x7f1bd002bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 -- 172.21.15.79:0/1830294950 shutdown_connections 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 --2- 172.21.15.79:0/1830294950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01072c0 0x7f1be010ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 --2- 172.21.15.79:0/1830294950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1be0102ba0 0x7f1be0106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 --2- 172.21.15.79:0/1830294950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01021f0 0x7f1be01025d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 -- 172.21.15.79:0/1830294950 >> 172.21.15.79:0/1830294950 conn(0x7f1be00fda70 msgr2=0x7f1be00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 -- 172.21.15.79:0/1830294950 shutdown_connections 2024-04-22T06:45:36.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.896+0000 7f1be4806640 1 -- 172.21.15.79:0/1830294950 wait complete. 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.897+0000 7f1be4806640 1 Processor -- start 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.897+0000 7f1be4806640 1 -- start start 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1be4806640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1be01021f0 0x7f1be019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1be4806640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 0x7f1be019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1be4806640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01985f0 0x7f1be0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1be4806640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1be0112250 con 0x7f1be01072c0 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1be4806640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01120d0 con 0x7f1be01985f0 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1be4806640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01123d0 con 0x7f1be01021f0 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 0x7f1be019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 0x7f1be019e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50130/0 (socket says 172.21.15.79:50130) 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 -- 172.21.15.79:0/204083293 learned_addr learned my addr 172.21.15.79:0/204083293 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bde59c640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1be01021f0 0x7f1be019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bded9d640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01985f0 0x7f1be0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 -- 172.21.15.79:0/204083293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1be01021f0 msgr2=0x7f1be019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1be01021f0 0x7f1be019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 -- 172.21.15.79:0/204083293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01985f0 msgr2=0x7f1be0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01985f0 0x7f1be0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 -- 172.21.15.79:0/204083293 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be0199110 con 0x7f1be01072c0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bded9d640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01985f0 0x7f1be0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 0x7f1be019e930 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c970 tx=0x7f1bd400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.898+0000 7f1bcf7fe640 1 -- 172.21.15.79:0/204083293 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4007bb0 con 0x7f1be01072c0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.899+0000 7f1bcf7fe640 1 -- 172.21.15.79:0/204083293 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd4007d50 con 0x7f1be01072c0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.899+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be0199400 con 0x7f1be01072c0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.899+0000 7f1bcf7fe640 1 -- 172.21.15.79:0/204083293 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd400f450 con 0x7f1be01072c0 2024-04-22T06:45:36.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.899+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be00744d0 con 0x7f1be01072c0 2024-04-22T06:45:36.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1bcf7fe640 1 -- 172.21.15.79:0/204083293 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f1bd4004ea0 con 0x7f1be01072c0 2024-04-22T06:45:36.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 Processor -- start 2024-04-22T06:45:36.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 -- start start 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a302e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a3036d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3039680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b1a2f8a1e0 con 0x55b1a302dd90 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b1a2f27390 con 0x55b1a3037290 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1be73fb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b1a2f861a0 con 0x55b1a302e6b0 2024-04-22T06:45:36.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1bded9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a302e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.902+0000 7f1bded9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a302e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50140/0 (socket says 172.21.15.79:50140) 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 -- 172.21.15.79:0/4182157247 learned_addr learned my addr 172.21.15.79:0/4182157247 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bde59c640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a3036d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bcf7fe640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1bc80a3ed0 0x7f1bc80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3039680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 -- 172.21.15.79:0/4182157247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 msgr2=0x55b1a3036d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a3036d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 -- 172.21.15.79:0/4182157247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 msgr2=0x55b1a3039680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3039680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 -- 172.21.15.79:0/4182157247 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a2d667d0 con 0x55b1a302dd90 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3039680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bde59c640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a3036d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bcf7fe640 1 -- 172.21.15.79:0/204083293 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f1bd40d08a0 con 0x7f1be01072c0 2024-04-22T06:45:36.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1bc80a3ed0 0x7f1bc80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bded9d640 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a302e170 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009f30 tx=0x7f1bd8010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f598e88b3d3668e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.903+0000 7f1bccff9640 1 -- 172.21.15.79:0/4182157247 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8022070 con 0x55b1a302dd90 2024-04-22T06:45:36.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.904+0000 7f1bded9d640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1bc80a3ed0 0x7f1bc80a6390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1bd801f000 tx=0x7f1bd8022000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.904+0000 7f1bccff9640 1 -- 172.21.15.79:0/4182157247 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd8011890 con 0x55b1a302dd90 2024-04-22T06:45:36.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.904+0000 7f1bccff9640 1 -- 172.21.15.79:0/4182157247 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8016070 con 0x55b1a302dd90 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 -- 172.21.15.79:0/4182157247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 msgr2=0x55b1a302e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a302e170 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009f30 tx=0x7f1bd8010800 comp rx=0 tx=0).stop 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 -- 172.21.15.79:0/4182157247 shutdown_connections 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3039680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a3036d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 --2- 172.21.15.79:0/4182157247 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a302e170 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 -- 172.21.15.79:0/4182157247 >> 172.21.15.79:0/4182157247 conn(0x55b1a30284c0 msgr2=0x55b1a30288a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 -- 172.21.15.79:0/4182157247 shutdown_connections 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 -- 172.21.15.79:0/4182157247 wait complete. 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 -- start start 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.905+0000 7f1be73fb740 1 Processor -- start 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302dd90 0x55b1a3034350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3034890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3034dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b1a303a080 con 0x55b1a302e6b0 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b1a2f432f0 con 0x55b1a3037290 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b1a303a200 con 0x55b1a302dd90 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1be73fb740 10 client.0 _ll_register_callbacks cb 0x55b1a2f50580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3034890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50144/0 (socket says 172.21.15.79:50144) 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 -- 172.21.15.79:0/3205362032 learned_addr learned my addr 172.21.15.79:0/3205362032 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bded9d640 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3034dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 -- 172.21.15.79:0/3205362032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302dd90 msgr2=0x55b1a3034350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302dd90 0x55b1a3034350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 -- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 msgr2=0x55b1a3034dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3034dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 -- 172.21.15.79:0/3205362032 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1a2c9c3f0 con 0x55b1a302e6b0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3034890 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40cde20 tx=0x7f1bd409c1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c795486d155f012 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.906+0000 7f1bc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/3205362032 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd401d040 con 0x55b1a302e6b0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/3205362032 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f1bd40984d0 con 0x55b1a302e6b0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bd8011890 con 0x55b1a302e6b0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/3205362032 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1bd409cc80 con 0x55b1a302e6b0 2024-04-22T06:45:36.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 10 client.16041 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1a2c5c0f0 con 0x55b1a302e6b0 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/3205362032 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd8011890 con 0x55b1a302e6b0 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/3205362032 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f1bd40af060 con 0x55b1a302e6b0 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 10 client.16041 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 10 client.16041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 20 client.16041 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a3047260 0x55b1a30496b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 4 client.16041 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.907+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1a2f97c00 con 0x55b1a3047260 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.908+0000 7f1bde59c640 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a3047260 0x55b1a30496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.908+0000 7f1bc5ffb640 10 client.16041 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.908+0000 7f1bde59c640 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a3047260 0x55b1a30496b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65922368a45e6623 server_cookie=73a78ff82d2016c in_seq=0 out_seq=0 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/3205362032 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1a2f97c00 con 0x55b1a3047260 2024-04-22T06:45:36.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1bc5ffb640 10 client.16041 handle_command_reply: tid=0 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 client.16041 shutdown 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 msgr2=0x55b1a3034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3034890 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40cde20 tx=0x7f1bd409c1b0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 shutdown_connections 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a3047260 0x55b1a30496b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a3034dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1bc5ffb640 0 client.16041 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3034890 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 --2- 172.21.15.79:0/3205362032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302dd90 0x55b1a3034350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 >> 172.21.15.79:0/3205362032 conn(0x55b1a30284c0 msgr2=0x55b1a302a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 shutdown_connections 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 1 -- 172.21.15.79:0/3205362032 wait complete. 2024-04-22T06:45:36.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.909+0000 7f1be73fb740 20 client.16041 trim_cache size 0 max 16384 2024-04-22T06:45:36.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 Processor -- start 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 -- start start 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a302dd90 0x55b1a3036a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3042f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a3037290 0x55b1a3033290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01ae3d0 con 0x55b1a302e6b0 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01abca0 con 0x55b1a302dd90 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1be73fb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01a9af0 con 0x55b1a3037290 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1bddd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3042f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1bddd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3042f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50154/0 (socket says 172.21.15.79:50154) 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1bde59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a302dd90 0x55b1a3036a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.942+0000 7f1bddd9b640 1 -- 172.21.15.79:0/1447984743 learned_addr learned my addr 172.21.15.79:0/1447984743 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bded9d640 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a3037290 0x55b1a3033290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bddd9b640 1 -- 172.21.15.79:0/1447984743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a3037290 msgr2=0x55b1a3033290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a3037290 0x55b1a3033290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bddd9b640 1 -- 172.21.15.79:0/1447984743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a302dd90 msgr2=0x55b1a3036a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a302dd90 0x55b1a3036a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bddd9b640 1 -- 172.21.15.79:0/1447984743 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a3027ff0 con 0x55b1a302e6b0 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bde59c640 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a302dd90 0x55b1a3036a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3042f40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40989e0 tx=0x7f1bd409aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1578c200802ead52 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bc7fff640 1 -- 172.21.15.79:0/1447984743 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40ba030 con 0x55b1a302e6b0 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bc7fff640 1 -- 172.21.15.79:0/1447984743 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd40a0930 con 0x55b1a302e6b0 2024-04-22T06:45:36.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.943+0000 7f1bc7fff640 1 -- 172.21.15.79:0/1447984743 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd401d040 con 0x55b1a302e6b0 2024-04-22T06:45:36.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 -- 172.21.15.79:0/1447984743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 msgr2=0x55b1a3042f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3042f40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40989e0 tx=0x7f1bd409aa90 comp rx=0 tx=0).stop 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 -- 172.21.15.79:0/1447984743 shutdown_connections 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a3037290 0x55b1a3033290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302e6b0 0x55b1a3042f40 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 --2- 172.21.15.79:0/1447984743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a302dd90 0x55b1a3036a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 -- 172.21.15.79:0/1447984743 >> 172.21.15.79:0/1447984743 conn(0x55b1a30284c0 msgr2=0x55b1a301ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 -- 172.21.15.79:0/1447984743 shutdown_connections 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 -- 172.21.15.79:0/1447984743 wait complete. 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 -- start start 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.944+0000 7f1be73fb740 1 Processor -- start 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a303f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a303f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a303c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01aa160 con 0x55b1a302dd90 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a303f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a303f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50168/0 (socket says 172.21.15.79:50168) 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 -- 172.21.15.79:0/2743507782 learned_addr learned my addr 172.21.15.79:0/2743507782 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01ab5d0 con 0x55b1a3037290 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01aa830 con 0x55b1a302e6b0 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1be73fb740 10 client.0 _ll_register_callbacks cb 0x7f1bd40058a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a303f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 -- 172.21.15.79:0/2743507782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 msgr2=0x55b1a303f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a303f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 -- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 msgr2=0x55b1a303c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bded9d640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a303c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a303c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 -- 172.21.15.79:0/2743507782 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1a303cc20 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bded9d640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a303c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bde59c640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a303f450 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009900 tx=0x7f1bd002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ce7db706065562 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.945+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/2743507782 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd002cc20 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/2743507782 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f1bd0005040 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bd40a0930 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/2743507782 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1bd0059050 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 10 client.16047 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1a30ae2d0 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/2743507782 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd40a0930 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/2743507782 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f1bd0054030 con 0x55b1a302dd90 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 10 client.16047 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:36.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 20 client.16047 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a30aeaa0 0x55b1a30b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 4 client.16047 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.946+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b1a2f91a70 con 0x55b1a30aeaa0 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a30aeaa0 0x55b1a30b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1bc5ffb640 10 client.16047 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1bddd9b640 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a30aeaa0 0x55b1a30b0ec0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c845fb8e33f7bef7 server_cookie=25cb2604e12df4dd in_seq=0 out_seq=0 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1bc5ffb640 1 -- 172.21.15.79:0/2743507782 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b1a2f91a70 con 0x55b1a30aeaa0 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1bc5ffb640 10 client.16047 handle_command_reply: tid=0 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1be73fb740 1 client.16047 shutdown 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 msgr2=0x55b1a303f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.947+0000 7f1be73fb740 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a303f450 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009900 tx=0x7f1bd002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 shutdown_connections 2024-04-22T06:45:36.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1a30aeaa0 0x55b1a30b0ec0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b1a3037290 0x55b1a303c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1bc5ffb640 0 client.16047 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1a302e6b0 0x55b1a303f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 --2- 172.21.15.79:0/2743507782 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b1a302dd90 0x55b1a303f450 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 >> 172.21.15.79:0/2743507782 conn(0x55b1a30284c0 msgr2=0x55b1a30297a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 shutdown_connections 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 1 -- 172.21.15.79:0/2743507782 wait complete. 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be73fb740 20 client.16047 trim_cache size 0 max 16384 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:45:36.954 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: "051b9dd4-b366-4a88-be39-1c6338699b4c": { 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "051b9dd4-b366-4a88-be39-1c6338699b4c", 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:45:36.955 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1bc80a3ed0 msgr2=0x7f1bc80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1bc80a3ed0 0x7f1bc80a6390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1bd801f000 tx=0x7f1bd8022000 comp rx=0 tx=0).stop 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 msgr2=0x7f1be019e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 0x7f1be019e930 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400c970 tx=0x7f1bd400ce30 comp rx=0 tx=0).stop 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 shutdown_connections 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1bc80a3ed0 0x7f1bc80a6390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.948+0000 7f1be4806640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1be01985f0 0x7f1be0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.949+0000 7f1be4806640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1be01072c0 0x7f1be019e930 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.949+0000 7f1be4806640 1 --2- 172.21.15.79:0/204083293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1be01021f0 0x7f1be019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:36.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.949+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 >> 172.21.15.79:0/204083293 conn(0x7f1be00fda70 msgr2=0x7f1be00feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:36.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.949+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 shutdown_connections 2024-04-22T06:45:36.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:45:36.949+0000 7f1be4806640 1 -- 172.21.15.79:0/204083293 wait complete. 2024-04-22T06:45:36.962 INFO:tasks.cephfs.filesystem:scrub status for tag:051b9dd4-b366-4a88-be39-1c6338699b4c - {'path': '/', 'tag': '051b9dd4-b366-4a88-be39-1c6338699b4c', 'options': 'recursive,force'} 2024-04-22T06:45:38.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:38 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:45:38.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:38 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:38 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:45:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:38 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:38.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:38 smithi193 ceph-mon[31815]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-22T06:45:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:38 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:45:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:38 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:38 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:45:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:38 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:38 smithi190 ceph-mon[31421]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-22T06:45:38.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:38 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:45:38.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:38 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:38.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:38 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:45:38.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:38 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:38.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:38 smithi079 ceph-mon[27946]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-22T06:45:39.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.206+0000 7f654c950640 1 -- 172.21.15.79:0/3212583840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6548106170 msgr2=0x7f654810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.206+0000 7f654c950640 1 --2- 172.21.15.79:0/3212583840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6548106170 0x7f654810dd40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6540009ca0 tx=0x7f654002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.206+0000 7f654c950640 1 -- 172.21.15.79:0/3212583840 shutdown_connections 2024-04-22T06:45:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.206+0000 7f654c950640 1 --2- 172.21.15.79:0/3212583840 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6548101aa0 0x7f6548105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.207+0000 7f654c950640 1 --2- 172.21.15.79:0/3212583840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6548106170 0x7f654810dd40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.207+0000 7f654c950640 1 --2- 172.21.15.79:0/3212583840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65481010f0 0x7f65481014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.207+0000 7f654c950640 1 -- 172.21.15.79:0/3212583840 >> 172.21.15.79:0/3212583840 conn(0x7f6548077fd0 msgr2=0x7f65480783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.207+0000 7f654c950640 1 -- 172.21.15.79:0/3212583840 shutdown_connections 2024-04-22T06:45:39.206 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.207+0000 7f654c950640 1 -- 172.21.15.79:0/3212583840 wait complete. 2024-04-22T06:45:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 Processor -- start 2024-04-22T06:45:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 -- start start 2024-04-22T06:45:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65481010f0 0x7f6548198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6548106170 0x7f6548198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 0x7f654819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f6546d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 0x7f654819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f6546d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 0x7f654819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50200/0 (socket says 172.21.15.79:50200) 2024-04-22T06:45:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f6546d9d640 1 -- 172.21.15.79:0/2566618419 learned_addr learned my addr 172.21.15.79:0/2566618419 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f65481110c0 con 0x7f654819ce30 2024-04-22T06:45:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6548110f40 con 0x7f65481010f0 2024-04-22T06:45:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6548111240 con 0x7f6548106170 2024-04-22T06:45:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f654659c640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65481010f0 0x7f6548198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.208+0000 7f6545d9b640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6548106170 0x7f6548198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f6546d9d640 1 -- 172.21.15.79:0/2566618419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6548106170 msgr2=0x7f6548198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f6546d9d640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6548106170 0x7f6548198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f6546d9d640 1 -- 172.21.15.79:0/2566618419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65481010f0 msgr2=0x7f6548198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f6546d9d640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65481010f0 0x7f6548198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f6546d9d640 1 -- 172.21.15.79:0/2566618419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654819d920 con 0x7f654819ce30 2024-04-22T06:45:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f654659c640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65481010f0 0x7f6548198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f6546d9d640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 0x7f654819d2e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6540005e30 tx=0x7f6540034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f65377fe640 1 -- 172.21.15.79:0/2566618419 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6540039040 con 0x7f654819ce30 2024-04-22T06:45:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654819dbb0 con 0x7f654819ce30 2024-04-22T06:45:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f65377fe640 1 -- 172.21.15.79:0/2566618419 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6540032970 con 0x7f654819ce30 2024-04-22T06:45:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f65377fe640 1 -- 172.21.15.79:0/2566618419 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65400042a0 con 0x7f654819ce30 2024-04-22T06:45:39.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.209+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6548102d50 con 0x7f654819ce30 2024-04-22T06:45:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 Processor -- start 2024-04-22T06:45:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f65377fe640 1 -- 172.21.15.79:0/2566618419 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f6540034ca0 con 0x7f654819ce30 2024-04-22T06:45:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 -- start start 2024-04-22T06:45:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e6311f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 0x558e6311ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e631290e0 0x558e6312b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558e630848d0 con 0x558e631208d0 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558e63021e80 con 0x558e631290e0 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.213+0000 7f654f619740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558e63080890 con 0x558e6311fb60 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 0x558e6311ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 0x558e6311ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:49256/0 (socket says 172.21.15.79:49256) 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 -- 172.21.15.79:0/3626248779 learned_addr learned my addr 172.21.15.79:0/3626248779 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f6545d9b640 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e6311f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f6545d9b640 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e631290e0 0x558e6312b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f65377fe640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f65300a3f70 0x7f65300a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 -- 172.21.15.79:0/3626248779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e631290e0 msgr2=0x558e6312b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e631290e0 0x558e6312b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 -- 172.21.15.79:0/3626248779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 msgr2=0x558e6311f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e6311f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.214+0000 7f654659c640 1 -- 172.21.15.79:0/3626248779 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e63043320 con 0x558e6311fb60 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.215+0000 7f6546d9d640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f65300a3f70 0x7f65300a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.215+0000 7f654659c640 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 0x558e6311ffe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f653800f4c0 tx=0x7f653800f980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82798cb46a251968 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.215+0000 7f65377fe640 1 -- 172.21.15.79:0/2566618419 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f65400f6940 con 0x7f654819ce30 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.215+0000 7f6534ff9640 1 -- 172.21.15.79:0/3626248779 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538005220 con 0x558e6311fb60 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.215+0000 7f6534ff9640 1 -- 172.21.15.79:0/3626248779 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65380047a0 con 0x558e6311fb60 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 -- 172.21.15.79:0/3626248779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 msgr2=0x558e6311ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f6546d9d640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f65300a3f70 0x7f65300a6430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f65400f3f40 tx=0x7f65400c2580 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 0x558e6311ffe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f653800f4c0 tx=0x7f653800f980 comp rx=0 tx=0).stop 2024-04-22T06:45:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f6534ff9640 1 -- 172.21.15.79:0/3626248779 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538010b00 con 0x558e6311fb60 2024-04-22T06:45:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 -- 172.21.15.79:0/3626248779 shutdown_connections 2024-04-22T06:45:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e631290e0 0x558e6312b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e6311fb60 0x558e6311ffe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 --2- 172.21.15.79:0/3626248779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e6311f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 -- 172.21.15.79:0/3626248779 >> 172.21.15.79:0/3626248779 conn(0x558e630e6c90 msgr2=0x558e6311a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 -- 172.21.15.79:0/3626248779 shutdown_connections 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.216+0000 7f654f619740 1 -- 172.21.15.79:0/3626248779 wait complete. 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 -- start start 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 Processor -- start 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e631289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e63125d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558e6312be60 con 0x558e631208d0 2024-04-22T06:45:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558e6300ceb0 con 0x558e6311fb60 2024-04-22T06:45:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558e6312bfe0 con 0x558e631290e0 2024-04-22T06:45:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654f619740 10 client.0 _ll_register_callbacks cb 0x558e63039b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.217+0000 7f654659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e63125d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e63125d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50208/0 (socket says 172.21.15.79:50208) 2024-04-22T06:45:39.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 -- 172.21.15.79:0/1463641120 learned_addr learned my addr 172.21.15.79:0/1463641120 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f6545d9b640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e631289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f6545d9b640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 -- 172.21.15.79:0/1463641120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 msgr2=0x558e631262c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631262c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 -- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 msgr2=0x558e631289f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e631289f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 -- 172.21.15.79:0/1463641120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e62d97210 con 0x558e631208d0 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f654659c640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e63125d80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6538010010 tx=0x7f6538010be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba8c5d67c07aa2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f652dffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.218+0000 7f652dffb640 1 -- 172.21.15.79:0/1463641120 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538026070 con 0x558e631208d0 2024-04-22T06:45:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.219+0000 7f652dffb640 1 -- 172.21.15.79:0/1463641120 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f6538004fe0 con 0x558e631208d0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.219+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65380047a0 con 0x558e631208d0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.219+0000 7f652dffb640 1 -- 172.21.15.79:0/1463641120 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6538032050 con 0x558e631208d0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.219+0000 7f654f619740 10 client.16059 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.219+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558e62e63620 con 0x558e631208d0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.219+0000 7f652dffb640 1 -- 172.21.15.79:0/1463641120 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65380047a0 con 0x558e631208d0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f652dffb640 1 -- 172.21.15.79:0/1463641120 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f6538018890 con 0x558e631208d0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f654f619740 10 client.16059 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f654f619740 10 client.16059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f654f619740 20 client.16059 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f654f619740 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e63139550 0x558e6313b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f654f619740 4 client.16059 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e63098880 con 0x558e63139550 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.220+0000 7f6545d9b640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e63139550 0x558e6313b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f652dffb640 10 client.16059 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f6545d9b640 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e63139550 0x558e6313b9f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36b7177081ec1371 server_cookie=213c04a651baef5f in_seq=0 out_seq=0 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f652dffb640 1 -- 172.21.15.79:0/1463641120 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e63098880 con 0x558e63139550 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f652dffb640 10 client.16059 handle_command_reply: tid=0 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f654f619740 1 client.16059 shutdown 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 msgr2=0x558e63125d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.221+0000 7f654f619740 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e63125d80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6538010010 tx=0x7f6538010be0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 shutdown_connections 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e63139550 0x558e6313b9f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631262c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e63125d80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 --2- 172.21.15.79:0/1463641120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e631289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 >> 172.21.15.79:0/1463641120 conn(0x558e630e6c90 msgr2=0x558e6311c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f652dffb640 0 client.16059 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 shutdown_connections 2024-04-22T06:45:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 1 -- 172.21.15.79:0/1463641120 wait complete. 2024-04-22T06:45:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.222+0000 7f654f619740 20 client.16059 trim_cache size 0 max 16384 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 Processor -- start 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 -- start start 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63134e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e631284b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f65481b6980 con 0x558e631208d0 2024-04-22T06:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f65480722d0 con 0x558e6311fb60 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654f619740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6548070140 con 0x558e631290e0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.251+0000 7f654659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e631284b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e631284b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50214/0 (socket says 172.21.15.79:50214) 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 -- 172.21.15.79:0/3841168204 learned_addr learned my addr 172.21.15.79:0/3841168204 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f6545d9b640 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63134e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f6545d9b640 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 -- 172.21.15.79:0/3841168204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 msgr2=0x558e631289f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631289f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 -- 172.21.15.79:0/3841168204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 msgr2=0x558e63134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63134e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 -- 172.21.15.79:0/3841168204 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e6311ee30 con 0x558e631208d0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f6545d9b640 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f654659c640 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e631284b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6538004f30 tx=0x7f653800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afe0aba2ced9b49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f652ffff640 1 -- 172.21.15.79:0/3841168204 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653803b070 con 0x558e631208d0 2024-04-22T06:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f652ffff640 1 -- 172.21.15.79:0/3841168204 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f653802f070 con 0x558e631208d0 2024-04-22T06:45:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.252+0000 7f652ffff640 1 -- 172.21.15.79:0/3841168204 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538026040 con 0x558e631208d0 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 -- 172.21.15.79:0/3841168204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 msgr2=0x558e631284b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e631284b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6538004f30 tx=0x7f653800ff70 comp rx=0 tx=0).stop 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 -- 172.21.15.79:0/3841168204 shutdown_connections 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e631289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e631208d0 0x558e631284b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 --2- 172.21.15.79:0/3841168204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63134e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 -- 172.21.15.79:0/3841168204 >> 172.21.15.79:0/3841168204 conn(0x558e630e6c90 msgr2=0x558e6311a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 -- 172.21.15.79:0/3841168204 shutdown_connections 2024-04-22T06:45:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.253+0000 7f654f619740 1 -- 172.21.15.79:0/3841168204 wait complete. 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 -- start start 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 Processor -- start 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63131840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e63123260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 0x558e63121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6548070790 con 0x558e63123900 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6548071c00 con 0x558e6311fb60 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6548070e60 con 0x558e631290e0 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.254+0000 7f654f619740 10 client.0 _ll_register_callbacks cb 0x7f653c00b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 0x558e63121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 0x558e63121ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50222/0 (socket says 172.21.15.79:50222) 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 -- 172.21.15.79:0/2264859286 learned_addr learned my addr 172.21.15.79:0/2264859286 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63131840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f654659c640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e63123260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 -- 172.21.15.79:0/2264859286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 msgr2=0x558e63123260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e63123260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 -- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 msgr2=0x558e63131840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63131840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 -- 172.21.15.79:0/2264859286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e63122530 con 0x558e63123900 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f6545d9b640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 0x558e63121ff0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f653c005d80 tx=0x7f653c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3342344eb1e80211 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f652dffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f652dffb640 1 -- 172.21.15.79:0/2264859286 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c014030 con 0x558e63123900 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f652dffb640 1 -- 172.21.15.79:0/2264859286 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f653c0141d0 con 0x558e63123900 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.255+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f653802f070 con 0x558e63123900 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f652dffb640 1 -- 172.21.15.79:0/2264859286 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f653c0143d0 con 0x558e63123900 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 10 client.16065 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558e6319f2b0 con 0x558e63123900 2024-04-22T06:45:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f652dffb640 1 -- 172.21.15.79:0/2264859286 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f653802f070 con 0x558e63123900 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f652dffb640 1 -- 172.21.15.79:0/2264859286 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f653c0054a0 con 0x558e63123900 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 10 client.16065 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 20 client.16065 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e6319fa80 0x558e631a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 4 client.16065 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e630243d0 con 0x558e6319fa80 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.256+0000 7f654659c640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e6319fa80 0x558e631a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.257+0000 7f652dffb640 10 client.16065 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.257+0000 7f654659c640 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e6319fa80 0x558e631a1ea0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c70114cf140bc43 server_cookie=d2a8580509b76904 in_seq=0 out_seq=0 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f652dffb640 1 -- 172.21.15.79:0/2264859286 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e630243d0 con 0x558e6319fa80 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f652dffb640 10 client.16065 handle_command_reply: tid=0 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 client.16065 shutdown 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 msgr2=0x558e63121ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 0x558e63121ff0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f653c005d80 tx=0x7f653c015040 comp rx=0 tx=0).stop 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 shutdown_connections 2024-04-22T06:45:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558e6319fa80 0x558e631a1ea0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558e63123900 0x558e63121ff0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e631290e0 0x558e63123260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 --2- 172.21.15.79:0/2264859286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558e6311fb60 0x558e63131840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 >> 172.21.15.79:0/2264859286 conn(0x558e630e6c90 msgr2=0x558e63124610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f652dffb640 0 client.16065 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 shutdown_connections 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.261+0000 7f654f619740 1 -- 172.21.15.79:0/2264859286 wait complete. 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654f619740 20 client.16065 trim_cache size 0 max 16384 2024-04-22T06:45:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:45:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:45:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:45:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:45:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f65300a3f70 msgr2=0x7f65300a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f65300a3f70 0x7f65300a6430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f65400f3f40 tx=0x7f65400c2580 comp rx=0 tx=0).stop 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 msgr2=0x7f654819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 0x7f654819d2e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6540005e30 tx=0x7f6540034670 comp rx=0 tx=0).stop 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 shutdown_connections 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f654819ce30 0x7f654819d2e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6548106170 0x7f6548198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f65300a3f70 0x7f65300a6430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 --2- 172.21.15.79:0/2566618419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65481010f0 0x7f6548198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 >> 172.21.15.79:0/2566618419 conn(0x7f6548077fd0 msgr2=0x7f6548069840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 shutdown_connections 2024-04-22T06:45:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:39.262+0000 7f654c950640 1 -- 172.21.15.79:0/2566618419 wait complete. 2024-04-22T06:45:39.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:39 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:45:39.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:39 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:45:39.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:39 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:45:40.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:40 smithi193 ceph-mon[31815]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-22T06:45:40.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:40 smithi190 ceph-mon[31421]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-22T06:45:40.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:40 smithi079 ceph-mon[27946]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-22T06:45:42.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:42 smithi193 ceph-mon[31815]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 907 B/s wr, 24 op/s 2024-04-22T06:45:42.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:42 smithi190 ceph-mon[31421]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 907 B/s wr, 24 op/s 2024-04-22T06:45:42.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:42 smithi079 ceph-mon[27946]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 907 B/s wr, 24 op/s 2024-04-22T06:45:44.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:44 smithi193 ceph-mon[31815]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 744 B/s wr, 20 op/s 2024-04-22T06:45:44.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:44 smithi190 ceph-mon[31421]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 744 B/s wr, 20 op/s 2024-04-22T06:45:44.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:44 smithi079 ceph-mon[27946]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 744 B/s wr, 20 op/s 2024-04-22T06:45:46.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:46 smithi193 ceph-mon[31815]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 682 B/s wr, 18 op/s 2024-04-22T06:45:46.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:46 smithi190 ceph-mon[31421]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 682 B/s wr, 18 op/s 2024-04-22T06:45:46.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:46 smithi079 ceph-mon[27946]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 682 B/s wr, 18 op/s 2024-04-22T06:45:48.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:48 smithi193 ceph-mon[31815]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 682 B/s wr, 21 op/s 2024-04-22T06:45:48.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:48 smithi190 ceph-mon[31421]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 682 B/s wr, 21 op/s 2024-04-22T06:45:48.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:48 smithi079 ceph-mon[27946]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 682 B/s wr, 21 op/s 2024-04-22T06:45:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:50 smithi193 ceph-mon[31815]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 18 op/s 2024-04-22T06:45:50.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:50 smithi190 ceph-mon[31421]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 18 op/s 2024-04-22T06:45:50.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:50 smithi079 ceph-mon[27946]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 255 B/s wr, 18 op/s 2024-04-22T06:45:52.398 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.399+0000 7f90c663a640 1 -- 172.21.15.79:0/887242574 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0101aa0 msgr2=0x7f90c0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.399+0000 7f90c663a640 1 --2- 172.21.15.79:0/887242574 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0101aa0 0x7f90c0105a90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009900 tx=0x7f90b402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 -- 172.21.15.79:0/887242574 shutdown_connections 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 --2- 172.21.15.79:0/887242574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90c0106170 0x7f90c010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 --2- 172.21.15.79:0/887242574 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0101aa0 0x7f90c0105a90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 --2- 172.21.15.79:0/887242574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 0x7f90c01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 -- 172.21.15.79:0/887242574 >> 172.21.15.79:0/887242574 conn(0x7f90c0077fd0 msgr2=0x7f90c00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 -- 172.21.15.79:0/887242574 shutdown_connections 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 -- 172.21.15.79:0/887242574 wait complete. 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 Processor -- start 2024-04-22T06:45:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.400+0000 7f90c663a640 1 -- start start 2024-04-22T06:45:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c663a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 0x7f90c0199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c663a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 0x7f90c0199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 0x7f90c0199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 0x7f90c0199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53580/0 (socket says 172.21.15.79:53580) 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c663a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90c0199fa0 0x7f90c01a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90bf7fe640 1 -- 172.21.15.79:0/937091833 learned_addr learned my addr 172.21.15.79:0/937091833 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01110e0 con 0x7f90c0106170 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f90c0110f60 con 0x7f90c0199fa0 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90c0111260 con 0x7f90c01010f0 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90bffff640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 0x7f90c0199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.401+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90c0199fa0 0x7f90c01a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bf7fe640 1 -- 172.21.15.79:0/937091833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 msgr2=0x7f90c0199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 0x7f90c0199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bf7fe640 1 -- 172.21.15.79:0/937091833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90c0199fa0 msgr2=0x7f90c01a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90c0199fa0 0x7f90c01a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bf7fe640 1 -- 172.21.15.79:0/937091833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c01a4d40 con 0x7f90c0106170 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bffff640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 0x7f90c0199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 0x7f90c0199a60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f90b4000c00 tx=0x7f90b40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bd7fa640 1 -- 172.21.15.79:0/937091833 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4043020 con 0x7f90c0106170 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bd7fa640 1 -- 172.21.15.79:0/937091833 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b402cb70 con 0x7f90c0106170 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c0197310 con 0x7f90c0106170 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90bd7fa640 1 -- 172.21.15.79:0/937091833 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4003b50 con 0x7f90c0106170 2024-04-22T06:45:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.402+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c01977f0 con 0x7f90c0106170 2024-04-22T06:45:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.405+0000 7f90bd7fa640 1 -- 172.21.15.79:0/937091833 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f90b4046020 con 0x7f90c0106170 2024-04-22T06:45:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.405+0000 7f90bd7fa640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f90a80a40b0 0x7f90a80a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.406+0000 7f90bd7fa640 1 -- 172.21.15.79:0/937091833 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f90b40f2ac0 con 0x7f90c0106170 2024-04-22T06:45:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.406+0000 7f90bffff640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f90a80a40b0 0x7f90a80a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.406+0000 7f90bffff640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f90a80a40b0 0x7f90a80a6570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005d00 tx=0x7f90b000f820 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.406+0000 7f90c9221740 1 Processor -- start 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 -- start start 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b44620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b44fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d5e6aa98d0 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.406 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d5e6a46e80 con 0x55d5e6b458d0 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c9221740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d5e6aa5890 con 0x55d5e6b44b60 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53582/0 (socket says 172.21.15.79:53582) 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 -- 172.21.15.79:0/3323629409 learned_addr learned my addr 172.21.15.79:0/3323629409 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b44fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90bffff640 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b44620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 -- 172.21.15.79:0/3323629409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 msgr2=0x55d5e6b44fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b44fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 -- 172.21.15.79:0/3323629409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 msgr2=0x55d5e6b44620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b44620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.407+0000 7f90c4bd7640 1 -- 172.21.15.79:0/3323629409 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5e6a68320 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.407 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.408+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b504d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f90b8009f40 tx=0x7f90b8010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b31fb6cd01c8697 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.408+0000 7f90a67fc640 1 -- 172.21.15.79:0/3323629409 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8003b70 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.408+0000 7f90a67fc640 1 -- 172.21.15.79:0/3323629409 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b8003d10 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.408+0000 7f90a67fc640 1 -- 172.21.15.79:0/3323629409 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8011920 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.408+0000 7f90c9221740 1 -- 172.21.15.79:0/3323629409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 msgr2=0x55d5e6b504d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.408+0000 7f90c9221740 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b504d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f90b8009f40 tx=0x7f90b8010860 comp rx=0 tx=0).stop 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- 172.21.15.79:0/3323629409 shutdown_connections 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b504d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b44fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 --2- 172.21.15.79:0/3323629409 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b44620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- 172.21.15.79:0/3323629409 >> 172.21.15.79:0/3323629409 conn(0x55d5e6b0bc90 msgr2=0x55d5e6b3f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- 172.21.15.79:0/3323629409 shutdown_connections 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- 172.21.15.79:0/3323629409 wait complete. 2024-04-22T06:45:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- start start 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 Processor -- start 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b4d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b4adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b4b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d5e6b50e10 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d5e6a31eb0 con 0x55d5e6b458d0 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d5e6b50f90 con 0x55d5e6b44b60 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.409+0000 7f90c9221740 10 client.0 _ll_register_callbacks cb 0x55d5e6a5eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b4b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b4b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53592/0 (socket says 172.21.15.79:53592) 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 -- 172.21.15.79:0/2909083822 learned_addr learned my addr 172.21.15.79:0/2909083822 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b4adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90bffff640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b4d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 -- 172.21.15.79:0/2909083822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 msgr2=0x55d5e6b4d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b4d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 -- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 msgr2=0x55d5e6b4adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b4adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 -- 172.21.15.79:0/2909083822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5e67bc210 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b4b350 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f90b8004990 tx=0x7f90b8009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d952fc55492020a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f909f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f909f7fe640 1 -- 172.21.15.79:0/2909083822 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8004d50 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90b8003d10 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.410+0000 7f909f7fe640 1 -- 172.21.15.79:0/2909083822 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f90b8003ec0 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f909f7fe640 1 -- 172.21.15.79:0/2909083822 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f90b80040c0 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 10 client.16077 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d5e6888620 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f909f7fe640 1 -- 172.21.15.79:0/2909083822 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b8003d10 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f909f7fe640 1 -- 172.21.15.79:0/2909083822 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f90b8005040 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 10 client.16077 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 20 client.16077 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6b603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 4 client.16077 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5e6abd880 con 0x55d5e6b5e000 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.411+0000 7f90bffff640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6b603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.412+0000 7f90bffff640 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6b603f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6301a894f7c0a7df server_cookie=3a2be3a6462ec099 in_seq=0 out_seq=0 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.412+0000 7f909f7fe640 10 client.16077 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.412+0000 7f909f7fe640 1 -- 172.21.15.79:0/2909083822 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5e6abd880 con 0x55d5e6b5e000 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.412+0000 7f909f7fe640 10 client.16077 handle_command_reply: tid=0 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.412+0000 7f90c9221740 1 client.16077 shutdown 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 msgr2=0x55d5e6b4b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b4b350 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f90b8004990 tx=0x7f90b8009cc0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 shutdown_connections 2024-04-22T06:45:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6b603f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f909f7fe640 0 client.16077 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b4adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b4b350 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 --2- 172.21.15.79:0/2909083822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b4d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 >> 172.21.15.79:0/2909083822 conn(0x55d5e6b0bc90 msgr2=0x55d5e6b415a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 shutdown_connections 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 1 -- 172.21.15.79:0/2909083822 wait complete. 2024-04-22T06:45:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.413+0000 7f90c9221740 20 client.16077 trim_cache size 0 max 16384 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.427+0000 7f90c9221740 1 Processor -- start 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.427+0000 7f90c9221740 1 -- start start 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.427+0000 7f90c9221740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b59d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.427+0000 7f90c9221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b65a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c9221740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b65f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c9221740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01ae280 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c9221740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01abb50 con 0x55d5e6b458d0 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c9221740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01a99a0 con 0x55d5e6b44b60 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b65f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b65f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53606/0 (socket says 172.21.15.79:53606) 2024-04-22T06:45:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 -- 172.21.15.79:0/1834311339 learned_addr learned my addr 172.21.15.79:0/1834311339 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90bffff640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b59d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b65a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 -- 172.21.15.79:0/1834311339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 msgr2=0x55d5e6b59d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b59d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 -- 172.21.15.79:0/1834311339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 msgr2=0x55d5e6b65a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b65a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 -- 172.21.15.79:0/1834311339 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5e6b3af30 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90bffff640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b59d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b65a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b65f60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f90b8002230 tx=0x7f90b8010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b082f1718769a18e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90a5ffb640 1 -- 172.21.15.79:0/1834311339 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8003d10 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90a5ffb640 1 -- 172.21.15.79:0/1834311339 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b8003eb0 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.428+0000 7f90a5ffb640 1 -- 172.21.15.79:0/1834311339 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8004590 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 -- 172.21.15.79:0/1834311339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 msgr2=0x55d5e6b65f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b65f60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f90b8002230 tx=0x7f90b8010860 comp rx=0 tx=0).stop 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 -- 172.21.15.79:0/1834311339 shutdown_connections 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b458d0 0x55d5e6b65a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b65f60 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 --2- 172.21.15.79:0/1834311339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b44b60 0x55d5e6b59d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 -- 172.21.15.79:0/1834311339 >> 172.21.15.79:0/1834311339 conn(0x55d5e6b0bc90 msgr2=0x55d5e6b496c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 -- 172.21.15.79:0/1834311339 shutdown_connections 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.429+0000 7f90c9221740 1 -- 172.21.15.79:0/1834311339 wait complete. 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 -- start start 2024-04-22T06:45:52.430 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 Processor -- start 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b44b60 0x55d5e6b56720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 0x55d5e6b51a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b521d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01aa010 con 0x55d5e6b458d0 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 0x55d5e6b51a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01ab480 con 0x55d5e6b44b60 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01aa6e0 con 0x55d5e6b4e0e0 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 0x55d5e6b51a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53616/0 (socket says 172.21.15.79:53616) 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90bf7fe640 1 -- 172.21.15.79:0/2194428851 learned_addr learned my addr 172.21.15.79:0/2194428851 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:45:52.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90c9221740 10 client.0 _ll_register_callbacks cb 0x7f90b00093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.430+0000 7f90bffff640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b44b60 0x55d5e6b56720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90c4bd7640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bf7fe640 1 -- 172.21.15.79:0/2194428851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b4e0e0 msgr2=0x55d5e6b521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bf7fe640 1 -- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b44b60 msgr2=0x55d5e6b56720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b44b60 0x55d5e6b56720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bf7fe640 1 -- 172.21.15.79:0/2194428851 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5e6bc3290 con 0x55d5e6b458d0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bffff640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b44b60 0x55d5e6b56720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90bf7fe640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 0x55d5e6b51a90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f90b40be5b0 tx=0x7f90b40df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2020da871952e447 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f909f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f909f7fe640 1 -- 172.21.15.79:0/2194428851 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b40be8b0 con 0x55d5e6b458d0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f909f7fe640 1 -- 172.21.15.79:0/2194428851 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f90b40a90b0 con 0x55d5e6b458d0 2024-04-22T06:45:52.434 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90b8003eb0 con 0x55d5e6b458d0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f909f7fe640 1 -- 172.21.15.79:0/2194428851 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f90b40a92b0 con 0x55d5e6b458d0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90c9221740 10 client.16083 fetch_fsmap learned FSMap version 31 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d5e6bc43d0 con 0x55d5e6b458d0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.431+0000 7f909f7fe640 1 -- 172.21.15.79:0/2194428851 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b40dc060 con 0x55d5e6b458d0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f909f7fe640 1 -- 172.21.15.79:0/2194428851 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f90b40bfbe0 con 0x55d5e6b458d0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90c9221740 10 client.16083 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90c9221740 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90c9221740 20 client.16083 populate_metadata read hostname 'smithi079' 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90c9221740 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6bc6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90c9221740 4 client.16083 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f90b8003eb0 con 0x55d5e6b5e000 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90bffff640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6bc6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f90bffff640 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6bc6b70 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de176383aab2c475 server_cookie=4d967062690205c in_seq=0 out_seq=0 2024-04-22T06:45:52.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.432+0000 7f909f7fe640 10 client.16083 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.436+0000 7f909f7fe640 1 -- 172.21.15.79:0/2194428851 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f90b8003eb0 con 0x55d5e6b5e000 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.436+0000 7f909f7fe640 10 client.16083 handle_command_reply: tid=0 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.436+0000 7f90c9221740 1 client.16083 shutdown 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 msgr2=0x55d5e6b51a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 0x55d5e6b51a90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f90b40be5b0 tx=0x7f90b40df040 comp rx=0 tx=0).stop 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 shutdown_connections 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d5e6b5e000 0x55d5e6bc6b70 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d5e6b4e0e0 0x55d5e6b521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f909f7fe640 0 client.16083 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d5e6b458d0 0x55d5e6b51a90 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 --2- 172.21.15.79:0/2194428851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d5e6b44b60 0x55d5e6b56720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 >> 172.21.15.79:0/2194428851 conn(0x55d5e6b0bc90 msgr2=0x55d5e6b3e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:52.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 shutdown_connections 2024-04-22T06:45:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 1 -- 172.21.15.79:0/2194428851 wait complete. 2024-04-22T06:45:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c9221740 20 client.16083 trim_cache size 0 max 16384 2024-04-22T06:45:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:45:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:45:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:45:52.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f90a80a40b0 msgr2=0x7f90a80a6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f90a80a40b0 0x7f90a80a6570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005d00 tx=0x7f90b000f820 comp rx=0 tx=0).stop 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 msgr2=0x7f90c0199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 0x7f90c0199a60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f90b4000c00 tx=0x7f90b40045e0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 shutdown_connections 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f90a80a40b0 0x7f90a80a6570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90c0199fa0 0x7f90c01a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90c0106170 0x7f90c0199a60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.437+0000 7f90c663a640 1 --2- 172.21.15.79:0/937091833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c01010f0 0x7f90c0199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.438+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 >> 172.21.15.79:0/937091833 conn(0x7f90c0077fd0 msgr2=0x7f90c00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.438+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 shutdown_connections 2024-04-22T06:45:52.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:45:52.438+0000 7f90c663a640 1 -- 172.21.15.79:0/937091833 wait complete. 2024-04-22T06:45:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:52 smithi193 ceph-mon[31815]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 20 op/s 2024-04-22T06:45:52.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:52 smithi190 ceph-mon[31421]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 20 op/s 2024-04-22T06:45:52.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:52 smithi079 ceph-mon[27946]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 255 B/s wr, 20 op/s 2024-04-22T06:45:54.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:54 smithi193 ceph-mon[31815]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:54.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:54 smithi190 ceph-mon[31421]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:54.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:54 smithi079 ceph-mon[27946]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:56 smithi193 ceph-mon[31815]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:56.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:56 smithi190 ceph-mon[31421]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:56.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:56 smithi079 ceph-mon[27946]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:58 smithi193 ceph-mon[31815]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-22T06:45:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:45:58 smithi193 ceph-mon[31815]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-22T06:45:58.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:58 smithi190 ceph-mon[31421]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-22T06:45:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:45:58 smithi190 ceph-mon[31421]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-22T06:45:58.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:58 smithi079 ceph-mon[27946]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-22T06:45:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:45:58 smithi079 ceph-mon[27946]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-22T06:46:00.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:00 smithi193 ceph-mon[31815]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T06:46:00.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:00 smithi190 ceph-mon[31421]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T06:46:00.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:00 smithi079 ceph-mon[27946]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-22T06:46:02.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:02 smithi193 ceph-mon[31815]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:02.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:02 smithi190 ceph-mon[31421]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:02.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:02 smithi079 ceph-mon[27946]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr:Note: switching to '86f7587a5a09af35f0895e1a2d08527638fae697'. 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T06:46:03.298 INFO:tasks.workunit.client.0.smithi079.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: git switch -c 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr:Or undo this operation with: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: git switch - 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:03.299 INFO:tasks.workunit.client.0.smithi079.stderr:HEAD is now at 86f7587a5a0 qa/fsx: use the master branch to build the xfstest-dev 2024-04-22T06:46:03.304 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T06:46:03.372 INFO:tasks.workunit.client.0.smithi079.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T06:46:03.374 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T06:46:03.374 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T06:46:04.002 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T06:46:04.072 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T06:46:04.112 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T06:46:04.114 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T06:46:04.114 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T06:46:04.152 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T06:46:04.155 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:46:04.155 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T06:46:04.211 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-04-22T06:46:04.212 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-04-22T06:46:04.212 DEBUG:teuthology.orchestra.run.smithi079:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=86f7587a5a09af35f0895e1a2d08527638fae697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-04-22T06:46:04.272 INFO:tasks.workunit.client.0.smithi079.stdout:getting iogen 2024-04-22T06:46:04.273 INFO:tasks.workunit.client.0.smithi079.stderr:+ echo 'getting iogen' 2024-04-22T06:46:04.273 INFO:tasks.workunit.client.0.smithi079.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-22T06:46:04.294 INFO:tasks.workunit.client.0.smithi079.stderr:--2024-04-22 06:46:04-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-22T06:46:04.295 INFO:tasks.workunit.client.0.smithi079.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-22T06:46:04.316 INFO:tasks.workunit.client.0.smithi079.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-22T06:46:04.337 INFO:tasks.workunit.client.0.smithi079.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T06:46:04.337 INFO:tasks.workunit.client.0.smithi079.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-04-22T06:46:04.338 INFO:tasks.workunit.client.0.smithi079.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-04-22T06:46:04.341 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:04.341 INFO:tasks.workunit.client.0.smithi079.stderr: 0K ....... 100% 1.34G=0s 2024-04-22T06:46:04.341 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:04.341 INFO:tasks.workunit.client.0.smithi079.stderr:2024-04-22 06:46:04 (1.34 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-04-22T06:46:04.341 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-04-22T06:46:04.341 INFO:tasks.workunit.client.0.smithi079.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-04-22T06:46:04.354 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0 2024-04-22T06:46:04.356 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0/src 2024-04-22T06:46:04.357 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0/src/iogen.8 2024-04-22T06:46:04.358 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0/src/iogen.c 2024-04-22T06:46:04.359 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0/README 2024-04-22T06:46:04.360 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0/Makefile.bsd 2024-04-22T06:46:04.361 INFO:tasks.workunit.client.0.smithi079.stdout:iogen_3.1p0/Makefile 2024-04-22T06:46:04.362 INFO:tasks.workunit.client.0.smithi079.stderr:+ cd iogen_3.1p0 2024-04-22T06:46:04.365 INFO:tasks.workunit.client.0.smithi079.stdout:making iogen 2024-04-22T06:46:04.365 INFO:tasks.workunit.client.0.smithi079.stderr:+ echo 'making iogen' 2024-04-22T06:46:04.366 INFO:tasks.workunit.client.0.smithi079.stderr:+ make 2024-04-22T06:46:04.371 INFO:tasks.workunit.client.0.smithi079.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-04-22T06:46:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:04 smithi193 ceph-mon[31815]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:04.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:04 smithi190 ceph-mon[31421]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:04.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:04 smithi079 ceph-mon[27946]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:04.607 INFO:tasks.workunit.client.0.smithi079.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-04-22T06:46:04.691 INFO:tasks.workunit.client.0.smithi079.stdout:running iogen 2024-04-22T06:46:04.692 INFO:tasks.workunit.client.0.smithi079.stderr:+ echo 'running iogen' 2024-04-22T06:46:04.692 INFO:tasks.workunit.client.0.smithi079.stderr:+ ./iogen -n 5 -s 2g 2024-04-22T06:46:04.692 INFO:tasks.workunit.client.0.smithi079.stderr:child spawned 82910 2024-04-22T06:46:04.692 INFO:tasks.workunit.client.0.smithi079.stderr:child spawned 82911 2024-04-22T06:46:04.693 INFO:tasks.workunit.client.0.smithi079.stderr:child spawned 82912 2024-04-22T06:46:04.693 INFO:tasks.workunit.client.0.smithi079.stderr:child spawned 82913 2024-04-22T06:46:04.693 INFO:tasks.workunit.client.0.smithi079.stderr:child spawned 82914 2024-04-22T06:46:04.693 INFO:tasks.workunit.client.0.smithi079.stdout:sleep for 10 min 2024-04-22T06:46:04.693 INFO:tasks.workunit.client.0.smithi079.stderr:+ echo 'sleep for 10 min' 2024-04-22T06:46:04.694 INFO:tasks.workunit.client.0.smithi079.stderr:+ sleep 600 2024-04-22T06:46:05.248 INFO:tasks.quiescer.fs.[cephfs]:Total duration (38.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T06:46:05.248 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.35 2024-04-22T06:46:05.248 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:05.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.827+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1569104177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 msgr2=0x7fcda8115360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:05.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.827+0000 7fcdacdb2640 1 --2- 172.21.15.79:0/1569104177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 0x7fcda8115360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcda001ace0 tx=0x7fcda003d150 comp rx=0 tx=0).stop 2024-04-22T06:46:05.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.829+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1569104177 shutdown_connections 2024-04-22T06:46:05.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.829+0000 7fcdacdb2640 1 --2- 172.21.15.79:0/1569104177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 0x7fcda8115360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.829+0000 7fcdacdb2640 1 --2- 172.21.15.79:0/1569104177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcda8108d90 0x7fcda8109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.829+0000 7fcdacdb2640 1 --2- 172.21.15.79:0/1569104177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda810a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.829+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1569104177 >> 172.21.15.79:0/1569104177 conn(0x7fcda80779a0 msgr2=0x7fcda8079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:05.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.830+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1569104177 shutdown_connections 2024-04-22T06:46:05.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.831+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1569104177 wait complete. 2024-04-22T06:46:05.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.832+0000 7fcdacdb2640 1 Processor -- start 2024-04-22T06:46:05.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 -- start start 2024-04-22T06:46:05.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcda8108d90 0x7fcda8114ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 0x7fcda8110f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda8111490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcda806e2a0 con 0x7fcda8108d90 2024-04-22T06:46:05.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcda806e120 con 0x7fcda810a380 2024-04-22T06:46:05.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.833+0000 7fcdacdb2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcda806e420 con 0x7fcda8109750 2024-04-22T06:46:05.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.835+0000 7fcda7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda8111490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.835+0000 7fcda7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda8111490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:48536/0 (socket says 172.21.15.79:48536) 2024-04-22T06:46:05.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.835+0000 7fcda7fff640 1 -- 172.21.15.79:0/1521377923 learned_addr learned my addr 172.21.15.79:0/1521377923 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:05.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.837+0000 7fcda7fff640 1 -- 172.21.15.79:0/1521377923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 msgr2=0x7fcda8110f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:05.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.837+0000 7fcda7fff640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 0x7fcda8110f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.837+0000 7fcda7fff640 1 -- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcda8108d90 msgr2=0x7fcda8114ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:05.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.837+0000 7fcda7fff640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcda8108d90 0x7fcda8114ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.837+0000 7fcda7fff640 1 -- 172.21.15.79:0/1521377923 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda8111a50 con 0x7fcda810a380 2024-04-22T06:46:05.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.840+0000 7fcda7fff640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda8111490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcda0048040 tx=0x7fcda0003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:05.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.845+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0052020 con 0x7fcda810a380 2024-04-22T06:46:05.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.845+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda00044f0 con 0x7fcda810a380 2024-04-22T06:46:05.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.845+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda004a040 con 0x7fcda810a380 2024-04-22T06:46:05.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.845+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1521377923 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda8111ce0 con 0x7fcda810a380 2024-04-22T06:46:05.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.854+0000 7fcdacdb2640 1 -- 172.21.15.79:0/1521377923 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda807ca00 con 0x7fcda810a380 2024-04-22T06:46:05.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.864+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcda0055020 con 0x7fcda810a380 2024-04-22T06:46:05.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.866+0000 7fcda4ff9640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd900a3eb0 0x7fcd900a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.867+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fcda0103540 con 0x7fcda810a380 2024-04-22T06:46:05.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.870+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda807c4f0 con 0x7fcda810a380 2024-04-22T06:46:05.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.875+0000 7fcda77fe640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd900a3eb0 0x7fcd900a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.878+0000 7fcda77fe640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd900a3eb0 0x7fcd900a6370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcda8076270 tx=0x7fcd98006c40 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:05.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:05.885+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcda00cfc10 con 0x7fcda810a380 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 -- 172.21.15.79:0/696920353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f41104c0 msgr2=0x7f06f41108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 --2- 172.21.15.79:0/696920353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f41104c0 0x7f06f41108a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06e801a910 tx=0x7f06e803cca0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 -- 172.21.15.79:0/696920353 shutdown_connections 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 --2- 172.21.15.79:0/696920353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f410ecc0 0x7f06f4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 --2- 172.21.15.79:0/696920353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f410e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 --2- 172.21.15.79:0/696920353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f41104c0 0x7f06f41108a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 -- 172.21.15.79:0/696920353 >> 172.21.15.79:0/696920353 conn(0x7f06f406cf00 msgr2=0x7f06f406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 -- 172.21.15.79:0/696920353 shutdown_connections 2024-04-22T06:46:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.894+0000 7f06fafdb640 1 -- 172.21.15.79:0/696920353 wait complete. 2024-04-22T06:46:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.895+0000 7f06fafdb640 1 Processor -- start 2024-04-22T06:46:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.895+0000 7f06fafdb640 1 -- start start 2024-04-22T06:46:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.896+0000 7f06fafdb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f41a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.896+0000 7f06fafdb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f410ecc0 0x7f06f41a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.896+0000 7f06f8d77640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f41a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.896+0000 7f06f8d77640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f41a48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49272/0 (socket says 172.21.15.79:49272) 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06f8d77640 1 -- 172.21.15.79:0/3352614435 learned_addr learned my addr 172.21.15.79:0/3352614435 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f41104c0 0x7f06f419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f06f4112680 con 0x7f06f410e220 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06f4112500 con 0x7f06f41104c0 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06f4112800 con 0x7f06f410ecc0 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06f8d77640 1 -- 172.21.15.79:0/3352614435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f410ecc0 msgr2=0x7f06f41a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06f8d77640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f410ecc0 0x7f06f41a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06f8d77640 1 -- 172.21.15.79:0/3352614435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f41104c0 msgr2=0x7f06f419eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06f8d77640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f41104c0 0x7f06f419eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.897+0000 7f06f8d77640 1 -- 172.21.15.79:0/3352614435 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f419f220 con 0x7f06f410e220 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.898+0000 7f06f8d77640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f41a48f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f06e803d2b0 tx=0x7f06e80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.898+0000 7f06f1ffb640 1 -- 172.21.15.79:0/3352614435 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e8051020 con 0x7f06f410e220 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.898+0000 7f06f1ffb640 1 -- 172.21.15.79:0/3352614435 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06e8045440 con 0x7f06f410e220 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.898+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f419f4b0 con 0x7f06f410e220 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.898+0000 7f06f1ffb640 1 -- 172.21.15.79:0/3352614435 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e804d420 con 0x7f06f410e220 2024-04-22T06:46:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.899+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f41ab7b0 con 0x7f06f410e220 2024-04-22T06:46:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.903+0000 7f06fdbc5740 1 Processor -- start 2024-04-22T06:46:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.903+0000 7f06fdbc5740 1 -- start start 2024-04-22T06:46:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.904+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baecd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.904+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baecdd50 0x55f2baece1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.904+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 0x55f2baed96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.904+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f2bae328d0 con 0x55f2baeceac0 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.904+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2badcfe80 con 0x55f2baecdd50 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.904+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f2bae2e890 con 0x55f2baed72d0 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f9578640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baecd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 0x55f2baed96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 0x55f2baed96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:52276/0 (socket says 172.21.15.79:52276) 2024-04-22T06:46:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 -- 172.21.15.79:0/3834466464 learned_addr learned my addr 172.21.15.79:0/3834466464 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 -- 172.21.15.79:0/3834466464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baecdd50 msgr2=0x55f2baece1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baecdd50 0x55f2baece1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 -- 172.21.15.79:0/3834466464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 msgr2=0x55f2baecd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baecd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.905+0000 7f06f3fff640 1 -- 172.21.15.79:0/3834466464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2badf1320 con 0x55f2baed72d0 2024-04-22T06:46:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.906+0000 7f06f3fff640 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 0x55f2baed96c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009d70 tx=0x7f06ec00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd3dd80f7973c5ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.906+0000 7f06daffd640 1 -- 172.21.15.79:0/3834466464 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec018070 con 0x55f2baed72d0 2024-04-22T06:46:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.907+0000 7f06daffd640 1 -- 172.21.15.79:0/3834466464 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06ec0119d0 con 0x55f2baed72d0 2024-04-22T06:46:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.907+0000 7f06daffd640 1 -- 172.21.15.79:0/3834466464 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec013aa0 con 0x55f2baed72d0 2024-04-22T06:46:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.908+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3834466464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 msgr2=0x55f2baed96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.908+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 0x55f2baed96c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009d70 tx=0x7f06ec00b810 comp rx=0 tx=0).stop 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06f1ffb640 1 -- 172.21.15.79:0/3352614435 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f06e8054020 con 0x7f06f410e220 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3834466464 shutdown_connections 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baecdd50 0x55f2baece1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed72d0 0x55f2baed96c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3834466464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baecd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3834466464 >> 172.21.15.79:0/3834466464 conn(0x55f2bae94c90 msgr2=0x55f2baec8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3834466464 shutdown_connections 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.911+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3834466464 wait complete. 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.912+0000 7f06fdbc5740 1 -- start start 2024-04-22T06:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.912+0000 7f06fdbc5740 1 Processor -- start 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 0x55f2baed6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 0x55f2baed4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f2baeda000 con 0x55f2baed46a0 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2badbaeb0 con 0x55f2baed72d0 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f2baeda180 con 0x55f2baeceac0 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06fdbc5740 10 client.0 _ll_register_callbacks cb 0x55f2bade7b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.913+0000 7f06f1ffb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f06dc0a3cf0 0x7f06dc0a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 0x55f2baed4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 0x55f2baed4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49286/0 (socket says 172.21.15.79:49286) 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 -- 172.21.15.79:0/3672097856 learned_addr learned my addr 172.21.15.79:0/3672097856 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f1ffb640 1 -- 172.21.15.79:0/3352614435 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f06e81030e0 con 0x7f06f410e220 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 -- 172.21.15.79:0/3672097856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 msgr2=0x55f2baed6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 0x55f2baed6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 -- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 msgr2=0x55f2baed4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.914+0000 7f06f3fff640 1 -- 172.21.15.79:0/3672097856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2bab45210 con 0x55f2baed46a0 2024-04-22T06:46:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.915+0000 7f06f3fff640 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 0x55f2baed4b20 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f06ec007190 tx=0x7f06ec011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a772cd2c262f8bab server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.915+0000 7f06d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.915+0000 7f06d3fff640 1 -- 172.21.15.79:0/3672097856 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec018040 con 0x55f2baed46a0 2024-04-22T06:46:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.916+0000 7f06d3fff640 1 -- 172.21.15.79:0/3672097856 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f06ec011520 con 0x55f2baed46a0 2024-04-22T06:46:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.916+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06ec0119d0 con 0x55f2baed46a0 2024-04-22T06:46:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.917+0000 7f06d3fff640 1 -- 172.21.15.79:0/3672097856 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f06ec02b050 con 0x55f2baed46a0 2024-04-22T06:46:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.917+0000 7f06fdbc5740 10 client.16092 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.917+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2bac11620 con 0x55f2baed46a0 2024-04-22T06:46:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.919+0000 7f06f9578640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f06dc0a3cf0 0x7f06dc0a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06d3fff640 1 -- 172.21.15.79:0/3672097856 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06ec0119d0 con 0x55f2baed46a0 2024-04-22T06:46:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06d3fff640 1 -- 172.21.15.79:0/3672097856 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f06ec01b050 con 0x55f2baed46a0 2024-04-22T06:46:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06fdbc5740 10 client.16092 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06fdbc5740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06fdbc5740 20 client.16092 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baee74a0 0x55f2baee9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06fdbc5740 4 client.16092 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.922+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2bae46880 con 0x55f2baee74a0 2024-04-22T06:46:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.924+0000 7f06f9578640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f06dc0a3cf0 0x7f06dc0a61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f06e40058c0 tx=0x7f06e4005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.925+0000 7f06f8d77640 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baee74a0 0x55f2baee9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.930+0000 7f06d3fff640 10 client.16092 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.930+0000 7f06f8d77640 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baee74a0 0x55f2baee9890 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32a8ff631f8a32e1 server_cookie=86d03520836152b9 in_seq=0 out_seq=0 2024-04-22T06:46:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.935+0000 7f06d3fff640 1 -- 172.21.15.79:0/3672097856 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2bae46880 con 0x55f2baee74a0 2024-04-22T06:46:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.935+0000 7f06d3fff640 10 client.16092 handle_command_reply: tid=0 2024-04-22T06:46:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.935+0000 7f06fdbc5740 1 client.16092 shutdown 2024-04-22T06:46:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 msgr2=0x55f2baed4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 0x55f2baed4b20 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f06ec007190 tx=0x7f06ec011db0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 shutdown_connections 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baee74a0 0x55f2baee9890 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed46a0 0x55f2baed4b20 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/3672097856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 0x55f2baed6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 >> 172.21.15.79:0/3672097856 conn(0x55f2bae94c90 msgr2=0x55f2baeca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.945+0000 7f06d3fff640 0 client.16092 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.946+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 shutdown_connections 2024-04-22T06:46:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.946+0000 7f06fdbc5740 1 -- 172.21.15.79:0/3672097856 wait complete. 2024-04-22T06:46:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:05.947+0000 7f06fdbc5740 20 client.16092 trim_cache size 0 max 16384 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.006+0000 7f06fdbc5740 1 Processor -- start 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.006+0000 7f06fdbc5740 1 -- start start 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 0x55f2baee2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed46a0 0x55f2baed6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 0x55f2baeeef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41b6b00 con 0x55f2baed72d0 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41b43d0 con 0x55f2baed46a0 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41b2220 con 0x55f2baeceac0 2024-04-22T06:46:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06f9578640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 0x55f2baeeef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06f9578640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 0x55f2baeeef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49290/0 (socket says 172.21.15.79:49290) 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.007+0000 7f06f9578640 1 -- 172.21.15.79:0/2797982614 learned_addr learned my addr 172.21.15.79:0/2797982614 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.008+0000 7f06f9578640 1 -- 172.21.15.79:0/2797982614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 msgr2=0x55f2baee2f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.008+0000 7f06f9578640 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 0x55f2baee2f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.008+0000 7f06f9578640 1 -- 172.21.15.79:0/2797982614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed46a0 msgr2=0x55f2baed6890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.008+0000 7f06f9578640 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed46a0 0x55f2baed6890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.008+0000 7f06f9578640 1 -- 172.21.15.79:0/2797982614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2baed9ee0 con 0x55f2baed72d0 2024-04-22T06:46:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.008+0000 7f06f9578640 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 0x55f2baeeef40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f06e400fd90 tx=0x7f06e4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9541168f735e589f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.013+0000 7f06da7fc640 1 -- 172.21.15.79:0/2797982614 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4022070 con 0x55f2baed72d0 2024-04-22T06:46:06.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.013+0000 7f06da7fc640 1 -- 172.21.15.79:0/2797982614 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06e4011e50 con 0x55f2baed72d0 2024-04-22T06:46:06.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.014+0000 7f06fdbc5740 1 -- 172.21.15.79:0/2797982614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 msgr2=0x55f2baeeef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.014+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 0x55f2baeeef40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f06e400fd90 tx=0x7f06e4012000 comp rx=0 tx=0).stop 2024-04-22T06:46:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 -- 172.21.15.79:0/2797982614 shutdown_connections 2024-04-22T06:46:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed46a0 0x55f2baed6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baed72d0 0x55f2baeeef40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/2797982614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baeceac0 0x55f2baee2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 -- 172.21.15.79:0/2797982614 >> 172.21.15.79:0/2797982614 conn(0x55f2bae94c90 msgr2=0x55f2baec8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 -- 172.21.15.79:0/2797982614 shutdown_connections 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 -- 172.21.15.79:0/2797982614 wait complete. 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.028+0000 7f06fdbc5740 1 -- start start 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 Processor -- start 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baedf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed46a0 0x55f2baed12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41b2890 con 0x55f2baeceac0 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41b3d00 con 0x55f2baed72d0 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41b2f60 con 0x55f2baed46a0 2024-04-22T06:46:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.029+0000 7f06fdbc5740 10 client.0 _ll_register_callbacks cb 0x7f06ec016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.032+0000 7f06f8d77640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baedf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.032+0000 7f06f8d77640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baedf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49294/0 (socket says 172.21.15.79:49294) 2024-04-22T06:46:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.032+0000 7f06f8d77640 1 -- 172.21.15.79:0/1978095223 learned_addr learned my addr 172.21.15.79:0/1978095223 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.032+0000 7f06f3fff640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed46a0 0x55f2baed12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.032+0000 7f06f9578640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.035+0000 7f06f8d77640 1 -- 172.21.15.79:0/1978095223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed46a0 msgr2=0x55f2baed12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.035+0000 7f06f8d77640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed46a0 0x55f2baed12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.035+0000 7f06f8d77640 1 -- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 msgr2=0x55f2baed1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.035+0000 7f06f8d77640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.035+0000 7f06f8d77640 1 -- 172.21.15.79:0/1978095223 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2baed1d70 con 0x55f2baeceac0 2024-04-22T06:46:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.036+0000 7f06f8d77640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baedf620 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f06e80cee10 tx=0x7f06e80cac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63801b19d0e7902d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.036+0000 7f06d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.037+0000 7f06d3fff640 1 -- 172.21.15.79:0/1978095223 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e80d34c0 con 0x55f2baeceac0 2024-04-22T06:46:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.037+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06e4011e50 con 0x55f2baeceac0 2024-04-22T06:46:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.037+0000 7f06d3fff640 1 -- 172.21.15.79:0/1978095223 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f06e801a410 con 0x55f2baeceac0 2024-04-22T06:46:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.038+0000 7f06d3fff640 1 -- 172.21.15.79:0/1978095223 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f06e80d7d80 con 0x55f2baeceac0 2024-04-22T06:46:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.038+0000 7f06fdbc5740 10 client.16098 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.038+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2baed87c0 con 0x55f2baeceac0 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06d3fff640 1 -- 172.21.15.79:0/1978095223 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06e4011e50 con 0x55f2baeceac0 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06d3fff640 1 -- 172.21.15.79:0/1978095223 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f06e80cb780 con 0x55f2baeceac0 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06fdbc5740 10 client.16098 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06fdbc5740 10 client.16098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06fdbc5740 20 client.16098 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baf4dc60 0x55f2baf50080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06fdbc5740 4 client.16098 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.039+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2badd23d0 con 0x55f2baf4dc60 2024-04-22T06:46:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.041+0000 7f06f3fff640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baf4dc60 0x55f2baf50080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.043+0000 7f06f3fff640 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baf4dc60 0x55f2baf50080 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8abb07ee4f216ea4 server_cookie=4ccad7157eb38215 in_seq=0 out_seq=0 2024-04-22T06:46:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.044+0000 7f06d3fff640 10 client.16098 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:06.314 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.315+0000 7f06d3fff640 1 -- 172.21.15.79:0/1978095223 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2badd23d0 con 0x55f2baf4dc60 2024-04-22T06:46:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.315+0000 7f06d3fff640 10 client.16098 handle_command_reply: tid=0 2024-04-22T06:46:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.315+0000 7f06fdbc5740 1 client.16098 shutdown 2024-04-22T06:46:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 msgr2=0x55f2baedf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.315 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baedf620 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f06e80cee10 tx=0x7f06e80cac90 comp rx=0 tx=0).stop 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 shutdown_connections 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2baf4dc60 0x55f2baf50080 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2baed72d0 0x55f2baed1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2baed46a0 0x55f2baed12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 --2- 172.21.15.79:0/1978095223 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2baeceac0 0x55f2baedf620 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 >> 172.21.15.79:0/1978095223 conn(0x55f2bae94c90 msgr2=0x55f2baecf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.316 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06d3fff640 0 client.16098 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 shutdown_connections 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 1 -- 172.21.15.79:0/1978095223 wait complete. 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.316+0000 7f06fdbc5740 20 client.16098 trim_cache size 0 max 16384 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:46:06.317 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f06dc0a3cf0 msgr2=0x7f06dc0a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f06dc0a3cf0 0x7f06dc0a61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f06e40058c0 tx=0x7f06e4005850 comp rx=0 tx=0).stop 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 msgr2=0x7f06f41a48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f41a48f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f06e803d2b0 tx=0x7f06e80048f0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 shutdown_connections 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f06dc0a3cf0 0x7f06dc0a61b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f41104c0 0x7f06f419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f410ecc0 0x7f06f41a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 --2- 172.21.15.79:0/3352614435 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06f410e220 0x7f06f41a48f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 >> 172.21.15.79:0/3352614435 conn(0x7f06f406cf00 msgr2=0x7f06f410a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 shutdown_connections 2024-04-22T06:46:06.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:06.319+0000 7f06fafdb640 1 -- 172.21.15.79:0/3352614435 wait complete. 2024-04-22T06:46:06.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:06 smithi193 ceph-mon[31815]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:06.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:06 smithi190 ceph-mon[31421]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:06.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.457+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcda8062b50 con 0x7fcda810a380 2024-04-22T06:46:06.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.471+0000 7fcda4ff9640 1 -- 172.21.15.79:0/1521377923 <== mon.2 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11884 (secure 0 0 0) 0x7fcda00cf510 con 0x7fcda810a380 2024-04-22T06:46:06.471 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:46:06.471 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:45:36.000223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24413,"mds_2":24406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":24,"state":"up:active","state_seq":57,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":26,"state":"up:active","state_seq":58,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24406,24413]},"id":1}]} 2024-04-22T06:46:06.472 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 31 2024-04-22T06:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.477+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd900a3eb0 msgr2=0x7fcd900a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.477+0000 7fcd8e7fc640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd900a3eb0 0x7fcd900a6370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcda8076270 tx=0x7fcd98006c40 comp rx=0 tx=0).stop 2024-04-22T06:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.477+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 msgr2=0x7fcda8111490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.477+0000 7fcd8e7fc640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda8111490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcda0048040 tx=0x7fcda0003f10 comp rx=0 tx=0).stop 2024-04-22T06:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.478+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 shutdown_connections 2024-04-22T06:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.478+0000 7fcd8e7fc640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd900a3eb0 0x7fcd900a6370 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcda8076270 tx=0x7fcd98006c40 comp rx=0 tx=0).stop 2024-04-22T06:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.479+0000 7fcd8e7fc640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcda810a380 0x7fcda8111490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.479+0000 7fcd8e7fc640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda8109750 0x7fcda8110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.479+0000 7fcd8e7fc640 1 --2- 172.21.15.79:0/1521377923 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcda8108d90 0x7fcda8114ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.478 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.479+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 >> 172.21.15.79:0/1521377923 conn(0x7fcda80779a0 msgr2=0x7fcda810c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.482+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 shutdown_connections 2024-04-22T06:46:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.484+0000 7fcd8e7fc640 1 -- 172.21.15.79:0/1521377923 wait complete. 2024-04-22T06:46:06.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:06 smithi079 ceph-mon[27946]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:06.510 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 79.35 --await' 2024-04-22T06:46:06.510 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 79.35 --await 2024-04-22T06:46:06.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.726+0000 7f4b70cf2640 1 -- 172.21.15.79:0/172225206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 msgr2=0x7f4b6c101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.726+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/172225206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c101480 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009900 tx=0x7f4b6002bc50 comp rx=0 tx=0).stop 2024-04-22T06:46:06.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.727+0000 7f4b70cf2640 1 -- 172.21.15.79:0/172225206 shutdown_connections 2024-04-22T06:46:06.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.727+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/172225206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c101a50 0x7f4b6c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.727+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/172225206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b6c106170 0x7f4b6c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.727+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/172225206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c101480 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.728+0000 7f4b70cf2640 1 -- 172.21.15.79:0/172225206 >> 172.21.15.79:0/172225206 conn(0x7f4b6c077f50 msgr2=0x7f4b6c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.731+0000 7f4b70cf2640 1 -- 172.21.15.79:0/172225206 shutdown_connections 2024-04-22T06:46:06.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.731+0000 7f4b70cf2640 1 -- 172.21.15.79:0/172225206 wait complete. 2024-04-22T06:46:06.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.732+0000 7f4b70cf2640 1 Processor -- start 2024-04-22T06:46:06.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 -- start start 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c106170 0x7f4b6c197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b6c19bfa0 0x7f4b6c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1111a0 con 0x7f4b6c1010a0 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c111020 con 0x7f4b6c106170 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b70cf2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c111320 con 0x7f4b6c19bfa0 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b6a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b6a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c1976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49304/0 (socket says 172.21.15.79:49304) 2024-04-22T06:46:06.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.733+0000 7f4b6a59c640 1 -- 172.21.15.79:0/1580771394 learned_addr learned my addr 172.21.15.79:0/1580771394 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.734+0000 7f4b69d9b640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c106170 0x7f4b6c197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.735+0000 7f4b6a59c640 1 -- 172.21.15.79:0/1580771394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b6c19bfa0 msgr2=0x7f4b6c19c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.735+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b6c19bfa0 0x7f4b6c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.735+0000 7f4b6a59c640 1 -- 172.21.15.79:0/1580771394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c106170 msgr2=0x7f4b6c197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.735+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c106170 0x7f4b6c197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.735+0000 7f4b6a59c640 1 -- 172.21.15.79:0/1580771394 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c19ca90 con 0x7f4b6c1010a0 2024-04-22T06:46:06.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.742+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c1976e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4b6002c260 tx=0x7f4b6002c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.743+0000 7f4b5b7fe640 1 -- 172.21.15.79:0/1580771394 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60043020 con 0x7f4b6c1010a0 2024-04-22T06:46:06.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.743+0000 7f4b5b7fe640 1 -- 172.21.15.79:0/1580771394 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b6002cc80 con 0x7f4b6c1010a0 2024-04-22T06:46:06.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.743+0000 7f4b5b7fe640 1 -- 172.21.15.79:0/1580771394 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60003ca0 con 0x7f4b6c1010a0 2024-04-22T06:46:06.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.744+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c19cd20 con 0x7f4b6c1010a0 2024-04-22T06:46:06.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.744+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c069380 con 0x7f4b6c1010a0 2024-04-22T06:46:06.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.749+0000 7f4b5b7fe640 1 -- 172.21.15.79:0/1580771394 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4b60046020 con 0x7f4b6c1010a0 2024-04-22T06:46:06.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.749+0000 7f4b739be740 1 Processor -- start 2024-04-22T06:46:06.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 -- start start 2024-04-22T06:46:06.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487a2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f0487080b0 con 0x55f0487a2740 2024-04-22T06:46:06.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f0486b9680 con 0x55f0487b4300 2024-04-22T06:46:06.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.751+0000 7f4b739be740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f0486c5c60 con 0x55f0487a1d90 2024-04-22T06:46:06.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.752+0000 7f4b6ad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.752+0000 7f4b5b7fe640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4b540a3da0 0x7f4b540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.753+0000 7f4b5b7fe640 1 -- 172.21.15.79:0/1580771394 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f4b600f2a80 con 0x7f4b6c1010a0 2024-04-22T06:46:06.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.754+0000 7f4b6ad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487a2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:52302/0 (socket says 172.21.15.79:52302) 2024-04-22T06:46:06.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.754+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/3014556591 learned_addr learned my addr 172.21.15.79:0/3014556591 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.754+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.754+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4b540a3da0 0x7f4b540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.755+0000 7f4b6a59c640 1 -- 172.21.15.79:0/3014556591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 msgr2=0x55f0487a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.755+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.755+0000 7f4b6a59c640 1 -- 172.21.15.79:0/3014556591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 msgr2=0x55f0487b4740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:06.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.755+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b4740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.755+0000 7f4b6a59c640 1 -- 172.21.15.79:0/3014556591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0484eac90 con 0x55f0487a2740 2024-04-22T06:46:06.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.756+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4b540a3da0 0x7f4b540a6260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4b640108b0 tx=0x7f4b64009970 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.757+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b3ce0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4b600f0260 tx=0x7f4b6002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc16d10d8b952486 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.759+0000 7f4b58ff9640 1 -- 172.21.15.79:0/3014556591 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6003c350 con 0x55f0487a2740 2024-04-22T06:46:06.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.759+0000 7f4b58ff9640 1 -- 172.21.15.79:0/3014556591 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b600bf1b0 con 0x55f0487a2740 2024-04-22T06:46:06.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.759+0000 7f4b58ff9640 1 -- 172.21.15.79:0/3014556591 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600f03d0 con 0x55f0487a2740 2024-04-22T06:46:06.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 -- 172.21.15.79:0/3014556591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 msgr2=0x55f0487b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b3ce0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4b600f0260 tx=0x7f4b6002bc50 comp rx=0 tx=0).stop 2024-04-22T06:46:06.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 -- 172.21.15.79:0/3014556591 shutdown_connections 2024-04-22T06:46:06.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b3ce0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 --2- 172.21.15.79:0/3014556591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 -- 172.21.15.79:0/3014556591 >> 172.21.15.79:0/3014556591 conn(0x55f0487a46c0 msgr2=0x55f0487a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 -- 172.21.15.79:0/3014556591 shutdown_connections 2024-04-22T06:46:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.760+0000 7f4b739be740 1 -- 172.21.15.79:0/3014556591 wait complete. 2024-04-22T06:46:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.761+0000 7f4b739be740 1 -- start start 2024-04-22T06:46:06.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.761+0000 7f4b739be740 1 Processor -- start 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f04870b400 con 0x55f0487a2740 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f0486c7840 con 0x55f0487b4300 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f048710720 con 0x55f0487a1d90 2024-04-22T06:46:06.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b739be740 10 client.0 _ll_register_callbacks cb 0x55f048718ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:06.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b69d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b69d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49324/0 (socket says 172.21.15.79:49324) 2024-04-22T06:46:06.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.762+0000 7f4b69d9b640 1 -- 172.21.15.79:0/2422427812 learned_addr learned my addr 172.21.15.79:0/2422427812 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.763+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.764+0000 7f4b69d9b640 1 -- 172.21.15.79:0/2422427812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 msgr2=0x55f0487b8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.764+0000 7f4b69d9b640 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.764+0000 7f4b69d9b640 1 -- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 msgr2=0x55f0487b99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.764+0000 7f4b69d9b640 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.764+0000 7f4b69d9b640 1 -- 172.21.15.79:0/2422427812 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f04854b420 con 0x55f0487a2740 2024-04-22T06:46:06.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.766+0000 7f4b69d9b640 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b9460 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00e680 tx=0x7f4b5c00eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9656c0bb59b60e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.766+0000 7f4b51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:06.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.766+0000 7f4b51ffb640 1 -- 172.21.15.79:0/2422427812 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c005220 con 0x55f0487a2740 2024-04-22T06:46:06.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.766+0000 7f4b51ffb640 1 -- 172.21.15.79:0/2422427812 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f4b5c0053c0 con 0x55f0487a2740 2024-04-22T06:46:06.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.767+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b600bf1b0 con 0x55f0487a2740 2024-04-22T06:46:06.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.769+0000 7f4b51ffb640 1 -- 172.21.15.79:0/2422427812 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4b5c00ee80 con 0x55f0487a2740 2024-04-22T06:46:06.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.769+0000 7f4b739be740 10 client.16110 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:06.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.769+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f0483de580 con 0x55f0487a2740 2024-04-22T06:46:06.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b51ffb640 1 -- 172.21.15.79:0/2422427812 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b600bf1b0 con 0x55f0487a2740 2024-04-22T06:46:06.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b51ffb640 1 -- 172.21.15.79:0/2422427812 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f4b5c020070 con 0x55f0487a2740 2024-04-22T06:46:06.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b739be740 10 client.16110 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:06.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b739be740 10 client.16110 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:46:06.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b739be740 20 client.16110 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:06.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b739be740 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f0487da180 0x55f0487dc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b739be740 4 client.16110 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:06.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.770+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f048719b10 con 0x55f0487da180 2024-04-22T06:46:06.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.773+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f0487da180 0x55f0487dc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.782+0000 7f4b51ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:06.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.782+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f0487da180 0x55f0487dc600 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5707d39429c0a8c server_cookie=6c46efec9949172c in_seq=0 out_seq=0 2024-04-22T06:46:06.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.788+0000 7f4b51ffb640 1 -- 172.21.15.79:0/2422427812 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f048719b10 con 0x55f0487da180 2024-04-22T06:46:06.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.788+0000 7f4b51ffb640 10 client.16110 handle_command_reply: tid=0 2024-04-22T06:46:06.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 client.16110 shutdown 2024-04-22T06:46:06.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 msgr2=0x55f0487b9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b9460 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00e680 tx=0x7f4b5c00eb40 comp rx=0 tx=0).stop 2024-04-22T06:46:06.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 shutdown_connections 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f0487da180 0x55f0487dc600 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487b4300 0x55f0487b99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487a2740 0x55f0487b9460 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b51ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 --2- 172.21.15.79:0/2422427812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a1d90 0x55f0487b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 >> 172.21.15.79:0/2422427812 conn(0x55f0487a46c0 msgr2=0x55f0487a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 shutdown_connections 2024-04-22T06:46:06.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 1 -- 172.21.15.79:0/2422427812 wait complete. 2024-04-22T06:46:06.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.789+0000 7f4b739be740 20 client.16110 trim_cache size 0 max 16384 2024-04-22T06:46:06.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.853+0000 7f4b739be740 1 Processor -- start 2024-04-22T06:46:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.853+0000 7f4b739be740 1 -- start start 2024-04-22T06:46:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b739be740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b739be740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b739be740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b739be740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c073460 con 0x55f0487b4300 2024-04-22T06:46:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b6ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b6ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487e1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49326/0 (socket says 172.21.15.79:49326) 2024-04-22T06:46:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.854+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/1722270877 learned_addr learned my addr 172.21.15.79:0/1722270877 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.856+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c0712b0 con 0x55f0487a1d90 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.856+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c072d90 con 0x55f0487a2740 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.857+0000 7f4b69d9b640 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.857+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/1722270877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 msgr2=0x55f0487b97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.857+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487b97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.857+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/1722270877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 msgr2=0x55f0487d5e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.857+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d5e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.857+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/1722270877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0487d8cd0 con 0x55f0487b4300 2024-04-22T06:46:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.858+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487e1ed0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4b64010ee0 tx=0x7f4b64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2afca803ba627a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.858+0000 7f4b53fff640 1 -- 172.21.15.79:0/1722270877 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64014ea0 con 0x55f0487b4300 2024-04-22T06:46:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.858+0000 7f4b53fff640 1 -- 172.21.15.79:0/1722270877 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b64013040 con 0x55f0487b4300 2024-04-22T06:46:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.858+0000 7f4b53fff640 1 -- 172.21.15.79:0/1722270877 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64023d20 con 0x55f0487b4300 2024-04-22T06:46:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.859+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 msgr2=0x55f0487e1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.859+0000 7f4b739be740 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487e1ed0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4b64010ee0 tx=0x7f4b64004ad0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 shutdown_connections 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487e1ed0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487b97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 --2- 172.21.15.79:0/1722270877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 >> 172.21.15.79:0/1722270877 conn(0x55f0487a46c0 msgr2=0x55f0487b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 shutdown_connections 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.860+0000 7f4b739be740 1 -- 172.21.15.79:0/1722270877 wait complete. 2024-04-22T06:46:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.861+0000 7f4b739be740 1 -- start start 2024-04-22T06:46:06.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.861+0000 7f4b739be740 1 Processor -- start 2024-04-22T06:46:06.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.861+0000 7f4b739be740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.861+0000 7f4b739be740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487d28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.862+0000 7f4b739be740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487cd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.862+0000 7f4b739be740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c071ff0 con 0x55f0487b4300 2024-04-22T06:46:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.862+0000 7f4b739be740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c071920 con 0x55f0487a1d90 2024-04-22T06:46:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.862+0000 7f4b739be740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c0726c0 con 0x55f0487a2740 2024-04-22T06:46:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.862+0000 7f4b739be740 10 client.0 _ll_register_callbacks cb 0x7f4b5c005f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:06.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.863+0000 7f4b6ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.863+0000 7f4b6ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49334/0 (socket says 172.21.15.79:49334) 2024-04-22T06:46:06.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.863+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/479351286 learned_addr learned my addr 172.21.15.79:0/479351286 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:06.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.865+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.865+0000 7f4b69d9b640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487d28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.866+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/479351286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 msgr2=0x55f0487d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.866+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487d28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.866+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 msgr2=0x55f0487d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:06.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.866+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:06.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.866+0000 7f4b6ad9d640 1 -- 172.21.15.79:0/479351286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0487cdd30 con 0x55f0487b4300 2024-04-22T06:46:06.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.868+0000 7f4b6ad9d640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487cd530 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4b640096b0 tx=0x7f4b64018c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1830f6540c5d50e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:06.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.868+0000 7f4b51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:06.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.870+0000 7f4b51ffb640 1 -- 172.21.15.79:0/479351286 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6402f020 con 0x55f0487b4300 2024-04-22T06:46:06.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.870+0000 7f4b51ffb640 1 -- 172.21.15.79:0/479351286 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f4b64023960 con 0x55f0487b4300 2024-04-22T06:46:06.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.870+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b64013040 con 0x55f0487b4300 2024-04-22T06:46:06.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.871+0000 7f4b51ffb640 1 -- 172.21.15.79:0/479351286 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4b6403a050 con 0x55f0487b4300 2024-04-22T06:46:06.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.871+0000 7f4b739be740 10 client.16116 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:06.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.871+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f048841340 con 0x55f0487b4300 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.873+0000 7f4b51ffb640 1 -- 172.21.15.79:0/479351286 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b64023b60 con 0x55f0487b4300 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b51ffb640 1 -- 172.21.15.79:0/479351286 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f4b6402bce0 con 0x55f0487b4300 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b739be740 10 client.16116 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b739be740 10 client.16116 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b739be740 20 client.16116 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b739be740 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f048841a50 0x55f048843e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:06.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b739be740 4 client.16116 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.35, "await": true}] 2024-04-22T06:46:06.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.35, "await": true}) -- 0x7f4b64013040 con 0x55f048841a50 2024-04-22T06:46:06.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.874+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f048841a50 0x55f048843e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:06.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.882+0000 7f4b6a59c640 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f048841a50 0x55f048843e70 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99744cf1df007ce7 server_cookie=66ff8929890f9464 in_seq=0 out_seq=0 2024-04-22T06:46:06.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:06.882+0000 7f4b51ffb640 10 client.16116 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:06.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:06.963 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:07.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 -- 172.21.15.79:0/1028955326 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa66806af50 msgr2=0x7fa66810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 --2- 172.21.15.79:0/1028955326 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa66806af50 0x7fa66810ede0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa660009ca0 tx=0x7fa66002c180 comp rx=0 tx=0).stop 2024-04-22T06:46:07.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 -- 172.21.15.79:0/1028955326 shutdown_connections 2024-04-22T06:46:07.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 --2- 172.21.15.79:0/1028955326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa66806a5b0 0x7fa66806aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 --2- 172.21.15.79:0/1028955326 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa66806af50 0x7fa66810ede0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 --2- 172.21.15.79:0/1028955326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa668108040 0x7fa668108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 -- 172.21.15.79:0/1028955326 >> 172.21.15.79:0/1028955326 conn(0x7fa668077c40 msgr2=0x7fa668078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 -- 172.21.15.79:0/1028955326 shutdown_connections 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.115+0000 7fa67049a640 1 -- 172.21.15.79:0/1028955326 wait complete. 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 Processor -- start 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 -- start start 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66806a5b0 0x7fa66819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa66806af50 0x7fa66819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 0x7fa6681985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa668112250 con 0x7fa668108040 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681120d0 con 0x7fa66806af50 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa67049a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681123d0 con 0x7fa66806a5b0 2024-04-22T06:46:07.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.116+0000 7fa66ea37640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 0x7fa6681985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 0x7fa6681985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49348/0 (socket says 172.21.15.79:49348) 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 -- 172.21.15.79:0/2775505834 learned_addr learned my addr 172.21.15.79:0/2775505834 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66e236640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66806a5b0 0x7fa66819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 -- 172.21.15.79:0/2775505834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66806a5b0 msgr2=0x7fa66819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66806a5b0 0x7fa66819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 -- 172.21.15.79:0/2775505834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa66806af50 msgr2=0x7fa66819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa66806af50 0x7fa66819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 -- 172.21.15.79:0/2775505834 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa668198d70 con 0x7fa668108040 2024-04-22T06:46:07.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa66ea37640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 0x7fa6681985f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa6600098d0 tx=0x7fa660003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa65f7fe640 1 -- 172.21.15.79:0/2775505834 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660039040 con 0x7fa668108040 2024-04-22T06:46:07.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa668199000 con 0x7fa668108040 2024-04-22T06:46:07.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.117+0000 7fa65f7fe640 1 -- 172.21.15.79:0/2775505834 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa660044070 con 0x7fa668108040 2024-04-22T06:46:07.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.118+0000 7fa65f7fe640 1 -- 172.21.15.79:0/2775505834 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660032a90 con 0x7fa668108040 2024-04-22T06:46:07.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.118+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6681a5300 con 0x7fa668108040 2024-04-22T06:46:07.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.121+0000 7fa65f7fe640 1 -- 172.21.15.79:0/2775505834 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa660004610 con 0x7fa668108040 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa65f7fe640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6540a3d20 0x7fa6540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa67316a740 1 Processor -- start 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa65f7fe640 1 -- 172.21.15.79:0/2775505834 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fa6600f2f70 con 0x7fa668108040 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa66e236640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6540a3d20 0x7fa6540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa67316a740 1 -- start start 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa67316a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa11770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa67316a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.122+0000 7fa67316a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa21280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa67316a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558cff9761e0 con 0x558cffa11d40 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa67316a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558cff913390 con 0x558cffa1ee90 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa67316a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558cff9721a0 con 0x558cffa11390 2024-04-22T06:46:07.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66e236640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6540a3d20 0x7fa6540a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa664009890 tx=0x7fa664008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66e236640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa11770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66e236640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa11770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:52362/0 (socket says 172.21.15.79:52362) 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66e236640 1 -- 172.21.15.79:0/2810051046 learned_addr learned my addr 172.21.15.79:0/2810051046 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 -- 172.21.15.79:0/2810051046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 msgr2=0x558cffa11770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa11770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 -- 172.21.15.79:0/2810051046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 msgr2=0x558cffa21280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa21280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 -- 172.21.15.79:0/2810051046 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cff7527d0 con 0x558cffa11d40 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.123+0000 7fa66da35640 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1e950 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa668199730 tx=0x7fa658009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e895f3d377217659 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.124+0000 7fa65cbfe640 1 -- 172.21.15.79:0/2810051046 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658015020 con 0x558cffa11d40 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.124+0000 7fa65cbfe640 1 -- 172.21.15.79:0/2810051046 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa658011ce0 con 0x558cffa11d40 2024-04-22T06:46:07.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.124+0000 7fa65cbfe640 1 -- 172.21.15.79:0/2810051046 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65800be40 con 0x558cffa11d40 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.124+0000 7fa67316a740 1 -- 172.21.15.79:0/2810051046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 msgr2=0x558cffa1e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1e950 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa668199730 tx=0x7fa658009af0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- 172.21.15.79:0/2810051046 shutdown_connections 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa21280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1e950 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- 172.21.15.79:0/2810051046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa11770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- 172.21.15.79:0/2810051046 >> 172.21.15.79:0/2810051046 conn(0x558cffa0bcb0 msgr2=0x558cffa0c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- 172.21.15.79:0/2810051046 shutdown_connections 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- 172.21.15.79:0/2810051046 wait complete. 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- start start 2024-04-22T06:46:07.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 Processor -- start 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 0x558cffa1bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 0x558cffa1c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa1ee90 0x558cffa1c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558cffa21c50 con 0x558cffa11390 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558cff92f2f0 con 0x558cffa11d40 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558cffa21dd0 con 0x558cffa1ee90 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.125+0000 7fa67316a740 10 client.0 _ll_register_callbacks cb 0x558cff93c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 0x558cffa1bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 0x558cffa1bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49358/0 (socket says 172.21.15.79:49358) 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 -- 172.21.15.79:0/3095097131 learned_addr learned my addr 172.21.15.79:0/3095097131 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66ea37640 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa1ee90 0x558cffa1c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 -- 172.21.15.79:0/3095097131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa1ee90 msgr2=0x558cffa1c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa1ee90 0x558cffa1c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 -- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 msgr2=0x558cffa1c430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:07.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 0x558cffa1c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 -- 172.21.15.79:0/3095097131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cff6883f0 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa66e236640 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 0x558cffa1bef0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa66400df50 tx=0x7fa6640047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4181a237c57c9455 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa651ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa651ffb640 1 -- 172.21.15.79:0/3095097131 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664024020 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa651ffb640 1 -- 172.21.15.79:0/3095097131 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fa66400f040 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa658011ce0 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa651ffb640 1 -- 172.21.15.79:0/3095097131 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa66402f050 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa67316a740 10 client.16128 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.126+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558cff6480f0 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa651ffb640 1 -- 172.21.15.79:0/3095097131 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa658011ce0 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa651ffb640 1 -- 172.21.15.79:0/3095097131 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa664013070 con 0x558cffa11390 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa67316a740 10 client.16128 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa67316a740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:07.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa67316a740 20 client.16128 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa67316a740 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa31510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa67316a740 4 client.16128 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:07.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cff983c00 con 0x558cffa2f070 2024-04-22T06:46:07.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.127+0000 7fa66da35640 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa31510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.128+0000 7fa66da35640 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa31510 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9e8f39131fd4c92 server_cookie=20b0618234e25a68 in_seq=0 out_seq=0 2024-04-22T06:46:07.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.128+0000 7fa651ffb640 10 client.16128 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.131+0000 7fa651ffb640 1 -- 172.21.15.79:0/3095097131 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558cff983c00 con 0x558cffa2f070 2024-04-22T06:46:07.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.131+0000 7fa651ffb640 10 client.16128 handle_command_reply: tid=0 2024-04-22T06:46:07.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.131+0000 7fa67316a740 1 client.16128 shutdown 2024-04-22T06:46:07.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 msgr2=0x558cffa1bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 0x558cffa1bef0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa66400df50 tx=0x7fa6640047d0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 shutdown_connections 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa31510 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 0x558cffa1c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa1ee90 0x558cffa1c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa651ffb640 0 client.16128 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 --2- 172.21.15.79:0/3095097131 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11390 0x558cffa1bef0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 >> 172.21.15.79:0/3095097131 conn(0x558cffa0bcb0 msgr2=0x558cffa0da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 shutdown_connections 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 1 -- 172.21.15.79:0/3095097131 wait complete. 2024-04-22T06:46:07.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.132+0000 7fa67316a740 20 client.16128 trim_cache size 0 max 16384 2024-04-22T06:46:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 Processor -- start 2024-04-22T06:46:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 -- start start 2024-04-22T06:46:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa2ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa0abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681ae390 con 0x558cffa11d40 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681abc60 con 0x558cffa1ee90 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.152+0000 7fa67316a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681a9ab0 con 0x558cffa11390 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49364/0 (socket says 172.21.15.79:49364) 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 -- 172.21.15.79:0/2969899706 learned_addr learned my addr 172.21.15.79:0/2969899706 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66e236640 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa2ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 -- 172.21.15.79:0/2969899706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 msgr2=0x558cffa2ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66ea37640 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa0abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa2ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 -- 172.21.15.79:0/2969899706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 msgr2=0x558cffa0abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa0abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 -- 172.21.15.79:0/2969899706 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cffa21280 con 0x558cffa11d40 2024-04-22T06:46:07.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66ea37640 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa66da35640 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1b050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa65800b0e0 tx=0x7fa658023ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee40e93c96212783 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa653fff640 1 -- 172.21.15.79:0/2969899706 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658028070 con 0x558cffa11d40 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa653fff640 1 -- 172.21.15.79:0/2969899706 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa658009260 con 0x558cffa11d40 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.153+0000 7fa653fff640 1 -- 172.21.15.79:0/2969899706 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658015040 con 0x558cffa11d40 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 -- 172.21.15.79:0/2969899706 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 msgr2=0x558cffa1b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1b050 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa65800b0e0 tx=0x7fa658023ae0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 -- 172.21.15.79:0/2969899706 shutdown_connections 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa1ee90 0x558cffa0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa11d40 0x558cffa1b050 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 --2- 172.21.15.79:0/2969899706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa2ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 -- 172.21.15.79:0/2969899706 >> 172.21.15.79:0/2969899706 conn(0x558cffa0bcb0 msgr2=0x558cffa0a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 -- 172.21.15.79:0/2969899706 shutdown_connections 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.154+0000 7fa67316a740 1 -- 172.21.15.79:0/2969899706 wait complete. 2024-04-22T06:46:07.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 -- start start 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 Processor -- start 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 0x558cffa22660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 0x558cffa22ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681aa120 con 0x558cffa1ee90 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681ab590 con 0x558cffa11d40 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6681aa7f0 con 0x558cffa11390 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa67316a740 10 client.0 _ll_register_callbacks cb 0x7fa6640095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 0x558cffa22ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 0x558cffa22ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49366/0 (socket says 172.21.15.79:49366) 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 -- 172.21.15.79:0/4235371382 learned_addr learned my addr 172.21.15.79:0/4235371382 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66e236640 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 -- 172.21.15.79:0/4235371382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 msgr2=0x558cffa272f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa272f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 -- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 msgr2=0x558cffa22660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 0x558cffa22660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 -- 172.21.15.79:0/4235371382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cffa23290 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa66ea37640 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 0x558cffa22ba0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa6600bb970 tx=0x7fa6600bb370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e76fb025122ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa651ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.155+0000 7fa651ffb640 1 -- 172.21.15.79:0/4235371382 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600335a0 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa658009260 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa651ffb640 1 -- 172.21.15.79:0/4235371382 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fa660032c30 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa651ffb640 1 -- 172.21.15.79:0/4235371382 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6600c3350 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 10 client.16134 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558cffa954f0 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa651ffb640 1 -- 172.21.15.79:0/4235371382 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa658009260 con 0x558cffa1ee90 2024-04-22T06:46:07.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa651ffb640 1 -- 172.21.15.79:0/4235371382 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa66003c030 con 0x558cffa1ee90 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 10 client.16134 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 20 client.16134 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa97d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 4 client.16134 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.156+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558cff97da70 con 0x558cffa2f070 2024-04-22T06:46:07.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.157+0000 7fa66e236640 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa97d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.157+0000 7fa66e236640 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa97d00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=187ade337338f026 server_cookie=76c38fb2e711e4cf in_seq=0 out_seq=0 2024-04-22T06:46:07.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.157+0000 7fa651ffb640 10 client.16134 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.160+0000 7fa651ffb640 1 -- 172.21.15.79:0/4235371382 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558cff97da70 con 0x558cffa2f070 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.160+0000 7fa651ffb640 10 client.16134 handle_command_reply: tid=0 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.160+0000 7fa67316a740 1 client.16134 shutdown 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 msgr2=0x558cffa22ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 0x558cffa22ba0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa6600bb970 tx=0x7fa6600bb370 comp rx=0 tx=0).stop 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 shutdown_connections 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558cffa2f070 0x558cffa97d00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558cffa11d40 0x558cffa22660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa651ffb640 0 client.16134 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558cffa1ee90 0x558cffa22ba0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 --2- 172.21.15.79:0/4235371382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cffa11390 0x558cffa272f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 >> 172.21.15.79:0/4235371382 conn(0x558cffa0bcb0 msgr2=0x558cffa0ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 shutdown_connections 2024-04-22T06:46:07.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 1 -- 172.21.15.79:0/4235371382 wait complete. 2024-04-22T06:46:07.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67316a740 20 client.16134 trim_cache size 0 max 16384 2024-04-22T06:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:46:07.163 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6540a3d20 msgr2=0x7fa6540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67049a640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6540a3d20 0x7fa6540a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa664009890 tx=0x7fa664008040 comp rx=0 tx=0).stop 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 msgr2=0x7fa6681985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67049a640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 0x7fa6681985f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa6600098d0 tx=0x7fa660003ba0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.161+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 shutdown_connections 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6540a3d20 0x7fa6540a61e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa66806af50 0x7fa66819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa668108040 0x7fa6681985f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 --2- 172.21.15.79:0/2775505834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa66806a5b0 0x7fa66819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 >> 172.21.15.79:0/2775505834 conn(0x7fa668077c40 msgr2=0x7fa6680ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 shutdown_connections 2024-04-22T06:46:07.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.162+0000 7fa67049a640 1 -- 172.21.15.79:0/2775505834 wait complete. 2024-04-22T06:46:07.170 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:46:07.170 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:07.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:07 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1521377923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 -- 172.21.15.79:0/2503022344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 msgr2=0x7fcacc106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 --2- 172.21.15.79:0/2503022344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc106210 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009930 tx=0x7fcac802bc30 comp rx=0 tx=0).stop 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 -- 172.21.15.79:0/2503022344 shutdown_connections 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 --2- 172.21.15.79:0/2503022344 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcacc101b60 0x7fcacc10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 --2- 172.21.15.79:0/2503022344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcacc1011a0 0x7fcacc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 --2- 172.21.15.79:0/2503022344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc106210 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 -- 172.21.15.79:0/2503022344 >> 172.21.15.79:0/2503022344 conn(0x7fcacc077e30 msgr2=0x7fcacc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 -- 172.21.15.79:0/2503022344 shutdown_connections 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.271+0000 7fcad348c640 1 -- 172.21.15.79:0/2503022344 wait complete. 2024-04-22T06:46:07.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 Processor -- start 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 -- start start 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcacc1011a0 0x7fcacc1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcacc101b60 0x7fcacc199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc1110c0 con 0x7fcacc105e30 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc110f40 con 0x7fcacc101b60 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad348c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc111240 con 0x7fcacc1011a0 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc1a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49374/0 (socket says 172.21.15.79:49374) 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 -- 172.21.15.79:0/2551804680 learned_addr learned my addr 172.21.15.79:0/2551804680 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad0a27640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcacc101b60 0x7fcacc199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1228640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcacc1011a0 0x7fcacc1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 -- 172.21.15.79:0/2551804680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcacc1011a0 msgr2=0x7fcacc1995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcacc1011a0 0x7fcacc1995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 -- 172.21.15.79:0/2551804680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcacc101b60 msgr2=0x7fcacc199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcacc101b60 0x7fcacc199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.272+0000 7fcad1a29640 1 -- 172.21.15.79:0/2551804680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcacc1a4b80 con 0x7fcacc105e30 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.273+0000 7fcad1a29640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc1a4640 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcac400eb10 tx=0x7fcac400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.273+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4004b20 con 0x7fcacc105e30 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.273+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcac4004cc0 con 0x7fcacc105e30 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.273+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4010670 con 0x7fcacc105e30 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.273+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcacc1a4d50 con 0x7fcacc105e30 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.273+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcacc1a5220 con 0x7fcacc105e30 2024-04-22T06:46:07.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.274+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcacc102570 con 0x7fcacc105e30 2024-04-22T06:46:07.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.275+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcac4005020 con 0x7fcacc105e30 2024-04-22T06:46:07.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.276+0000 7fcac27fc640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcab80a3d20 0x7fcab80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.276+0000 7fcad1228640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcab80a3d20 0x7fcab80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.276+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fcac40d0d60 con 0x7fcacc105e30 2024-04-22T06:46:07.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.276+0000 7fcad1228640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcab80a3d20 0x7fcab80a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcac8005ab0 tx=0x7fcac8005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.280+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcac409d380 con 0x7fcacc105e30 2024-04-22T06:46:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:07 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1521377923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:07.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:07 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1521377923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:07.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.508+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcacc109330 con 0x7fcacc105e30 2024-04-22T06:46:07.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.509+0000 7fcac27fc640 1 -- 172.21.15.79:0/2551804680 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11884 (secure 0 0 0) 0x7fcac409cc80 con 0x7fcacc105e30 2024-04-22T06:46:07.507 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:46:07.507 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:45:36.000223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24413,"mds_2":24406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":24,"state":"up:active","state_seq":57,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":26,"state":"up:active","state_seq":58,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24406,24413]},"id":1}]} 2024-04-22T06:46:07.507 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 31 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.510+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcab80a3d20 msgr2=0x7fcab80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.510+0000 7fcad348c640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcab80a3d20 0x7fcab80a61e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcac8005ab0 tx=0x7fcac8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.510+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 msgr2=0x7fcacc1a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.510+0000 7fcad348c640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc1a4640 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcac400eb10 tx=0x7fcac400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 shutdown_connections 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcab80a3d20 0x7fcab80a61e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcacc101b60 0x7fcacc199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcacc105e30 0x7fcacc1a4640 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 --2- 172.21.15.79:0/2551804680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcacc1011a0 0x7fcacc1995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 >> 172.21.15.79:0/2551804680 conn(0x7fcacc077e30 msgr2=0x7fcacc0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 shutdown_connections 2024-04-22T06:46:07.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.511+0000 7fcad348c640 1 -- 172.21.15.79:0/2551804680 wait complete. 2024-04-22T06:46:07.520 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 -- 172.21.15.79:0/1281532514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250106160 msgr2=0x7f525010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 --2- 172.21.15.79:0/1281532514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250106160 0x7f525010dd30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5248009ca0 tx=0x7f524802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 -- 172.21.15.79:0/1281532514 shutdown_connections 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 --2- 172.21.15.79:0/1281532514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250101a40 0x7f5250105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 --2- 172.21.15.79:0/1281532514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250106160 0x7f525010dd30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 --2- 172.21.15.79:0/1281532514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5250101090 0x7f5250101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 -- 172.21.15.79:0/1281532514 >> 172.21.15.79:0/1281532514 conn(0x7f5250077f50 msgr2=0x7f5250078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 -- 172.21.15.79:0/1281532514 shutdown_connections 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 -- 172.21.15.79:0/1281532514 wait complete. 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.624+0000 7f5257cba640 1 Processor -- start 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 -- start start 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 0x7f52501995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250106160 0x7f5250199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f525019a030 0x7f52501a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f52501110d0 con 0x7f5250101090 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5250110f50 con 0x7f5250106160 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5250111250 con 0x7f525019a030 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 0x7f52501995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 0x7f52501995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49396/0 (socket says 172.21.15.79:49396) 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 -- 172.21.15.79:0/1359722836 learned_addr learned my addr 172.21.15.79:0/1359722836 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255255640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250106160 0x7f5250199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5256257640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f525019a030 0x7f52501a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 -- 172.21.15.79:0/1359722836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f525019a030 msgr2=0x7f52501a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f525019a030 0x7f52501a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 -- 172.21.15.79:0/1359722836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250106160 msgr2=0x7f5250199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250106160 0x7f5250199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 -- 172.21.15.79:0/1359722836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52501a4db0 con 0x7f5250101090 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255255640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250106160 0x7f5250199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5255a56640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 0x7f52501995b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f524c00e9f0 tx=0x7f524c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5246ffd640 1 -- 172.21.15.79:0/1359722836 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c00cd80 con 0x7f5250101090 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5246ffd640 1 -- 172.21.15.79:0/1359722836 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f524c0054d0 con 0x7f5250101090 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52501a50a0 con 0x7f5250101090 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5246ffd640 1 -- 172.21.15.79:0/1359722836 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c010690 con 0x7f5250101090 2024-04-22T06:46:07.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.625+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52501a55f0 con 0x7f5250101090 2024-04-22T06:46:07.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.627+0000 7f5246ffd640 1 -- 172.21.15.79:0/1359722836 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f524c004e90 con 0x7f5250101090 2024-04-22T06:46:07.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.628+0000 7f5246ffd640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f523c0a3da0 0x7f523c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.628+0000 7f525a984740 1 Processor -- start 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.628+0000 7f5255255640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f523c0a3da0 0x7f523c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.628+0000 7f5246ffd640 1 -- 172.21.15.79:0/1359722836 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f524c0d0cd0 con 0x7f5250101090 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.628+0000 7f525a984740 1 -- start start 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f525a984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f2910818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f525a984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f525a984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f29108ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f525a984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f290fe61f0 con 0x55f291081e80 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f525a984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f290f834b0 con 0x55f29108aa00 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f525a984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f290fe21b0 con 0x55f2910814d0 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255a56640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f2910818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5256257640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f29108ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255a56640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f2910818b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:52460/0 (socket says 172.21.15.79:52460) 2024-04-22T06:46:07.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f523c0a3da0 0x7f523c0a6260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5250069be0 tx=0x7f5240005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255a56640 1 -- 172.21.15.79:0/1972386225 learned_addr learned my addr 172.21.15.79:0/1972386225 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 -- 172.21.15.79:0/1972386225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 msgr2=0x55f2910818b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f2910818b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 -- 172.21.15.79:0/1972386225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 msgr2=0x55f29108ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f29108ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 -- 172.21.15.79:0/1972386225 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f290f751b0 con 0x55f291081e80 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f5255255640 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108a430 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5240015000 tx=0x7f5240014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8a49b883b41e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f523bfff640 1 -- 172.21.15.79:0/1972386225 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240004250 con 0x55f291081e80 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f523bfff640 1 -- 172.21.15.79:0/1972386225 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52400043f0 con 0x55f291081e80 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.629+0000 7f523bfff640 1 -- 172.21.15.79:0/1972386225 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524001f960 con 0x55f291081e80 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 -- 172.21.15.79:0/1972386225 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 msgr2=0x55f29108a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108a430 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5240015000 tx=0x7f5240014000 comp rx=0 tx=0).stop 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 -- 172.21.15.79:0/1972386225 shutdown_connections 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f29108ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108a430 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 --2- 172.21.15.79:0/1972386225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f2910818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 -- 172.21.15.79:0/1972386225 >> 172.21.15.79:0/1972386225 conn(0x55f29107bba0 msgr2=0x55f29107bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 -- 172.21.15.79:0/1972386225 shutdown_connections 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.630+0000 7f525a984740 1 -- 172.21.15.79:0/1972386225 wait complete. 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 -- start start 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 Processor -- start 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f291087660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f291087bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f29108d970 con 0x55f291081e80 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f290f9f500 con 0x55f29108aa00 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f29108daf0 con 0x55f2910814d0 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f525a984740 10 client.0 _ll_register_callbacks cb 0x55f290f6db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f291087660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f291087660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49402/0 (socket says 172.21.15.79:49402) 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 -- 172.21.15.79:0/3699065844 learned_addr learned my addr 172.21.15.79:0/3699065844 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255a56640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5256257640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f291087bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 -- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 msgr2=0x55f29108a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 -- 172.21.15.79:0/3699065844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 msgr2=0x55f291087bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f291087bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255255640 1 -- 172.21.15.79:0/3699065844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f290cf83f0 con 0x55f291081e80 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.631+0000 7f5255a56640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f5255255640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f291087660 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5240009590 tx=0x7f5240004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3749f762d52d316 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f52397fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f52397fa640 1 -- 172.21.15.79:0/3699065844 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240005050 con 0x55f291081e80 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f52397fa640 1 -- 172.21.15.79:0/3699065844 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f52400051f0 con 0x55f291081e80 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52400043f0 con 0x55f291081e80 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f52397fa640 1 -- 172.21.15.79:0/3699065844 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f52400053f0 con 0x55f291081e80 2024-04-22T06:46:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 10 client.16152 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f290dc27d0 con 0x55f291081e80 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f52397fa640 1 -- 172.21.15.79:0/3699065844 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52400043f0 con 0x55f291081e80 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f52397fa640 1 -- 172.21.15.79:0/3699065844 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f524001f660 con 0x55f291081e80 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 10 client.16152 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 10 client.16152 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 20 client.16152 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f29109aac0 0x55f29109ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 4 client.16152 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.632+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 --> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f290f85930 con 0x55f29109aac0 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.633+0000 7f5255a56640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f29109aac0 0x55f29109ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.633+0000 7f5255a56640 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f29109aac0 0x55f29109ceb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f1074c01e804c7f3 server_cookie=d955d8f540d9c6e3 in_seq=0 out_seq=0 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.633+0000 7f52397fa640 10 client.16152 ms_handle_connect on v2:172.21.15.193:6834/3775212644 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f52397fa640 1 -- 172.21.15.79:0/3699065844 <== mds.2 v2:172.21.15.193:6834/3775212644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f290f85930 con 0x55f29109aac0 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f52397fa640 10 client.16152 handle_command_reply: tid=0 2024-04-22T06:46:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 client.16152 shutdown 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 msgr2=0x55f291087660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f291087660 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5240009590 tx=0x7f5240004610 comp rx=0 tx=0).stop 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 shutdown_connections 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f291087bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f29109aac0 0x55f29109ceb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f291087660 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 --2- 172.21.15.79:0/3699065844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f52397fa640 0 client.16152 ms_handle_reset on v2:172.21.15.193:6834/3775212644 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 >> 172.21.15.79:0/3699065844 conn(0x55f29107bba0 msgr2=0x55f29107d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 shutdown_connections 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 1 -- 172.21.15.79:0/3699065844 wait complete. 2024-04-22T06:46:07.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.634+0000 7f525a984740 20 client.16152 trim_cache size 0 max 16384 2024-04-22T06:46:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 Processor -- start 2024-04-22T06:46:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 -- start start 2024-04-22T06:46:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f291096830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f291081e80 0x55f291089d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 0x55f29108a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f52501b05a0 con 0x55f29108aa00 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52501ade70 con 0x55f291081e80 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f525a984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f52501abcc0 con 0x55f2910814d0 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 0x55f29108a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 0x55f29108a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49416/0 (socket says 172.21.15.79:49416) 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 -- 172.21.15.79:0/3698150577 learned_addr learned my addr 172.21.15.79:0/3698150577 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5255a56640 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f291096830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5255255640 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f291081e80 0x55f291089d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 -- 172.21.15.79:0/3698150577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 msgr2=0x55f291096830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f291096830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 -- 172.21.15.79:0/3698150577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f291081e80 msgr2=0x55f291089d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f291081e80 0x55f291089d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 -- 172.21.15.79:0/3698150577 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f291085bb0 con 0x55f29108aa00 2024-04-22T06:46:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.648+0000 7f5256257640 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 0x55f29108a2d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5248037040 tx=0x7f5248005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21eb783b4cd2100 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f523b7fe640 1 -- 172.21.15.79:0/3698150577 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248039040 con 0x55f29108aa00 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f523b7fe640 1 -- 172.21.15.79:0/3698150577 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5248032970 con 0x55f29108aa00 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f523b7fe640 1 -- 172.21.15.79:0/3698150577 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248003e30 con 0x55f29108aa00 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 -- 172.21.15.79:0/3698150577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 msgr2=0x55f29108a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 0x55f29108a2d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5248037040 tx=0x7f5248005c50 comp rx=0 tx=0).stop 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 -- 172.21.15.79:0/3698150577 shutdown_connections 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f291081e80 0x55f291089d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f29108aa00 0x55f29108a2d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 --2- 172.21.15.79:0/3698150577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f291096830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 -- 172.21.15.79:0/3698150577 >> 172.21.15.79:0/3698150577 conn(0x55f29107bba0 msgr2=0x55f291086690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 -- 172.21.15.79:0/3698150577 shutdown_connections 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 -- 172.21.15.79:0/3698150577 wait complete. 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.649+0000 7f525a984740 1 -- start start 2024-04-22T06:46:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 Processor -- start 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f2910f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f52501ac330 con 0x55f291081e80 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52501ad7a0 con 0x55f29108aa00 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f52501aca00 con 0x55f2910814d0 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f525a984740 10 client.0 _ll_register_callbacks cb 0x7f5240011df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49432/0 (socket says 172.21.15.79:49432) 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 -- 172.21.15.79:0/3610689952 learned_addr learned my addr 172.21.15.79:0/3610689952 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5256257640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f2910f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255a56640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 -- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 msgr2=0x55f29108dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 -- 172.21.15.79:0/3610689952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 msgr2=0x55f2910f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f2910f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 -- 172.21.15.79:0/3610689952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2910f94b0 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f5255255640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108e4f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f524001fc20 tx=0x7f524001f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d676df1cd28bc1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.650+0000 7f52397fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f52397fa640 1 -- 172.21.15.79:0/3610689952 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524001f730 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f52397fa640 1 -- 172.21.15.79:0/3610689952 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f524003b430 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5248032970 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f52397fa640 1 -- 172.21.15.79:0/3610689952 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5240049050 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 10 client.16158 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f291100850 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f52397fa640 1 -- 172.21.15.79:0/3610689952 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5248032970 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f52397fa640 1 -- 172.21.15.79:0/3610689952 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f5240033450 con 0x55f291081e80 2024-04-22T06:46:07.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 10 client.16158 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 10 client.16158 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 20 client.16158 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f291100f70 0x55f291103390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 4 client.16158 mds_command: new command op to 24406 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 --> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f290ff3c10 con 0x55f291100f70 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.651+0000 7f5255a56640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f291100f70 0x55f291103390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f52397fa640 10 client.16158 ms_handle_connect on v2:172.21.15.193:6834/3775212644 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f5255a56640 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f291100f70 0x55f291103390 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c60df02161eba82e server_cookie=d6d6fb451cbc5dce in_seq=0 out_seq=0 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f52397fa640 1 -- 172.21.15.79:0/3610689952 <== mds.2 v2:172.21.15.193:6834/3775212644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f290ff3c10 con 0x55f291100f70 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f52397fa640 10 client.16158 handle_command_reply: tid=0 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 client.16158 shutdown 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 msgr2=0x55f29108e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108e4f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f524001fc20 tx=0x7f524001f4c0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 shutdown_connections 2024-04-22T06:46:07.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f29108aa00 0x55f2910f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:6834/3775212644,v1:172.21.15.193:6835/3775212644] conn(0x55f291100f70 0x55f291103390 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f291081e80 0x55f29108e4f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 --2- 172.21.15.79:0/3610689952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2910814d0 0x55f29108dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f52397fa640 0 client.16158 ms_handle_reset on v2:172.21.15.193:6834/3775212644 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 >> 172.21.15.79:0/3610689952 conn(0x55f29107bba0 msgr2=0x55f29107a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 shutdown_connections 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 1 -- 172.21.15.79:0/3610689952 wait complete. 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.652+0000 7f525a984740 20 client.16158 trim_cache size 0 max 16384 2024-04-22T06:46:07.657 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:46:07.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f523c0a3da0 msgr2=0x7f523c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f523c0a3da0 0x7f523c0a6260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5250069be0 tx=0x7f5240005f90 comp rx=0 tx=0).stop 2024-04-22T06:46:07.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 msgr2=0x7f52501995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 0x7f52501995b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f524c00e9f0 tx=0x7f524c00edf0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 shutdown_connections 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f523c0a3da0 0x7f523c0a6260 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5250106160 0x7f5250199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f525019a030 0x7f52501a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 --2- 172.21.15.79:0/1359722836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5250101090 0x7f52501995b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 >> 172.21.15.79:0/1359722836 conn(0x7f5250077f50 msgr2=0x7f52500fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 shutdown_connections 2024-04-22T06:46:07.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.653+0000 7f5257cba640 1 -- 172.21.15.79:0/1359722836 wait complete. 2024-04-22T06:46:07.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:46:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- 172.21.15.79:0/806894568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 msgr2=0x7f0830101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- 172.21.15.79:0/806894568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830101470 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f082c009ca0 tx=0x7f082c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- 172.21.15.79:0/806894568 shutdown_connections 2024-04-22T06:46:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- 172.21.15.79:0/806894568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0830106160 0x7f083010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- 172.21.15.79:0/806894568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0830101a40 0x7f0830105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- 172.21.15.79:0/806894568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830101470 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- 172.21.15.79:0/806894568 >> 172.21.15.79:0/806894568 conn(0x7f0830077f50 msgr2=0x7f0830078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- 172.21.15.79:0/806894568 shutdown_connections 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- 172.21.15.79:0/806894568 wait complete. 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 Processor -- start 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- start start 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0830106160 0x7f0830197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f083019bee0 0x7f083019c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f08301110a0 con 0x7f0830101090 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0830110f20 con 0x7f083019bee0 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.770+0000 7f0838255640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0830111220 con 0x7f0830106160 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49456/0 (socket says 172.21.15.79:49456) 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 -- 172.21.15.79:0/3263856257 learned_addr learned my addr 172.21.15.79:0/3263856257 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f08367f2640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f083019bee0 0x7f083019c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f08357f0640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0830106160 0x7f0830197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 -- 172.21.15.79:0/3263856257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0830106160 msgr2=0x7f0830197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0830106160 0x7f0830197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 -- 172.21.15.79:0/3263856257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f083019bee0 msgr2=0x7f083019c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f083019bee0 0x7f083019c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 -- 172.21.15.79:0/3263856257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083019c9d0 con 0x7f0830101090 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0835ff1640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830197620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f082c005e30 tx=0x7f082c0346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0826ffd640 1 -- 172.21.15.79:0/3263856257 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c039040 con 0x7f0830101090 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083019cc60 con 0x7f0830101090 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0826ffd640 1 -- 172.21.15.79:0/3263856257 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082c044070 con 0x7f0830101090 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0826ffd640 1 -- 172.21.15.79:0/3263856257 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c0042a0 con 0x7f0830101090 2024-04-22T06:46:07.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.771+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08301a4420 con 0x7f0830101090 2024-04-22T06:46:07.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f0826ffd640 1 -- 172.21.15.79:0/3263856257 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f082c004440 con 0x7f0830101090 2024-04-22T06:46:07.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 Processor -- start 2024-04-22T06:46:07.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 -- start start 2024-04-22T06:46:07.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f0826ffd640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f081c0a3d20 0x7f081c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9573650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9581040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ebb94d81f0 con 0x55ebb9573c20 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebb94754b0 con 0x55ebb957ec10 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f083ae47740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ebb94d41b0 con 0x55ebb9573270 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f08367f2640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f081c0a3d20 0x7f081c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.774+0000 7f08357f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f0835ff1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9573650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49468/0 (socket says 172.21.15.79:49468) 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 -- 172.21.15.79:0/2629312090 learned_addr learned my addr 172.21.15.79:0/2629312090 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08367f2640 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9581040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f0826ffd640 1 -- 172.21.15.79:0/3263856257 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f082c0f2940 con 0x7f0830101090 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 -- 172.21.15.79:0/2629312090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 msgr2=0x55ebb9573650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9573650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 -- 172.21.15.79:0/2629312090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 msgr2=0x55ebb9581040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9581040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 -- 172.21.15.79:0/2629312090 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb94671b0 con 0x55ebb9573c20 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f08357f0640 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957e640 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f082000bc90 tx=0x7f08200107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854ce2d1604cd7b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f081bfff640 1 -- 172.21.15.79:0/2629312090 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082001c480 con 0x55ebb9573c20 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f081bfff640 1 -- 172.21.15.79:0/2629312090 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082001cad0 con 0x55ebb9573c20 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.775+0000 7f081bfff640 1 -- 172.21.15.79:0/2629312090 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0820011ab0 con 0x55ebb9573c20 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f08367f2640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f081c0a3d20 0x7f081c0a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0828005f60 tx=0x7f0828009ae0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 -- 172.21.15.79:0/2629312090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 msgr2=0x55ebb957e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957e640 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f082000bc90 tx=0x7f08200107a0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 -- 172.21.15.79:0/2629312090 shutdown_connections 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9581040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957e640 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 --2- 172.21.15.79:0/2629312090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9573650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 -- 172.21.15.79:0/2629312090 >> 172.21.15.79:0/2629312090 conn(0x55ebb956dba0 msgr2=0x55ebb956df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 -- 172.21.15.79:0/2629312090 shutdown_connections 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.776+0000 7f083ae47740 1 -- 172.21.15.79:0/2629312090 wait complete. 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 -- start start 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 Processor -- start 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573270 0x55ebb957e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb957bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9581b60 con 0x55ebb9573c20 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9491500 con 0x55ebb9573270 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9581ce0 con 0x55ebb957ec10 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.777+0000 7f083ae47740 10 client.0 _ll_register_callbacks cb 0x55ebb945fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f0835ff1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573270 0x55ebb957e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49476/0 (socket says 172.21.15.79:49476) 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 -- 172.21.15.79:0/3396602669 learned_addr learned my addr 172.21.15.79:0/3396602669 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08367f2640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb957bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 -- 172.21.15.79:0/3396602669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 msgr2=0x55ebb957bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb957bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 -- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573270 msgr2=0x55ebb957e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573270 0x55ebb957e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 -- 172.21.15.79:0/3396602669 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb91ea3f0 con 0x55ebb9573c20 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f0835ff1640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573270 0x55ebb957e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08357f0640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957b830 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f082000bc60 tx=0x7f082000bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9652013f6a5cd12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08197fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08197fa640 1 -- 172.21.15.79:0/3396602669 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082002b060 con 0x55ebb9573c20 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08197fa640 1 -- 172.21.15.79:0/3396602669 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f082001c480 con 0x55ebb9573c20 2024-04-22T06:46:07.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f082001cad0 con 0x55ebb9573c20 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f08197fa640 1 -- 172.21.15.79:0/3396602669 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f082001c680 con 0x55ebb9573c20 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f083ae47740 10 client.16170 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.778+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ebb92b47d0 con 0x55ebb9573c20 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f08197fa640 1 -- 172.21.15.79:0/3396602669 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082001cad0 con 0x55ebb9573c20 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f08197fa640 1 -- 172.21.15.79:0/3396602669 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f082001a510 con 0x55ebb9573c20 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f083ae47740 10 client.16170 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f083ae47740 10 client.16170 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.e 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f083ae47740 20 client.16170 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f083ae47740 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb958eca0 0x55ebb9591090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f083ae47740 4 client.16170 mds_command: new command op to 24413 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 --> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebb9477930 con 0x55ebb958eca0 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f0835ff1640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb958eca0 0x55ebb9591090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f0835ff1640 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb958eca0 0x55ebb9591090 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73d95f0638778a75 server_cookie=8e592e864898a3f0 in_seq=0 out_seq=0 2024-04-22T06:46:07.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.779+0000 7f08197fa640 10 client.16170 ms_handle_connect on v2:172.21.15.190:6834/2543246858 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f08197fa640 1 -- 172.21.15.79:0/3396602669 <== mds.1 v2:172.21.15.190:6834/2543246858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebb9477930 con 0x55ebb958eca0 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f08197fa640 10 client.16170 handle_command_reply: tid=0 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 client.16170 shutdown 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 msgr2=0x55ebb957b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957b830 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f082000bc60 tx=0x7f082000bef0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 shutdown_connections 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb958eca0 0x55ebb9591090 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb957bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb957b830 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f08197fa640 0 client.16170 ms_handle_reset on v2:172.21.15.190:6834/2543246858 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 --2- 172.21.15.79:0/3396602669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573270 0x55ebb957e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 >> 172.21.15.79:0/3396602669 conn(0x55ebb956dba0 msgr2=0x55ebb956f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 shutdown_connections 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 1 -- 172.21.15.79:0/3396602669 wait complete. 2024-04-22T06:46:07.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.780+0000 7f083ae47740 20 client.16170 trim_cache size 0 max 16384 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 Processor -- start 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 -- start start 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 0x55ebb958aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573c20 0x55ebb957a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb9576cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f08301ae210 con 0x55ebb9573270 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f08301abae0 con 0x55ebb9573c20 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f083ae47740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08301a9930 con 0x55ebb957ec10 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 0x55ebb958aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 0x55ebb958aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49492/0 (socket says 172.21.15.79:49492) 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 -- 172.21.15.79:0/1169568698 learned_addr learned my addr 172.21.15.79:0/1169568698 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f08357f0640 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573c20 0x55ebb957a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f08367f2640 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb9576cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 -- 172.21.15.79:0/1169568698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 msgr2=0x55ebb9576cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb9576cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 -- 172.21.15.79:0/1169568698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573c20 msgr2=0x55ebb957a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573c20 0x55ebb957a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 -- 172.21.15.79:0/1169568698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb9581cc0 con 0x55ebb9573270 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.794+0000 7f0835ff1640 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 0x55ebb958aa60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f082c0bb3d0 tx=0x7f082c045e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c53aa04a0302391 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f081b7fe640 1 -- 172.21.15.79:0/1169568698 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c035df0 con 0x55ebb9573270 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f081b7fe640 1 -- 172.21.15.79:0/1169568698 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082c003970 con 0x55ebb9573270 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f081b7fe640 1 -- 172.21.15.79:0/1169568698 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c0bcaa0 con 0x55ebb9573270 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 -- 172.21.15.79:0/1169568698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 msgr2=0x55ebb958aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 0x55ebb958aa60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f082c0bb3d0 tx=0x7f082c045e80 comp rx=0 tx=0).stop 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 -- 172.21.15.79:0/1169568698 shutdown_connections 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb9573c20 0x55ebb957a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb957ec10 0x55ebb9576cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 --2- 172.21.15.79:0/1169568698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573270 0x55ebb958aa60 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 -- 172.21.15.79:0/1169568698 >> 172.21.15.79:0/1169568698 conn(0x55ebb956dba0 msgr2=0x55ebb95765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 -- 172.21.15.79:0/1169568698 shutdown_connections 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 -- 172.21.15.79:0/1169568698 wait complete. 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.795+0000 7f083ae47740 1 -- start start 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 Processor -- start 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9576fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb9575340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9575970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f08301a9fa0 con 0x55ebb9573c20 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f08301ab410 con 0x55ebb957ec10 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08301aa670 con 0x55ebb9573270 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f083ae47740 10 client.0 _ll_register_callbacks cb 0x7f0820005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb9575340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb9575340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49502/0 (socket says 172.21.15.79:49502) 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 -- 172.21.15.79:0/2430249640 learned_addr learned my addr 172.21.15.79:0/2430249640 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f0835ff1640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9576fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08367f2640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9575970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 -- 172.21.15.79:0/2430249640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 msgr2=0x55ebb9576fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9576fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 -- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 msgr2=0x55ebb9575970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9575970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.796+0000 7f08357f0640 1 -- 172.21.15.79:0/2430249640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb9576140 con 0x55ebb9573c20 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08357f0640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb9575340 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f082001b000 tx=0x7f0820000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f497080ffd135b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08197fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08197fa640 1 -- 172.21.15.79:0/2430249640 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082002b040 con 0x55ebb9573c20 2024-04-22T06:46:07.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ebb94e5c10 con 0x55ebb9573c20 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08197fa640 1 -- 172.21.15.79:0/2430249640 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f082001c480 con 0x55ebb9573c20 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08197fa640 1 -- 172.21.15.79:0/2430249640 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f082003d050 con 0x55ebb9573c20 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 10 client.16176 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ebb9587020 con 0x55ebb9573c20 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08197fa640 1 -- 172.21.15.79:0/2430249640 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ebb94e5c10 con 0x55ebb9573c20 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f08197fa640 1 -- 172.21.15.79:0/2430249640 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f08200052b0 con 0x55ebb9573c20 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 10 client.16176 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 10 client.16176 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.e 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 20 client.16176 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb95f7300 0x55ebb95f96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 4 client.16176 mds_command: new command op to 24413 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 --> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebb94dfa80 con 0x55ebb95f7300 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.797+0000 7f0835ff1640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb95f7300 0x55ebb95f96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f08197fa640 10 client.16176 ms_handle_connect on v2:172.21.15.190:6834/2543246858 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f0835ff1640 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb95f7300 0x55ebb95f96f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fffb30250842f40d server_cookie=45125199d70e7051 in_seq=0 out_seq=0 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f08197fa640 1 -- 172.21.15.79:0/2430249640 <== mds.1 v2:172.21.15.190:6834/2543246858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebb94dfa80 con 0x55ebb95f7300 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f08197fa640 10 client.16176 handle_command_reply: tid=0 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 client.16176 shutdown 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 msgr2=0x55ebb9575340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb9575340 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f082001b000 tx=0x7f0820000bc0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 shutdown_connections 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:6834/2543246858,v1:172.21.15.190:6835/2543246858] conn(0x55ebb95f7300 0x55ebb95f96f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebb957ec10 0x55ebb9575970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ebb9573c20 0x55ebb9575340 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 --2- 172.21.15.79:0/2430249640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebb9573270 0x55ebb9576fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f08197fa640 0 client.16176 ms_handle_reset on v2:172.21.15.190:6834/2543246858 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 >> 172.21.15.79:0/2430249640 conn(0x55ebb956dba0 msgr2=0x55ebb957a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 shutdown_connections 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 1 -- 172.21.15.79:0/2430249640 wait complete. 2024-04-22T06:46:07.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.798+0000 7f083ae47740 20 client.16176 trim_cache size 0 max 16384 2024-04-22T06:46:07.804 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f081c0a3d20 msgr2=0x7f081c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f081c0a3d20 0x7f081c0a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0828005f60 tx=0x7f0828009ae0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 msgr2=0x7f0830197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830197620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f082c005e30 tx=0x7f082c0346f0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 shutdown_connections 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f081c0a3d20 0x7f081c0a61e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f083019bee0 0x7f083019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0830106160 0x7f0830197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 --2- 172.21.15.79:0/3263856257 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830101090 0x7f0830197620 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 >> 172.21.15.79:0/3263856257 conn(0x7f0830077f50 msgr2=0x7f08300ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 shutdown_connections 2024-04-22T06:46:07.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.799+0000 7f0838255640 1 -- 172.21.15.79:0/3263856257 wait complete. 2024-04-22T06:46:07.808 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.901+0000 7f4b51ffb640 1 -- 172.21.15.79:0/479351286 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x7f4b64013040 con 0x55f048841a50 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.901+0000 7f4b51ffb640 10 client.16116 handle_command_reply: tid=0 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 client.16116 shutdown 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 msgr2=0x55f0487cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487cd530 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4b640096b0 tx=0x7f4b64018c50 comp rx=0 tx=0).stop 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 shutdown_connections 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f048841a50 0x55f048843e70 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f0487b4300 0x55f0487cd530 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0487a2740 0x55f0487d28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 --2- 172.21.15.79:0/479351286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f0487a1d90 0x55f0487d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b51ffb640 0 client.16116 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 >> 172.21.15.79:0/479351286 conn(0x55f0487a46c0 msgr2=0x55f0487a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 shutdown_connections 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 1 -- 172.21.15.79:0/479351286 wait complete. 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b739be740 20 client.16116 trim_cache size 0 max 16384 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 31, 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 6, 2024-04-22T06:46:07.902 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "551eb3a7": { 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 6, 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 79.3, 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:46:07.903 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:07.904 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.902+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4b540a3da0 msgr2=0x7f4b540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4b540a3da0 0x7f4b540a6260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4b640108b0 tx=0x7f4b64009970 comp rx=0 tx=0).stop 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 msgr2=0x7f4b6c1976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c1976e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4b6002c260 tx=0x7f4b6002c880 comp rx=0 tx=0).stop 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 shutdown_connections 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4b540a3da0 0x7f4b540a6260 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b6c106170 0x7f4b6c197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b6c19bfa0 0x7f4b6c19c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 --2- 172.21.15.79:0/1580771394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c1010a0 0x7f4b6c1976e0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 >> 172.21.15.79:0/1580771394 conn(0x7f4b6c077f50 msgr2=0x7f4b6c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 shutdown_connections 2024-04-22T06:46:07.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.903+0000 7f4b70cf2640 1 -- 172.21.15.79:0/1580771394 wait complete. 2024-04-22T06:46:07.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 -- 172.21.15.79:0/2298090938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff6410c510 msgr2=0x7eff6410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 --2- 172.21.15.79:0/2298090938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff6410c510 0x7eff6410e9b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009ca0 tx=0x7eff5c02c230 comp rx=0 tx=0).stop 2024-04-22T06:46:07.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 -- 172.21.15.79:0/2298090938 shutdown_connections 2024-04-22T06:46:07.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 --2- 172.21.15.79:0/2298090938 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff6410c510 0x7eff6410e9b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 --2- 172.21.15.79:0/2298090938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff64103af0 0x7eff6410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 --2- 172.21.15.79:0/2298090938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff641031d0 0x7eff641035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 -- 172.21.15.79:0/2298090938 >> 172.21.15.79:0/2298090938 conn(0x7eff64077f50 msgr2=0x7eff64078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.913 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 551eb3a7, quiesce took 2.7 seconds. Will release after: 19.35 2024-04-22T06:46:07.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 -- 172.21.15.79:0/2298090938 shutdown_connections 2024-04-22T06:46:07.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 -- 172.21.15.79:0/2298090938 wait complete. 2024-04-22T06:46:07.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 Processor -- start 2024-04-22T06:46:07.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.914+0000 7eff69677640 1 -- start start 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff641031d0 0x7eff641986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 0x7eff64198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff6410c510 0x7eff6419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff641121a0 con 0x7eff64103af0 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff64112020 con 0x7eff6410c510 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff64112320 con 0x7eff641031d0 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 0x7eff64198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 0x7eff64198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49512/0 (socket says 172.21.15.79:49512) 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 -- 172.21.15.79:0/2806538424 learned_addr learned my addr 172.21.15.79:0/2806538424 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff641031d0 0x7eff641986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff637fe640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff6410c510 0x7eff6419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 -- 172.21.15.79:0/2806538424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff641031d0 msgr2=0x7eff641986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff641031d0 0x7eff641986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 -- 172.21.15.79:0/2806538424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff6410c510 msgr2=0x7eff6419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff6410c510 0x7eff6419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 -- 172.21.15.79:0/2806538424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6419d6d0 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff627fc640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 0x7eff64198c20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7eff5400d770 tx=0x7eff5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff4bfff640 1 -- 172.21.15.79:0/2806538424 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54017070 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6419d9c0 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff4bfff640 1 -- 172.21.15.79:0/2806538424 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff540054d0 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff4bfff640 1 -- 172.21.15.79:0/2806538424 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54004c30 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.915+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff641a5250 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.917+0000 7eff4bfff640 1 -- 172.21.15.79:0/2806538424 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7eff54004e10 con 0x7eff64103af0 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 Processor -- start 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 -- start start 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff4bfff640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff4c0a3d20 0x7eff4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98330 0x560371f98710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560371efd1f0 con 0x560371fa1e40 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560371e9a4b0 con 0x560371f98ce0 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff6c25f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560371ef91b0 con 0x560371f98330 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff637fe640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff4c0a3d20 0x7eff4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff4bfff640 1 -- 172.21.15.79:0/2806538424 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7eff540d0190 con 0x7eff64103af0 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff627fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff627fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:48876/0 (socket says 172.21.15.79:48876) 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff627fc640 1 -- 172.21.15.79:0/2646627232 learned_addr learned my addr 172.21.15.79:0/2646627232 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.918+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98330 0x560371f98710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 -- 172.21.15.79:0/2646627232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98330 msgr2=0x560371f98710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98330 0x560371f98710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 -- 172.21.15.79:0/2646627232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 msgr2=0x560371fa1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 -- 172.21.15.79:0/2646627232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560371e8c1b0 con 0x560371fa1e40 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff627fc640 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff4c0a3d20 0x7eff4c0a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff5c004af0 tx=0x7eff5c032340 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff637fe640 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa22c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7eff5c02c840 tx=0x7eff5c047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af66131d6f4c6022 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff497fa640 1 -- 172.21.15.79:0/2646627232 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c039040 con 0x560371fa1e40 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff497fa640 1 -- 172.21.15.79:0/2646627232 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff5c004d40 con 0x560371fa1e40 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.919+0000 7eff497fa640 1 -- 172.21.15.79:0/2646627232 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c005030 con 0x560371fa1e40 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- 172.21.15.79:0/2646627232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 msgr2=0x560371fa22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa22c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7eff5c02c840 tx=0x7eff5c047000 comp rx=0 tx=0).stop 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- 172.21.15.79:0/2646627232 shutdown_connections 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa22c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2646627232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98330 0x560371f98710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- 172.21.15.79:0/2646627232 >> 172.21.15.79:0/2646627232 conn(0x560371f92ba0 msgr2=0x560371f92f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- 172.21.15.79:0/2646627232 shutdown_connections 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- 172.21.15.79:0/2646627232 wait complete. 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- start start 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 Processor -- start 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371fa8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371fa55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560371fa8d60 con 0x560371fa1e40 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560371eb6500 con 0x560371f98330 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560371fa8ee0 con 0x560371f98ce0 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.920+0000 7eff6c25f740 10 client.0 _ll_register_callbacks cb 0x560371e84b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49522/0 (socket says 172.21.15.79:49522) 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 -- 172.21.15.79:0/2411807978 learned_addr learned my addr 172.21.15.79:0/2411807978 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371fa8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff627fc640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371fa55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 -- 172.21.15.79:0/2411807978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 msgr2=0x560371fa55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371fa55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 -- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 msgr2=0x560371fa8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371fa8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 -- 172.21.15.79:0/2411807978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560371c0f3f0 con 0x560371fa1e40 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff637fe640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa5ae0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7eff5c045040 tx=0x7eff5c004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fd17e4406493bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff46ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff46ff5640 1 -- 172.21.15.79:0/2411807978 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c039040 con 0x560371fa1e40 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff46ff5640 1 -- 172.21.15.79:0/2411807978 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7eff5c004b40 con 0x560371fa1e40 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5c004d40 con 0x560371fa1e40 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff46ff5640 1 -- 172.21.15.79:0/2411807978 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7eff5c063050 con 0x560371fa1e40 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff6c25f740 10 client.16188 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560371cd97d0 con 0x560371fa1e40 2024-04-22T06:46:07.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.921+0000 7eff46ff5640 1 -- 172.21.15.79:0/2411807978 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff5c004d40 con 0x560371fa1e40 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff46ff5640 1 -- 172.21.15.79:0/2411807978 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7eff5c03f070 con 0x560371fa1e40 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff6c25f740 10 client.16188 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff6c25f740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff6c25f740 20 client.16188 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560371fb5ec0 0x560371fb82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff6c25f740 4 client.16188 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560371e9c930 con 0x560371fb5ec0 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560371fb5ec0 0x560371fb82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff46ff5640 10 client.16188 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.922+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560371fb5ec0 0x560371fb82b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=309185fa81a09f07 server_cookie=5b0c49969b36eea4 in_seq=0 out_seq=0 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff46ff5640 1 -- 172.21.15.79:0/2411807978 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560371e9c930 con 0x560371fb5ec0 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff46ff5640 10 client.16188 handle_command_reply: tid=0 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 client.16188 shutdown 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 msgr2=0x560371fa5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa5ae0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7eff5c045040 tx=0x7eff5c004700 comp rx=0 tx=0).stop 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 shutdown_connections 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560371fb5ec0 0x560371fb82b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371fa5ae0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371fa55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff46ff5640 0 client.16188 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2411807978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371fa8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 >> 172.21.15.79:0/2411807978 conn(0x560371f92ba0 msgr2=0x560371f948f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 shutdown_connections 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 1 -- 172.21.15.79:0/2411807978 wait complete. 2024-04-22T06:46:07.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.923+0000 7eff6c25f740 20 client.16188 trim_cache size 0 max 16384 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.937+0000 7eff6c25f740 1 Processor -- start 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.937+0000 7eff6c25f740 1 -- start start 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 0x560371fb1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371fa1e40 0x560371fa8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff6c25f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff641ae2e0 con 0x560371f98330 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff6c25f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff641abbb0 con 0x560371f98ce0 2024-04-22T06:46:07.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff6c25f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff641a9a00 con 0x560371fa1e40 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 0x560371fb1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 0x560371fb1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49534/0 (socket says 172.21.15.79:49534) 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 -- 172.21.15.79:0/3770143078 learned_addr learned my addr 172.21.15.79:0/3770143078 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff627fc640 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 -- 172.21.15.79:0/3770143078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371fa1e40 msgr2=0x560371fa8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371fa1e40 0x560371fa8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 -- 172.21.15.79:0/3770143078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 msgr2=0x560371fa7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 -- 172.21.15.79:0/3770143078 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560371f8cfe0 con 0x560371f98330 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff627fc640 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff62ffd640 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 0x560371fb1cb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7eff58009e80 tx=0x7eff58005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13de48edfd9334d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff48ff9640 1 -- 172.21.15.79:0/3770143078 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5801f050 con 0x560371f98330 2024-04-22T06:46:07.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff48ff9640 1 -- 172.21.15.79:0/3770143078 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff5801f6a0 con 0x560371f98330 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.938+0000 7eff48ff9640 1 -- 172.21.15.79:0/3770143078 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58010a00 con 0x560371f98330 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 -- 172.21.15.79:0/3770143078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 msgr2=0x560371fb1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 0x560371fb1cb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7eff58009e80 tx=0x7eff58005f90 comp rx=0 tx=0).stop 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 -- 172.21.15.79:0/3770143078 shutdown_connections 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98ce0 0x560371fa7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371fa1e40 0x560371fa8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 --2- 172.21.15.79:0/3770143078 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371f98330 0x560371fb1cb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 -- 172.21.15.79:0/3770143078 >> 172.21.15.79:0/3770143078 conn(0x560371f92ba0 msgr2=0x560371fa45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 -- 172.21.15.79:0/3770143078 shutdown_connections 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 -- 172.21.15.79:0/3770143078 wait complete. 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 -- start start 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.939+0000 7eff6c25f740 1 Processor -- start 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371f997c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371f99d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371f9a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff641aa070 con 0x560371fa1e40 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff641ab4e0 con 0x560371f98330 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff641aa740 con 0x560371f98ce0 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 10 client.0 _ll_register_callbacks cb 0x7eff5c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371f9a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371f9a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49548/0 (socket says 172.21.15.79:49548) 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 -- 172.21.15.79:0/2423911425 learned_addr learned my addr 172.21.15.79:0/2423911425 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff627fc640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371f99d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371f997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 -- 172.21.15.79:0/2423911425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 msgr2=0x560371f99d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371f99d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 -- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 msgr2=0x560371f997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371f997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 -- 172.21.15.79:0/2423911425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560371fad1b0 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff637fe640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371f9a240 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7eff5c034b10 tx=0x7eff5c034f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a7a4063565cace server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff46ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff46ff5640 1 -- 172.21.15.79:0/2423911425 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c06c060 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff46ff5640 1 -- 172.21.15.79:0/2423911425 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7eff5c0046e0 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5801f6a0 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff46ff5640 1 -- 172.21.15.79:0/2423911425 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7eff5c039040 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 10 client.16194 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.940+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560371fae490 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff46ff5640 1 -- 172.21.15.79:0/2423911425 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff5801f6a0 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff46ff5640 1 -- 172.21.15.79:0/2423911425 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7eff5c03f050 con 0x560371fa1e40 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 10 client.16194 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:07.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 10 client.16194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 20 client.16194 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5603720208e0 0x560372022cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 4 client.16194 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560371f0ac10 con 0x5603720208e0 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5603720208e0 0x560372022cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff62ffd640 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5603720208e0 0x560372022cd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ecc977598dcadd server_cookie=b0080c6300acff5f in_seq=0 out_seq=0 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff46ff5640 10 client.16194 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff46ff5640 1 -- 172.21.15.79:0/2423911425 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560371f0ac10 con 0x5603720208e0 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff46ff5640 10 client.16194 handle_command_reply: tid=0 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.941+0000 7eff6c25f740 1 client.16194 shutdown 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 msgr2=0x560371f9a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371f9a240 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7eff5c034b10 tx=0x7eff5c034f30 comp rx=0 tx=0).stop 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 shutdown_connections 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5603720208e0 0x560372022cd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560371fa1e40 0x560371f9a240 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560371f98ce0 0x560371f99d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff46ff5640 0 client.16194 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 --2- 172.21.15.79:0/2423911425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560371f98330 0x560371f997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 >> 172.21.15.79:0/2423911425 conn(0x560371f92ba0 msgr2=0x560371f9fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 shutdown_connections 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 1 -- 172.21.15.79:0/2423911425 wait complete. 2024-04-22T06:46:07.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff6c25f740 20 client.16194 trim_cache size 0 max 16384 2024-04-22T06:46:07.948 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:46:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff4c0a3d20 msgr2=0x7eff4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff4c0a3d20 0x7eff4c0a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff5c004af0 tx=0x7eff5c032340 comp rx=0 tx=0).stop 2024-04-22T06:46:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 msgr2=0x7eff64198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:07.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 0x7eff64198c20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7eff5400d770 tx=0x7eff5400dc30 comp rx=0 tx=0).stop 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 shutdown_connections 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff4c0a3d20 0x7eff4c0a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff6410c510 0x7eff6419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff64103af0 0x7eff64198c20 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 --2- 172.21.15.79:0/2806538424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff641031d0 0x7eff641986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 >> 172.21.15.79:0/2806538424 conn(0x7eff64077f50 msgr2=0x7eff640fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 shutdown_connections 2024-04-22T06:46:07.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:07.942+0000 7eff69677640 1 -- 172.21.15.79:0/2806538424 wait complete. 2024-04-22T06:46:08.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:08 smithi193 ceph-mon[31815]: pgmap v56: 97 pgs: 97 active+clean; 655 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 55 MiB/s wr, 70 op/s 2024-04-22T06:46:08.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:08 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2551804680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:08.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:08 smithi190 ceph-mon[31421]: pgmap v56: 97 pgs: 97 active+clean; 655 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 55 MiB/s wr, 70 op/s 2024-04-22T06:46:08.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:08 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2551804680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:08 smithi079 ceph-mon[27946]: pgmap v56: 97 pgs: 97 active+clean; 655 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 55 MiB/s wr, 70 op/s 2024-04-22T06:46:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:08 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2551804680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:08.952 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:46:08.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:46:08.952 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:46:09.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 -- 172.21.15.79:0/3498443493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd0101ab0 msgr2=0x7fefd0105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 --2- 172.21.15.79:0/3498443493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd0101ab0 0x7fefd0105af0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009900 tx=0x7fefc402bc30 comp rx=0 tx=0).stop 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 -- 172.21.15.79:0/3498443493 shutdown_connections 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 --2- 172.21.15.79:0/3498443493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefd01061d0 0x7fefd010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 --2- 172.21.15.79:0/3498443493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd0101ab0 0x7fefd0105af0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 --2- 172.21.15.79:0/3498443493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefd0101100 0x7fefd01014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 -- 172.21.15.79:0/3498443493 >> 172.21.15.79:0/3498443493 conn(0x7fefd0077fe0 msgr2=0x7fefd00783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 -- 172.21.15.79:0/3498443493 shutdown_connections 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.069+0000 7fefd5d15640 1 -- 172.21.15.79:0/3498443493 wait complete. 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 Processor -- start 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 -- start start 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefd0101100 0x7fefd019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 0x7fefd019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefd0198620 0x7fefd0198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fefd0111110 con 0x7fefd01061d0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fefd0110f90 con 0x7fefd0101100 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefd5d15640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefd0111290 con 0x7fefd0198620 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 0x7fefd019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 0x7fefd019e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49558/0 (socket says 172.21.15.79:49558) 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefceffd640 1 -- 172.21.15.79:0/377421419 learned_addr learned my addr 172.21.15.79:0/377421419 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.070+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefd0101100 0x7fefd019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefceffd640 1 -- 172.21.15.79:0/377421419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefd0198620 msgr2=0x7fefd0198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefceffd640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefd0198620 0x7fefd0198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefceffd640 1 -- 172.21.15.79:0/377421419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefd0101100 msgr2=0x7fefd019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefceffd640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefd0101100 0x7fefd019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefceffd640 1 -- 172.21.15.79:0/377421419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd0199140 con 0x7fefd01061d0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefceffd640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 0x7fefd019e960 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fefc4036040 tx=0x7fefc40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefccff9640 1 -- 172.21.15.79:0/377421419 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc4043020 con 0x7fefd01061d0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd01993d0 con 0x7fefd01061d0 2024-04-22T06:46:09.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefccff9640 1 -- 172.21.15.79:0/377421419 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc402cb70 con 0x7fefd01061d0 2024-04-22T06:46:09.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefccff9640 1 -- 172.21.15.79:0/377421419 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc4003bf0 con 0x7fefd01061d0 2024-04-22T06:46:09.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.071+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd01a5620 con 0x7fefd01061d0 2024-04-22T06:46:09.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.073+0000 7fefccff9640 1 -- 172.21.15.79:0/377421419 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fefc4046020 con 0x7fefd01061d0 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.073+0000 7fefd89e5740 1 Processor -- start 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.073+0000 7fefd89e5740 1 -- start start 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa283cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa28c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa28f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefd89e5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5627fa1e7f00 con 0x5627fa284290 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefd89e5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5627fa185590 con 0x5627fa28ce10 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefd89e5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627fa1e3ec0 con 0x5627fa2838e0 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa28c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefccff9640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fefb80a3d20 0x7fefb80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa28c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49572/0 (socket says 172.21.15.79:49572) 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 -- 172.21.15.79:0/277124864 learned_addr learned my addr 172.21.15.79:0/277124864 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcffff640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fefb80a3d20 0x7fefb80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefceffd640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa28f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcffff640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa283cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 -- 172.21.15.79:0/277124864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 msgr2=0x5627fa283cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa283cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefccff9640 1 -- 172.21.15.79:0/377421419 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fefc40f29f0 con 0x7fefd01061d0 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 -- 172.21.15.79:0/277124864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 msgr2=0x5627fa28f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa28f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 -- 172.21.15.79:0/277124864 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f9fc27e0 con 0x5627fa284290 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefceffd640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa28f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcffff640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa283cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa28c840 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009c80 tx=0x7fefc00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67214b7bd941b63a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefcffff640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fefb80a3d20 0x7fefb80a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefc80092f0 tx=0x7fefc8009420 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefb5ffb640 1 -- 172.21.15.79:0/277124864 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0012450 con 0x5627fa284290 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefb5ffb640 1 -- 172.21.15.79:0/277124864 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc0012aa0 con 0x5627fa284290 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.074+0000 7fefb5ffb640 1 -- 172.21.15.79:0/277124864 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc000fb20 con 0x5627fa284290 2024-04-22T06:46:09.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 -- 172.21.15.79:0/277124864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 msgr2=0x5627fa28c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa28c840 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009c80 tx=0x7fefc00117b0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 -- 172.21.15.79:0/277124864 shutdown_connections 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa28f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa28c840 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 --2- 172.21.15.79:0/277124864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa283cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 -- 172.21.15.79:0/277124864 >> 172.21.15.79:0/277124864 conn(0x5627fa27dfb0 msgr2=0x5627fa27e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 -- 172.21.15.79:0/277124864 shutdown_connections 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 -- 172.21.15.79:0/277124864 wait complete. 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.075+0000 7fefd89e5740 1 -- start start 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 Processor -- start 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa289db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa28a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa28a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5627fa28fd80 con 0x5627fa2838e0 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5627fa1a1580 con 0x5627fa284290 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627fa28ff00 con 0x5627fa28ce10 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefd89e5740 10 client.0 _ll_register_callbacks cb 0x5627fa1a7650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa289db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa289db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49574/0 (socket says 172.21.15.79:49574) 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 -- 172.21.15.79:0/1035371544 learned_addr learned my addr 172.21.15.79:0/1035371544 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefceffd640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa28a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcffff640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa28a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 -- 172.21.15.79:0/1035371544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 msgr2=0x5627fa28a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa28a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 -- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 msgr2=0x5627fa28a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa28a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.076+0000 7fefcf7fe640 1 -- 172.21.15.79:0/1035371544 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627fa022f70 con 0x5627fa2838e0 2024-04-22T06:46:09.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa289db0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fefc001c040 tx=0x7fefc0009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd514c69d3aed5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefaeffd640 1 -- 172.21.15.79:0/1035371544 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0028020 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefaeffd640 1 -- 172.21.15.79:0/1035371544 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fefc0012660 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefc0012aa0 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefaeffd640 1 -- 172.21.15.79:0/1035371544 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fefc0033050 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 10 client.16206 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5627f9eba170 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefaeffd640 1 -- 172.21.15.79:0/1035371544 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc0012aa0 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefaeffd640 1 -- 172.21.15.79:0/1035371544 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fefc000fc90 con 0x5627fa2838e0 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 10 client.16206 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 20 client.16206 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa29f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 4 client.16206 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.077+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627fa1f5920 con 0x5627fa29d220 2024-04-22T06:46:09.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.078+0000 7fefceffd640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa29f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.078+0000 7fefceffd640 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa29f610 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e052ad0ba676a00f server_cookie=e1cb59cd53bf9600 in_seq=0 out_seq=0 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.078+0000 7fefaeffd640 10 client.16206 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefaeffd640 1 -- 172.21.15.79:0/1035371544 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627fa1f5920 con 0x5627fa29d220 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefaeffd640 10 client.16206 handle_command_reply: tid=0 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 client.16206 shutdown 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 msgr2=0x5627fa289db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa289db0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fefc001c040 tx=0x7fefc0009fa0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 shutdown_connections 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa29f610 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa28a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa28a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1035371544 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa289db0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefaeffd640 0 client.16206 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 >> 172.21.15.79:0/1035371544 conn(0x5627fa27dfb0 msgr2=0x5627fa27fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 shutdown_connections 2024-04-22T06:46:09.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 1 -- 172.21.15.79:0/1035371544 wait complete. 2024-04-22T06:46:09.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.079+0000 7fefd89e5740 20 client.16206 trim_cache size 0 max 16384 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 Processor -- start 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 -- start start 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa298c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa285500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa289690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fefd01b0610 con 0x5627fa2838e0 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fefd01adee0 con 0x5627fa284290 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.096+0000 7fefd89e5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefd01abd50 con 0x5627fa28ce10 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa298c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa298c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49582/0 (socket says 172.21.15.79:49582) 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 -- 172.21.15.79:0/2419178426 learned_addr learned my addr 172.21.15.79:0/2419178426 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 -- 172.21.15.79:0/2419178426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 msgr2=0x5627fa289690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa289690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 -- 172.21.15.79:0/2419178426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 msgr2=0x5627fa285500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa285500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 -- 172.21.15.79:0/2419178426 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627fa27d790 con 0x5627fa2838e0 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa298c40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fefc000a010 tx=0x7fefc0004a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58dca01e95f69e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefb57fa640 1 -- 172.21.15.79:0/2419178426 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc003c070 con 0x5627fa2838e0 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefb57fa640 1 -- 172.21.15.79:0/2419178426 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc00044e0 con 0x5627fa2838e0 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.097+0000 7fefb57fa640 1 -- 172.21.15.79:0/2419178426 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0028070 con 0x5627fa2838e0 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- 172.21.15.79:0/2419178426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 msgr2=0x5627fa298c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa298c40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fefc000a010 tx=0x7fefc0004a30 comp rx=0 tx=0).stop 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- 172.21.15.79:0/2419178426 shutdown_connections 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa284290 0x5627fa285500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa28ce10 0x5627fa289690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- 172.21.15.79:0/2419178426 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa2838e0 0x5627fa298c40 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- 172.21.15.79:0/2419178426 >> 172.21.15.79:0/2419178426 conn(0x5627fa27dfb0 msgr2=0x5627fa289d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- 172.21.15.79:0/2419178426 shutdown_connections 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- 172.21.15.79:0/2419178426 wait complete. 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- start start 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 Processor -- start 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa295350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa290710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa290c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fefd01ac3a0 con 0x5627fa284290 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fefd01ad810 con 0x5627fa28ce10 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.098+0000 7fefd89e5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefd01aca70 con 0x5627fa2838e0 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefd89e5740 10 client.0 _ll_register_callbacks cb 0x7fefc0009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa290710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa290710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49592/0 (socket says 172.21.15.79:49592) 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 -- 172.21.15.79:0/1531896314 learned_addr learned my addr 172.21.15.79:0/1531896314 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa295350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefcffff640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa290c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 -- 172.21.15.79:0/1531896314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 msgr2=0x5627fa295350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa295350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 -- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 msgr2=0x5627fa290c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa290c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 -- 172.21.15.79:0/1531896314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627fa291290 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa295350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefceffd640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa290710 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fefc40f06c0 tx=0x7fefc404a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837bc86b59590257 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefaeffd640 1 -- 172.21.15.79:0/1531896314 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40d7070 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefaeffd640 1 -- 172.21.15.79:0/1531896314 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fefc40054d0 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefc00044e0 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefaeffd640 1 -- 172.21.15.79:0/1531896314 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fefc40be380 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefd89e5740 10 client.16212 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5627fa303400 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.099+0000 7fefaeffd640 1 -- 172.21.15.79:0/1531896314 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc00044e0 con 0x5627fa284290 2024-04-22T06:46:09.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefaeffd640 1 -- 172.21.15.79:0/1531896314 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fefc40dc060 con 0x5627fa284290 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefd89e5740 10 client.16212 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefd89e5740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefd89e5740 20 client.16212 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa305b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefd89e5740 4 client.16212 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5627fa1ef790 con 0x5627fa29d220 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa305b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefaeffd640 10 client.16212 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.100+0000 7fefcf7fe640 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa305b70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f2195eb8f6f36ef server_cookie=da0f51a55f38ba81 in_seq=0 out_seq=0 2024-04-22T06:46:09.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefaeffd640 1 -- 172.21.15.79:0/1531896314 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5627fa1ef790 con 0x5627fa29d220 2024-04-22T06:46:09.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefaeffd640 10 client.16212 handle_command_reply: tid=0 2024-04-22T06:46:09.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 client.16212 shutdown 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 msgr2=0x5627fa290710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa290710 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fefc40f06c0 tx=0x7fefc404a000 comp rx=0 tx=0).stop 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 shutdown_connections 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5627fa29d220 0x5627fa305b70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefaeffd640 0 client.16212 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5627fa28ce10 0x5627fa290c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5627fa284290 0x5627fa290710 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 --2- 172.21.15.79:0/1531896314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627fa2838e0 0x5627fa295350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 >> 172.21.15.79:0/1531896314 conn(0x5627fa27dfb0 msgr2=0x5627fa27d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 shutdown_connections 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 1 -- 172.21.15.79:0/1531896314 wait complete. 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd89e5740 20 client.16212 trim_cache size 0 max 16384 2024-04-22T06:46:09.105 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:09.106 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:46:09.106 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "ac088f9a-f713-4d12-9446-a026884c70ad", 2024-04-22T06:46:09.106 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:46:09.106 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fefb80a3d20 msgr2=0x7fefb80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fefb80a3d20 0x7fefb80a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefc80092f0 tx=0x7fefc8009420 comp rx=0 tx=0).stop 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 msgr2=0x7fefd019e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 0x7fefd019e960 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fefc4036040 tx=0x7fefc40045e0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 shutdown_connections 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefd0198620 0x7fefd0198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fefd01061d0 0x7fefd019e960 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fefb80a3d20 0x7fefb80a61e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.102+0000 7fefd5d15640 1 --2- 172.21.15.79:0/377421419 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fefd0101100 0x7fefd019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.103+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 >> 172.21.15.79:0/377421419 conn(0x7fefd0077fe0 msgr2=0x7fefd006aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.103+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 shutdown_connections 2024-04-22T06:46:09.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.103+0000 7fefd5d15640 1 -- 172.21.15.79:0/377421419 wait complete. 2024-04-22T06:46:09.112 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:09.112 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:09.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.214+0000 7f300cb0e640 1 -- 172.21.15.79:0/906527932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3008101b40 msgr2=0x7f300810dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.214+0000 7f300cb0e640 1 --2- 172.21.15.79:0/906527932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3008101b40 0x7f300810dc40 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3000009ca0 tx=0x7f300002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 -- 172.21.15.79:0/906527932 shutdown_connections 2024-04-22T06:46:09.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 --2- 172.21.15.79:0/906527932 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3008101b40 0x7f300810dc40 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 --2- 172.21.15.79:0/906527932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30081011a0 0x7f3008101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 --2- 172.21.15.79:0/906527932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3008106f90 0x7f3008107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 -- 172.21.15.79:0/906527932 >> 172.21.15.79:0/906527932 conn(0x7f3008076c50 msgr2=0x7f3008077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 -- 172.21.15.79:0/906527932 shutdown_connections 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 -- 172.21.15.79:0/906527932 wait complete. 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 Processor -- start 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.215+0000 7f300cb0e640 1 -- start start 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 0x7f3008199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3008101b40 0x7f30081999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3008106f90 0x7f30081a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f30081110f0 con 0x7f30081011a0 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3008110f70 con 0x7f3008101b40 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3008111270 con 0x7f3008106f90 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 0x7f3008199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 0x7f3008199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49618/0 (socket says 172.21.15.79:49618) 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 -- 172.21.15.79:0/3969451547 learned_addr learned my addr 172.21.15.79:0/3969451547 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f3006d9d640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3008106f90 0x7f30081a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 -- 172.21.15.79:0/3969451547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3008106f90 msgr2=0x7f30081a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3008106f90 0x7f30081a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f2ffffff640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3008101b40 0x7f30081999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 -- 172.21.15.79:0/3969451547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3008101b40 msgr2=0x7f30081999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3008101b40 0x7f30081999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 -- 172.21.15.79:0/3969451547 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30081a4a30 con 0x7f30081011a0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f3006d9d640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3008106f90 0x7f30081a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300659c640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 0x7f3008199470 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c920 tx=0x7f2ff800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f2fff7fe640 1 -- 172.21.15.79:0/3969451547 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8007c70 con 0x7f30081011a0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30081a4c60 con 0x7f30081011a0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f2fff7fe640 1 -- 172.21.15.79:0/3969451547 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff8007e10 con 0x7f30081011a0 2024-04-22T06:46:09.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f2fff7fe640 1 -- 172.21.15.79:0/3969451547 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8018820 con 0x7f30081011a0 2024-04-22T06:46:09.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.216+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30081a51d0 con 0x7f30081011a0 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f2fff7fe640 1 -- 172.21.15.79:0/3969451547 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ff800f450 con 0x7f30081011a0 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 Processor -- start 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 -- start start 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f9328960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9331540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9331a80 0x5599f9333e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5599f928d1e0 con 0x5599f9328ea0 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5599f922a390 con 0x5599f9328580 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300f7db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5599f92891a0 con 0x5599f9331a80 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9331540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9331540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49634/0 (socket says 172.21.15.79:49634) 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f300659c640 1 -- 172.21.15.79:0/516328746 learned_addr learned my addr 172.21.15.79:0/516328746 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f3006d9d640 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f9328960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f2ffffff640 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9331a80 0x5599f9333e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.219+0000 7f2fff7fe640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2ff40a3d20 0x7f2ff40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f3006d9d640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2ff40a3d20 0x7f2ff40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f300659c640 1 -- 172.21.15.79:0/516328746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9331a80 msgr2=0x5599f9333e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f300659c640 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9331a80 0x5599f9333e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f300659c640 1 -- 172.21.15.79:0/516328746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 msgr2=0x5599f9328960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f300659c640 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f9328960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f2fff7fe640 1 -- 172.21.15.79:0/3969451547 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2ff80d0a20 con 0x7f30081011a0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f300659c640 1 -- 172.21.15.79:0/516328746 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599f90697d0 con 0x5599f9328ea0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f300659c640 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9331540 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80cdfd0 tx=0x7f2ff800fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35a777fc351edd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f3006d9d640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2ff40a3d20 0x7f2ff40a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f300002c630 tx=0x7f300002c0f0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f2ffcff9640 1 -- 172.21.15.79:0/516328746 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff809c420 con 0x5599f9328ea0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f2ffcff9640 1 -- 172.21.15.79:0/516328746 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff809cfe0 con 0x5599f9328ea0 2024-04-22T06:46:09.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.220+0000 7f2ffcff9640 1 -- 172.21.15.79:0/516328746 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff809c240 con 0x5599f9328ea0 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- 172.21.15.79:0/516328746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 msgr2=0x5599f9331540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9331540 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80cdfd0 tx=0x7f2ff800fe60 comp rx=0 tx=0).stop 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- 172.21.15.79:0/516328746 shutdown_connections 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9331a80 0x5599f9333e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9331540 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- 172.21.15.79:0/516328746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f9328960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- 172.21.15.79:0/516328746 >> 172.21.15.79:0/516328746 conn(0x5599f9322cb0 msgr2=0x5599f9323090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- 172.21.15.79:0/516328746 shutdown_connections 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- 172.21.15.79:0/516328746 wait complete. 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- start start 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 Processor -- start 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f932e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328ea0 0x5599f932ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 0x5599f932f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5599f9334810 con 0x5599f9331a80 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5599f92462f0 con 0x5599f9328580 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5599f9334990 con 0x5599f9328ea0 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.221+0000 7f300f7db740 10 client.0 _ll_register_callbacks cb 0x5599f9253580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 0x5599f932f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 0x5599f932f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49638/0 (socket says 172.21.15.79:49638) 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 -- 172.21.15.79:0/361268877 learned_addr learned my addr 172.21.15.79:0/361268877 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f2ffffff640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328ea0 0x5599f932ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f300659c640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f932e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 -- 172.21.15.79:0/361268877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328ea0 msgr2=0x5599f932ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328ea0 0x5599f932ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 -- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 msgr2=0x5599f932e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f932e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 -- 172.21.15.79:0/361268877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599f8f9f3f0 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f300659c640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f932e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f3006d9d640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 0x5599f932f460 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3000003c20 tx=0x7f30000095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f7d3450cc11f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f2fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f2fedffb640 1 -- 172.21.15.79:0/361268877 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000039030 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f2fedffb640 1 -- 172.21.15.79:0/361268877 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f3000004540 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff809cfe0 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f2fedffb640 1 -- 172.21.15.79:0/361268877 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f300005b050 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f300f7db740 10 client.16224 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599f8f5f0f0 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.222+0000 7f2fedffb640 1 -- 172.21.15.79:0/361268877 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff809cfe0 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f2fedffb640 1 -- 172.21.15.79:0/361268877 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f3000034cc0 con 0x5599f9331a80 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300f7db740 10 client.16224 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300f7db740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300f7db740 20 client.16224 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:09.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300f7db740 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f9341a10 0x5599f9343e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300f7db740 4 client.16224 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599f929ac00 con 0x5599f9341a10 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.223+0000 7f300659c640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f9341a10 0x5599f9343e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f2fedffb640 10 client.16224 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300659c640 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f9341a10 0x5599f9343e90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d65d2deaa571b1a8 server_cookie=1780ad23a3cbd22b in_seq=0 out_seq=0 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f2fedffb640 1 -- 172.21.15.79:0/361268877 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599f929ac00 con 0x5599f9341a10 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f2fedffb640 10 client.16224 handle_command_reply: tid=0 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 client.16224 shutdown 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 msgr2=0x5599f932f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 0x5599f932f460 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3000003c20 tx=0x7f30000095c0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 shutdown_connections 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f9341a10 0x5599f9343e90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9331a80 0x5599f932f460 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328ea0 0x5599f932ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f2fedffb640 0 client.16224 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 --2- 172.21.15.79:0/361268877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9328580 0x5599f932e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 >> 172.21.15.79:0/361268877 conn(0x5599f9322cb0 msgr2=0x5599f9324a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.224+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 shutdown_connections 2024-04-22T06:46:09.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.225+0000 7f300f7db740 1 -- 172.21.15.79:0/361268877 wait complete. 2024-04-22T06:46:09.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.225+0000 7f300f7db740 20 client.16224 trim_cache size 0 max 16384 2024-04-22T06:46:09.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 Processor -- start 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 -- start start 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 0x5599f9331110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f933d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f932db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f30081b0520 con 0x5599f9328ea0 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f30081addf0 con 0x5599f9331a80 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f300f7db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f30081abc40 con 0x5599f9328580 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f933d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f933d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49654/0 (socket says 172.21.15.79:49654) 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.242+0000 7f2ffffff640 1 -- 172.21.15.79:0/3214251067 learned_addr learned my addr 172.21.15.79:0/3214251067 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f3006d9d640 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f932db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2ffffff640 1 -- 172.21.15.79:0/3214251067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 msgr2=0x5599f9331110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2ffffff640 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 0x5599f9331110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2ffffff640 1 -- 172.21.15.79:0/3214251067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 msgr2=0x5599f932db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2ffffff640 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f932db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2ffffff640 1 -- 172.21.15.79:0/3214251067 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599f9323ee0 con 0x5599f9328ea0 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2ffffff640 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f933d700 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000bb70 tx=0x7f2ff0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708ac84f8ca0512a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2feffff640 1 -- 172.21.15.79:0/3214251067 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0011e60 con 0x5599f9328ea0 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2feffff640 1 -- 172.21.15.79:0/3214251067 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff00054d0 con 0x5599f9328ea0 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f2feffff640 1 -- 172.21.15.79:0/3214251067 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0004cb0 con 0x5599f9328ea0 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.243+0000 7f300f7db740 1 -- 172.21.15.79:0/3214251067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 msgr2=0x5599f933d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f933d700 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000bb70 tx=0x7f2ff0011810 comp rx=0 tx=0).stop 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- 172.21.15.79:0/3214251067 shutdown_connections 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f932db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f933d700 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- 172.21.15.79:0/3214251067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 0x5599f9331110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- 172.21.15.79:0/3214251067 >> 172.21.15.79:0/3214251067 conn(0x5599f9322cb0 msgr2=0x5599f9324100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- 172.21.15.79:0/3214251067 shutdown_connections 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- 172.21.15.79:0/3214251067 wait complete. 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- start start 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 Processor -- start 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 0x5599f9335880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9335dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f9336300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f30081ac2b0 con 0x5599f9328ea0 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f30081ad720 con 0x5599f9331a80 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f30081ac980 con 0x5599f9328580 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f300f7db740 10 client.0 _ll_register_callbacks cb 0x7f3000037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9335dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9335dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49656/0 (socket says 172.21.15.79:49656) 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 -- 172.21.15.79:0/1160094532 learned_addr learned my addr 172.21.15.79:0/1160094532 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f3006d9d640 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f9336300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 -- 172.21.15.79:0/1160094532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 msgr2=0x5599f9335880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 0x5599f9335880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 -- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 msgr2=0x5599f9336300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f9336300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.244+0000 7f2ffffff640 1 -- 172.21.15.79:0/1160094532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599f93a71d0 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2ffffff640 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9335dc0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b7f0 tx=0x7f2ff0005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147e430688cc0a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2fedffb640 1 -- 172.21.15.79:0/1160094532 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0028060 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2fedffb640 1 -- 172.21.15.79:0/1160094532 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2ff0004ac0 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff00054d0 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2fedffb640 1 -- 172.21.15.79:0/1160094532 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ff00046c0 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 10 client.16230 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599f93a8480 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2fedffb640 1 -- 172.21.15.79:0/1160094532 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff00054d0 con 0x5599f9328ea0 2024-04-22T06:46:09.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f2fedffb640 1 -- 172.21.15.79:0/1160094532 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2ff0022030 con 0x5599f9328ea0 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 10 client.16230 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 20 client.16230 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f93a8bd0 0x5599f93aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 4 client.16230 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.245+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5599f9294a70 con 0x5599f93a8bd0 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300659c640 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f93a8bd0 0x5599f93aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300659c640 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f93a8bd0 0x5599f93aaff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9806d723f21403e4 server_cookie=6741057d1e9c3f4a in_seq=0 out_seq=0 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f2fedffb640 10 client.16230 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f2fedffb640 1 -- 172.21.15.79:0/1160094532 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5599f9294a70 con 0x5599f93a8bd0 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f2fedffb640 10 client.16230 handle_command_reply: tid=0 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 client.16230 shutdown 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 msgr2=0x5599f9335dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9335dc0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b7f0 tx=0x7f2ff0005250 comp rx=0 tx=0).stop 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 shutdown_connections 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5599f93a8bd0 0x5599f93aaff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5599f9331a80 0x5599f9336300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5599f9328ea0 0x5599f9335dc0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f2fedffb640 0 client.16230 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 --2- 172.21.15.79:0/1160094532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599f9328580 0x5599f9335880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 >> 172.21.15.79:0/1160094532 conn(0x5599f9322cb0 msgr2=0x5599f9324100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 shutdown_connections 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.246+0000 7f300f7db740 1 -- 172.21.15.79:0/1160094532 wait complete. 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300f7db740 20 client.16230 trim_cache size 0 max 16384 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:46:09.251 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout: "ac088f9a-f713-4d12-9446-a026884c70ad": { 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "ac088f9a-f713-4d12-9446-a026884c70ad", 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:09.252 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2ff40a3d20 msgr2=0x7f2ff40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2ff40a3d20 0x7f2ff40a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f300002c630 tx=0x7f300002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 msgr2=0x7f3008199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 0x7f3008199470 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c920 tx=0x7f2ff800cde0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 shutdown_connections 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2ff40a3d20 0x7f2ff40a61e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3008101b40 0x7f30081999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3008106f90 0x7f30081a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 --2- 172.21.15.79:0/3969451547 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30081011a0 0x7f3008199470 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 >> 172.21.15.79:0/3969451547 conn(0x7f3008076c50 msgr2=0x7f30080ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 shutdown_connections 2024-04-22T06:46:09.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:09.247+0000 7f300cb0e640 1 -- 172.21.15.79:0/3969451547 wait complete. 2024-04-22T06:46:09.256 INFO:tasks.cephfs.filesystem:scrub status for tag:ac088f9a-f713-4d12-9446-a026884c70ad - {'path': '/', 'tag': 'ac088f9a-f713-4d12-9446-a026884c70ad', 'options': 'recursive,force'} 2024-04-22T06:46:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:10 smithi193 ceph-mon[31815]: pgmap v57: 97 pgs: 97 active+clean; 655 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 55 MiB/s wr, 66 op/s 2024-04-22T06:46:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:10 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:46:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:10 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:10 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:46:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:10 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:10.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:10 smithi190 ceph-mon[31421]: pgmap v57: 97 pgs: 97 active+clean; 655 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 55 MiB/s wr, 66 op/s 2024-04-22T06:46:10.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:10 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:46:10.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:10 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:10.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:10 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:46:10.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:10 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:10 smithi079 ceph-mon[27946]: pgmap v57: 97 pgs: 97 active+clean; 655 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 55 MiB/s wr, 66 op/s 2024-04-22T06:46:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:10 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:46:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:10 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:10 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:46:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:10 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:12.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:12 smithi193 ceph-mon[31815]: pgmap v58: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 139 MiB/s wr, 178 op/s 2024-04-22T06:46:12.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:12 smithi190 ceph-mon[31421]: pgmap v58: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 139 MiB/s wr, 178 op/s 2024-04-22T06:46:12.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:12 smithi079 ceph-mon[27946]: pgmap v58: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 139 MiB/s wr, 178 op/s 2024-04-22T06:46:14.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:14 smithi193 ceph-mon[31815]: pgmap v59: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 139 MiB/s wr, 160 op/s 2024-04-22T06:46:14.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:14 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:46:14.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:14 smithi190 ceph-mon[31421]: pgmap v59: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 139 MiB/s wr, 160 op/s 2024-04-22T06:46:14.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:14 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:46:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:14 smithi079 ceph-mon[27946]: pgmap v59: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 139 MiB/s wr, 160 op/s 2024-04-22T06:46:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:14 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:46:16.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:16 smithi193 ceph-mon[31815]: pgmap v60: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 139 MiB/s wr, 160 op/s 2024-04-22T06:46:16.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:16 smithi190 ceph-mon[31421]: pgmap v60: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 139 MiB/s wr, 160 op/s 2024-04-22T06:46:16.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:16 smithi079 ceph-mon[27946]: pgmap v60: 97 pgs: 97 active+clean; 1.6 GiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 139 MiB/s wr, 160 op/s 2024-04-22T06:46:18.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:18 smithi193 ceph-mon[31815]: pgmap v61: 97 pgs: 97 active+clean; 1.8 GiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 157 MiB/s wr, 168 op/s 2024-04-22T06:46:18.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:18 smithi190 ceph-mon[31421]: pgmap v61: 97 pgs: 97 active+clean; 1.8 GiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 157 MiB/s wr, 168 op/s 2024-04-22T06:46:18.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:18 smithi079 ceph-mon[27946]: pgmap v61: 97 pgs: 97 active+clean; 1.8 GiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 157 MiB/s wr, 168 op/s 2024-04-22T06:46:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/4127204148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b88106170 msgr2=0x7f2b8810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/4127204148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b88106170 0x7f2b8810dd40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009ca0 tx=0x7f2b8002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/4127204148 shutdown_connections 2024-04-22T06:46:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/4127204148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b88106170 0x7f2b8810dd40 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/4127204148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b88101aa0 0x7f2b88105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/4127204148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b881010f0 0x7f2b881014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/4127204148 >> 172.21.15.79:0/4127204148 conn(0x7f2b88077fd0 msgr2=0x7f2b880783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/4127204148 shutdown_connections 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.436+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/4127204148 wait complete. 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 Processor -- start 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 -- start start 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b881010f0 0x7f2b881986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b88106170 0x7f2b88198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 0x7f2b8819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881110c0 con 0x7f2b8819cf60 2024-04-22T06:46:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88110f40 con 0x7f2b88106170 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8e8a5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88111240 con 0x7f2b881010f0 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8ce42640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 0x7f2b8819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8ce42640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 0x7f2b8819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54932/0 (socket says 172.21.15.79:54932) 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b8ce42640 1 -- 172.21.15.79:0/1605032793 learned_addr learned my addr 172.21.15.79:0/1605032793 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.437+0000 7f2b877fe640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b88106170 0x7f2b88198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b87fff640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b881010f0 0x7f2b881986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8ce42640 1 -- 172.21.15.79:0/1605032793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b881010f0 msgr2=0x7f2b881986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8ce42640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b881010f0 0x7f2b881986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8ce42640 1 -- 172.21.15.79:0/1605032793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b88106170 msgr2=0x7f2b88198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8ce42640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b88106170 0x7f2b88198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8ce42640 1 -- 172.21.15.79:0/1605032793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8819da50 con 0x7f2b8819cf60 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b87fff640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b881010f0 0x7f2b881986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8ce42640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 0x7f2b8819d410 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2b80005e30 tx=0x7f2b80032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b857fa640 1 -- 172.21.15.79:0/1605032793 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80041020 con 0x7f2b8819cf60 2024-04-22T06:46:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8819dce0 con 0x7f2b8819cf60 2024-04-22T06:46:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b857fa640 1 -- 172.21.15.79:0/1605032793 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b80032970 con 0x7f2b8819cf60 2024-04-22T06:46:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b857fa640 1 -- 172.21.15.79:0/1605032793 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80039040 con 0x7f2b8819cf60 2024-04-22T06:46:19.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.438+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b881a54a0 con 0x7f2b8819cf60 2024-04-22T06:46:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.440+0000 7f2b857fa640 1 -- 172.21.15.79:0/1605032793 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2b80044020 con 0x7f2b8819cf60 2024-04-22T06:46:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 Processor -- start 2024-04-22T06:46:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 -- start start 2024-04-22T06:46:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0bd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0bdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc0228d0 con 0x55ecdc0c70e0 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ecdbfbfe80 con 0x55ecdc0bdb60 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b91574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc01e890 con 0x55ecdc0be8d0 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b857fa640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b700a3dc0 0x7f2b700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54942/0 (socket says 172.21.15.79:54942) 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 -- 172.21.15.79:0/1602157632 learned_addr learned my addr 172.21.15.79:0/1602157632 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b8ce42640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b700a3dc0 0x7f2b700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b87fff640 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0bdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0bd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b857fa640 1 -- 172.21.15.79:0/1605032793 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2b800f2540 con 0x7f2b8819cf60 2024-04-22T06:46:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 -- 172.21.15.79:0/1602157632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 msgr2=0x55ecdc0bd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0bd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 -- 172.21.15.79:0/1602157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 msgr2=0x55ecdc0bdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0bdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.441+0000 7f2b877fe640 1 -- 172.21.15.79:0/1602157632 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdbfe1320 con 0x55ecdc0c70e0 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b877fe640 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c94d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005c50 tx=0x7f2b7c00a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e260a895e638f60e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b8ce42640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b700a3dc0 0x7f2b700a6280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2b800be270 tx=0x7f2b800be340 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b6e7fc640 1 -- 172.21.15.79:0/1602157632 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004020 con 0x55ecdc0c70e0 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b6e7fc640 1 -- 172.21.15.79:0/1602157632 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c0054d0 con 0x55ecdc0c70e0 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b6e7fc640 1 -- 172.21.15.79:0/1602157632 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c004af0 con 0x55ecdc0c70e0 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 -- 172.21.15.79:0/1602157632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 msgr2=0x55ecdc0c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c94d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005c50 tx=0x7f2b7c00a600 comp rx=0 tx=0).stop 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 -- 172.21.15.79:0/1602157632 shutdown_connections 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0bdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c94d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 --2- 172.21.15.79:0/1602157632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0bd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.442+0000 7f2b91574740 1 -- 172.21.15.79:0/1602157632 >> 172.21.15.79:0/1602157632 conn(0x55ecdc084c90 msgr2=0x55ecdc0b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 -- 172.21.15.79:0/1602157632 shutdown_connections 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 -- 172.21.15.79:0/1602157632 wait complete. 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 -- start start 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 Processor -- start 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.451 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc0c9e10 con 0x55ecdc0bdb60 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ecdbfaaeb0 con 0x55ecdc0be8d0 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ecdc0c9f90 con 0x55ecdc0c70e0 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b91574740 10 client.0 _ll_register_callbacks cb 0x55ecdbfd7b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.443+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54954/0 (socket says 172.21.15.79:54954) 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 -- 172.21.15.79:0/2530891634 learned_addr learned my addr 172.21.15.79:0/2530891634 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 -- 172.21.15.79:0/2530891634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0c70e0 msgr2=0x55ecdc0c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 -- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0be8d0 msgr2=0x55ecdc0c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 -- 172.21.15.79:0/2530891634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdbd35210 con 0x55ecdc0bdb60 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:19.452 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c6ac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0045b0 tx=0x7f2b7c00afb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4029c0aae9aa3932 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b677fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b677fe640 1 -- 172.21.15.79:0/2530891634 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0046c0 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b677fe640 1 -- 172.21.15.79:0/2530891634 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2b7c00f640 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7c0054d0 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b677fe640 1 -- 172.21.15.79:0/2530891634 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2b7c011020 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b91574740 10 client.16242 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ecdbe01620 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.444+0000 7f2b677fe640 1 -- 172.21.15.79:0/2530891634 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c0054d0 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b677fe640 1 -- 172.21.15.79:0/2530891634 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2b7c01e470 con 0x55ecdc0bdb60 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b91574740 10 client.16242 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b91574740 10 client.16242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b91574740 20 client.16242 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b91574740 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc0d7100 0x55ecdc0d94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b91574740 4 client.16242 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecdc036880 con 0x55ecdc0d7100 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.445+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc0d7100 0x55ecdc0d94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.446+0000 7f2b677fe640 10 client.16242 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.446+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc0d7100 0x55ecdc0d94f0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fee90860eea9e6a6 server_cookie=41f8c195dcf8bffb in_seq=0 out_seq=0 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.446+0000 7f2b677fe640 1 -- 172.21.15.79:0/2530891634 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecdc036880 con 0x55ecdc0d7100 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.446+0000 7f2b677fe640 10 client.16242 handle_command_reply: tid=0 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.446+0000 7f2b91574740 1 client.16242 shutdown 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 msgr2=0x55ecdc0c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c6ac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0045b0 tx=0x7f2b7c00afb0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 shutdown_connections 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc0d7100 0x55ecdc0d94f0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 --2- 172.21.15.79:0/2530891634 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c6ac0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b677fe640 0 client.16242 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 >> 172.21.15.79:0/2530891634 conn(0x55ecdc084c90 msgr2=0x55ecdc0ba5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 shutdown_connections 2024-04-22T06:46:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 1 -- 172.21.15.79:0/2530891634 wait complete. 2024-04-22T06:46:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.447+0000 7f2b91574740 20 client.16242 trim_cache size 0 max 16384 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 Processor -- start 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 -- start start 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881b0500 con 0x55ecdc0c70e0 2024-04-22T06:46:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881addd0 con 0x55ecdc0bdb60 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.462+0000 7f2b91574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881abc40 con 0x55ecdc0be8d0 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54960/0 (socket says 172.21.15.79:54960) 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 -- 172.21.15.79:0/606595730 learned_addr learned my addr 172.21.15.79:0/606595730 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b87fff640 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 -- 172.21.15.79:0/606595730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 msgr2=0x55ecdc0cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0cf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 -- 172.21.15.79:0/606595730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 msgr2=0x55ecdc0d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0d2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 -- 172.21.15.79:0/606595730 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdc0bce30 con 0x55ecdc0c70e0 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b877fe640 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c2e50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005b10 tx=0x7f2b7c005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1098cb86feb6e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b6dffb640 1 -- 172.21.15.79:0/606595730 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c005490 con 0x55ecdc0c70e0 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b6dffb640 1 -- 172.21.15.79:0/606595730 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c015040 con 0x55ecdc0c70e0 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.463+0000 7f2b6dffb640 1 -- 172.21.15.79:0/606595730 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c01e470 con 0x55ecdc0c70e0 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 -- 172.21.15.79:0/606595730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 msgr2=0x55ecdc0c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c2e50 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005b10 tx=0x7f2b7c005700 comp rx=0 tx=0).stop 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 -- 172.21.15.79:0/606595730 shutdown_connections 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c2e50 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.468 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 --2- 172.21.15.79:0/606595730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 -- 172.21.15.79:0/606595730 >> 172.21.15.79:0/606595730 conn(0x55ecdc084c90 msgr2=0x55ecdc0c2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 -- 172.21.15.79:0/606595730 shutdown_connections 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 -- 172.21.15.79:0/606595730 wait complete. 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 -- start start 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.464+0000 7f2b91574740 1 Processor -- start 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881ac290 con 0x55ecdc0c70e0 2024-04-22T06:46:19.469 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881ad700 con 0x55ecdc0bdb60 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2b881ac960 con 0x55ecdc0be8d0 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 10 client.0 _ll_register_callbacks cb 0x7f2b780095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54968/0 (socket says 172.21.15.79:54968) 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 -- 172.21.15.79:0/2594876403 learned_addr learned my addr 172.21.15.79:0/2594876403 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 -- 172.21.15.79:0/2594876403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 msgr2=0x55ecdc0c0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 -- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 msgr2=0x55ecdc0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 -- 172.21.15.79:0/2594876403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdc0c0bd0 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b877fe640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c0690 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005390 tx=0x7f2b7c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ad893ab9664bd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b677fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b677fe640 1 -- 172.21.15.79:0/2594876403 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c01e470 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b677fe640 1 -- 172.21.15.79:0/2594876403 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2b7c01e610 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7c015040 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b677fe640 1 -- 172.21.15.79:0/2594876403 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2b7c01eaa0 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 10 client.16248 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.465+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ecdc141190 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b677fe640 1 -- 172.21.15.79:0/2594876403 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c015040 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b677fe640 1 -- 172.21.15.79:0/2594876403 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2b7c04b070 con 0x55ecdc0c70e0 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b91574740 10 client.16248 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b91574740 10 client.16248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b91574740 20 client.16248 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b91574740 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc1417d0 0x55ecdc143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b91574740 4 client.16248 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ecdbfc23d0 con 0x55ecdc1417d0 2024-04-22T06:46:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc1417d0 0x55ecdc143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b87fff640 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc1417d0 0x55ecdc143bf0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b76d79850ad6f8c9 server_cookie=33139f56ce364bd7 in_seq=0 out_seq=0 2024-04-22T06:46:19.472 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.466+0000 7f2b677fe640 10 client.16248 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b677fe640 1 -- 172.21.15.79:0/2594876403 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ecdbfc23d0 con 0x55ecdc1417d0 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b677fe640 10 client.16248 handle_command_reply: tid=0 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 client.16248 shutdown 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 msgr2=0x55ecdc0c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c0690 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005390 tx=0x7f2b7c0058a0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 shutdown_connections 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ecdc1417d0 0x55ecdc143bf0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ecdc0c70e0 0x55ecdc0c0690 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecdc0be8d0 0x55ecdc0c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 --2- 172.21.15.79:0/2594876403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ecdc0bdb60 0x55ecdc0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b677fe640 0 client.16248 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:19.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 >> 172.21.15.79:0/2594876403 conn(0x55ecdc084c90 msgr2=0x55ecdc0b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 shutdown_connections 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 1 -- 172.21.15.79:0/2594876403 wait complete. 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.481+0000 7f2b91574740 20 client.16248 trim_cache size 0 max 16384 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:46:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b700a3dc0 msgr2=0x7f2b700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b700a3dc0 0x7f2b700a6280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2b800be270 tx=0x7f2b800be340 comp rx=0 tx=0).stop 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 msgr2=0x7f2b8819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 0x7f2b8819d410 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2b80005e30 tx=0x7f2b80032420 comp rx=0 tx=0).stop 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 shutdown_connections 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b700a3dc0 0x7f2b700a6280 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b88106170 0x7f2b88198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8819cf60 0x7f2b8819d410 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 --2- 172.21.15.79:0/1605032793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b881010f0 0x7f2b881986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 >> 172.21.15.79:0/1605032793 conn(0x7f2b88077fd0 msgr2=0x7f2b880698a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 shutdown_connections 2024-04-22T06:46:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:19.482+0000 7f2b8e8a5640 1 -- 172.21.15.79:0/1605032793 wait complete. 2024-04-22T06:46:20.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:20 smithi193 ceph-mon[31815]: pgmap v62: 97 pgs: 97 active+clean; 1.8 GiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 103 MiB/s wr, 119 op/s 2024-04-22T06:46:20.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:20 smithi190 ceph-mon[31421]: pgmap v62: 97 pgs: 97 active+clean; 1.8 GiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 103 MiB/s wr, 119 op/s 2024-04-22T06:46:20.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:20 smithi079 ceph-mon[27946]: pgmap v62: 97 pgs: 97 active+clean; 1.8 GiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 103 MiB/s wr, 119 op/s 2024-04-22T06:46:22.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:22 smithi193 ceph-mon[31815]: pgmap v63: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 123 MiB/s wr, 130 op/s 2024-04-22T06:46:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:22 smithi190 ceph-mon[31421]: pgmap v63: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 123 MiB/s wr, 130 op/s 2024-04-22T06:46:22.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:22 smithi079 ceph-mon[27946]: pgmap v63: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 123 MiB/s wr, 130 op/s 2024-04-22T06:46:24.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:24 smithi193 ceph-mon[31815]: pgmap v64: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 38 MiB/s wr, 18 op/s 2024-04-22T06:46:24.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:24 smithi190 ceph-mon[31421]: pgmap v64: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 38 MiB/s wr, 18 op/s 2024-04-22T06:46:24.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:24 smithi079 ceph-mon[27946]: pgmap v64: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 38 MiB/s wr, 18 op/s 2024-04-22T06:46:26.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:26 smithi079 ceph-mon[27946]: pgmap v65: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 38 MiB/s wr, 18 op/s 2024-04-22T06:46:26.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:26 smithi193 ceph-mon[31815]: pgmap v65: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 38 MiB/s wr, 18 op/s 2024-04-22T06:46:26.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:26 smithi190 ceph-mon[31421]: pgmap v65: 97 pgs: 97 active+clean; 2.1 GiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 38 MiB/s wr, 18 op/s 2024-04-22T06:46:27.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- 172.21.15.79:0/57165067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c108010 msgr2=0x7fad3c1083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- 172.21.15.79:0/57165067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c108010 0x7fad3c1083f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fad34009ca0 tx=0x7fad3402c180 comp rx=0 tx=0).stop 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- 172.21.15.79:0/57165067 shutdown_connections 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- 172.21.15.79:0/57165067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c101f50 0x7fad3c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- 172.21.15.79:0/57165067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad3c1028f0 0x7fad3c10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- 172.21.15.79:0/57165067 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c108010 0x7fad3c1083f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- 172.21.15.79:0/57165067 >> 172.21.15.79:0/57165067 conn(0x7fad3c0793e0 msgr2=0x7fad3c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- 172.21.15.79:0/57165067 shutdown_connections 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- 172.21.15.79:0/57165067 wait complete. 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 Processor -- start 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- start start 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad3c101f50 0x7fad3c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 0x7fad3c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c108010 0x7fad3c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.376+0000 7fad4397e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c112220 con 0x7fad3c1028f0 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad4397e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c1120a0 con 0x7fad3c108010 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad4397e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c1123a0 con 0x7fad3c101f50 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 0x7fad3c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 0x7fad3c198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50954/0 (socket says 172.21.15.79:50954) 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 -- 172.21.15.79:0/2246195907 learned_addr learned my addr 172.21.15.79:0/2246195907 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:27.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad4171a640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad3c101f50 0x7fad3c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad41f1b640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c108010 0x7fad3c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 -- 172.21.15.79:0/2246195907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad3c101f50 msgr2=0x7fad3c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad3c101f50 0x7fad3c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 -- 172.21.15.79:0/2246195907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c108010 msgr2=0x7fad3c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c108010 0x7fad3c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 -- 172.21.15.79:0/2246195907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3c19d750 con 0x7fad3c1028f0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad41f1b640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c108010 0x7fad3c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad40f19640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 0x7fad3c198ca0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fad3000b4a0 tx=0x7fad3000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30014070 con 0x7fad3c1028f0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad30010ac0 con 0x7fad3c1028f0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3c19da40 con 0x7fad3c1028f0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30017c30 con 0x7fad3c1028f0 2024-04-22T06:46:27.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.377+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad3c1a52d0 con 0x7fad3c1028f0 2024-04-22T06:46:27.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.378+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad3c068de0 con 0x7fad3c1028f0 2024-04-22T06:46:27.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.379+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad30017070 con 0x7fad3c1028f0 2024-04-22T06:46:27.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.380+0000 7fad3a7fc640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fad240a3d20 0x7fad240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.379 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.380+0000 7fad4171a640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fad240a3d20 0x7fad240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.380+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fad300cff00 con 0x7fad3c1028f0 2024-04-22T06:46:27.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.381+0000 7fad4171a640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fad240a3d20 0x7fad240a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fad34006010 tx=0x7fad34003990 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.385+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad3009c630 con 0x7fad3c1028f0 2024-04-22T06:46:27.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.753+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad3c199ab0 con 0x7fad3c1028f0 2024-04-22T06:46:27.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.754+0000 7fad3a7fc640 1 -- 172.21.15.79:0/2246195907 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11884 (secure 0 0 0) 0x7fad3009bf30 con 0x7fad3c1028f0 2024-04-22T06:46:27.753 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:46:27.753 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:45:36.000223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24413,"mds_2":24406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":24,"state":"up:active","state_seq":57,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":26,"state":"up:active","state_seq":58,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24406,24413]},"id":1}]} 2024-04-22T06:46:27.753 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 31 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fad240a3d20 msgr2=0x7fad240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fad240a3d20 0x7fad240a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fad34006010 tx=0x7fad34003990 comp rx=0 tx=0).stop 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 msgr2=0x7fad3c198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 0x7fad3c198ca0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fad3000b4a0 tx=0x7fad3000b960 comp rx=0 tx=0).stop 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 shutdown_connections 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fad240a3d20 0x7fad240a61e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c108010 0x7fad3c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad3c1028f0 0x7fad3c198ca0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 --2- 172.21.15.79:0/2246195907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad3c101f50 0x7fad3c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 >> 172.21.15.79:0/2246195907 conn(0x7fad3c0793e0 msgr2=0x7fad3c100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 shutdown_connections 2024-04-22T06:46:27.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.757+0000 7fad4397e640 1 -- 172.21.15.79:0/2246195907 wait complete. 2024-04-22T06:46:27.769 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id 551eb3a7 --release --await' 2024-04-22T06:46:27.769 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id 551eb3a7 --release --await 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.877+0000 7fee59f8f640 1 -- 172.21.15.79:0/3988197243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee541031d0 msgr2=0x7fee541035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3988197243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee541031d0 0x7fee541035b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fee44009900 tx=0x7fee4402bd60 comp rx=0 tx=0).stop 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- 172.21.15.79:0/3988197243 shutdown_connections 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3988197243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee54103af0 0x7fee5410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3988197243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee5410c510 0x7fee5410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3988197243 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee541031d0 0x7fee541035b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- 172.21.15.79:0/3988197243 >> 172.21.15.79:0/3988197243 conn(0x7fee54077f50 msgr2=0x7fee54078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- 172.21.15.79:0/3988197243 shutdown_connections 2024-04-22T06:46:27.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- 172.21.15.79:0/3988197243 wait complete. 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 Processor -- start 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- start start 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee541031d0 0x7fee5419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee54103af0 0x7fee5419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 0x7fee541985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fee54112200 con 0x7fee5410c510 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee54112080 con 0x7fee54103af0 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.878+0000 7fee59f8f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fee54112380 con 0x7fee541031d0 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 0x7fee541985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 0x7fee541985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50974/0 (socket says 172.21.15.79:50974) 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 -- 172.21.15.79:0/3714106585 learned_addr learned my addr 172.21.15.79:0/3714106585 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee52ffd640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee54103af0 0x7fee5419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee537fe640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee541031d0 0x7fee5419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 -- 172.21.15.79:0/3714106585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee541031d0 msgr2=0x7fee5419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee541031d0 0x7fee5419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 -- 172.21.15.79:0/3714106585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee54103af0 msgr2=0x7fee5419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee54103af0 0x7fee5419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 -- 172.21.15.79:0/3714106585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee54198d60 con 0x7fee5410c510 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee53fff640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 0x7fee541985e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00b970 tx=0x7fee4c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee50ff9640 1 -- 172.21.15.79:0/3714106585 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c019070 con 0x7fee5410c510 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee54199050 con 0x7fee5410c510 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee50ff9640 1 -- 172.21.15.79:0/3714106585 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c014440 con 0x7fee5410c510 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee50ff9640 1 -- 172.21.15.79:0/3714106585 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0075d0 con 0x7fee5410c510 2024-04-22T06:46:27.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.879+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee541a52f0 con 0x7fee5410c510 2024-04-22T06:46:27.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.881+0000 7fee50ff9640 1 -- 172.21.15.79:0/3714106585 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee4c01c070 con 0x7fee5410c510 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 Processor -- start 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 -- start start 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230286f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 0x5598230312d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee50ff9640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fee3c0a3d20 0x7fee3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823031810 0x559823033c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x559822f8d500 con 0x559823028c30 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559822f336a0 con 0x559823028310 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559822f894c0 con 0x559823031810 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee537fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 0x5598230312d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee537fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 0x5598230312d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50978/0 (socket says 172.21.15.79:50978) 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee537fe640 1 -- 172.21.15.79:0/200933807 learned_addr learned my addr 172.21.15.79:0/200933807 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:27.887 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee53fff640 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823031810 0x559823033c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee52ffd640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fee3c0a3d20 0x7fee3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee52ffd640 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee50ff9640 1 -- 172.21.15.79:0/3714106585 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fee4c0cfcf0 con 0x7fee5410c510 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee53fff640 1 -- 172.21.15.79:0/200933807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 msgr2=0x5598230286f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee53fff640 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230286f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee53fff640 1 -- 172.21.15.79:0/200933807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 msgr2=0x5598230312d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee53fff640 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 0x5598230312d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee53fff640 1 -- 172.21.15.79:0/200933807 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559822f525e0 con 0x559823031810 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee537fe640 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 0x5598230312d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.882+0000 7fee52ffd640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fee3c0a3d20 0x7fee3c0a61e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fee541008d0 tx=0x7fee4800bad0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee53fff640 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823031810 0x559823033c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0cd4d0 tx=0x7fee4c014d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52b2f6417bc25dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee39ffb640 1 -- 172.21.15.79:0/200933807 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c099420 con 0x559823031810 2024-04-22T06:46:27.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee39ffb640 1 -- 172.21.15.79:0/200933807 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c09bdd0 con 0x559823031810 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee39ffb640 1 -- 172.21.15.79:0/200933807 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c00f160 con 0x559823031810 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 -- 172.21.15.79:0/200933807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823031810 msgr2=0x559823033c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823031810 0x559823033c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0cd4d0 tx=0x7fee4c014d00 comp rx=0 tx=0).stop 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 -- 172.21.15.79:0/200933807 shutdown_connections 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823031810 0x559823033c00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823028c30 0x5598230312d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/200933807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230286f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 -- 172.21.15.79:0/200933807 >> 172.21.15.79:0/200933807 conn(0x559823022a40 msgr2=0x559823022e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 -- 172.21.15.79:0/200933807 shutdown_connections 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.883+0000 7fee5cb7f740 1 -- 172.21.15.79:0/200933807 wait complete. 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 -- start start 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 Processor -- start 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230312c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x55982302e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x55982302ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x559823034600 con 0x559823031810 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559822f46a60 con 0x559823028310 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559823034780 con 0x559823028c30 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee5cb7f740 10 client.0 _ll_register_callbacks cb 0x559822c9f110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x55982302ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x55982302ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50988/0 (socket says 172.21.15.79:50988) 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee52ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x55982302e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 -- 172.21.15.79:0/954508584 learned_addr learned my addr 172.21.15.79:0/954508584 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee537fe640 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 -- 172.21.15.79:0/954508584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 msgr2=0x55982302e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x55982302e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 -- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 msgr2=0x5598230312c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.884+0000 7fee53fff640 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230312c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee53fff640 1 -- 172.21.15.79:0/954508584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559822c9f920 con 0x559823031810 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee53fff640 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x55982302ed90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fee4c09bad0 tx=0x7fee4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9554bc8387a3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee32ffd640 1 -- 172.21.15.79:0/954508584 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0b4070 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee32ffd640 1 -- 172.21.15.79:0/954508584 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fee4c09a0d0 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee4c09bdd0 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee32ffd640 1 -- 172.21.15.79:0/954508584 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fee4c0a00d0 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 10 client.16266 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559822d6bd30 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee32ffd640 1 -- 172.21.15.79:0/954508584 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c09bdd0 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee32ffd640 1 -- 172.21.15.79:0/954508584 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fee4c098440 con 0x559823031810 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 10 client.16266 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 10 client.16266 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 20 client.16266 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x559823041830 0x559823043c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 4 client.16266 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.885+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559822f60a10 con 0x559823041830 2024-04-22T06:46:27.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.886+0000 7fee537fe640 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x559823041830 0x559823043c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.886+0000 7fee32ffd640 10 client.16266 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.886+0000 7fee537fe640 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x559823041830 0x559823043c20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85837626ddaf94 server_cookie=9313420d5e5fdf9 in_seq=0 out_seq=0 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee32ffd640 1 -- 172.21.15.79:0/954508584 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559822f60a10 con 0x559823041830 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee32ffd640 10 client.16266 handle_command_reply: tid=0 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 client.16266 shutdown 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 msgr2=0x55982302ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x55982302ed90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fee4c09bad0 tx=0x7fee4c004a40 comp rx=0 tx=0).stop 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 shutdown_connections 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x559823041830 0x559823043c20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x55982302ed90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x55982302e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee32ffd640 0 client.16266 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/954508584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x5598230312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 >> 172.21.15.79:0/954508584 conn(0x559823022a40 msgr2=0x559823024790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.892 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 shutdown_connections 2024-04-22T06:46:27.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 1 -- 172.21.15.79:0/954508584 wait complete. 2024-04-22T06:46:27.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.887+0000 7fee5cb7f740 20 client.16266 trim_cache size 0 max 16384 2024-04-22T06:46:27.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 Processor -- start 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 -- start start 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823030d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x5598230312c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fee541b05e0 con 0x559823031810 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee541adeb0 con 0x559823028310 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fee541abd00 con 0x559823028c30 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50990/0 (socket says 172.21.15.79:50990) 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 -- 172.21.15.79:0/1230015134 learned_addr learned my addr 172.21.15.79:0/1230015134 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee52ffd640 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x5598230312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee537fe640 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823030d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 -- 172.21.15.79:0/1230015134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 msgr2=0x5598230312c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x5598230312c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 -- 172.21.15.79:0/1230015134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 msgr2=0x559823030d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823030d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee53fff640 1 -- 172.21.15.79:0/1230015134 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55982302cba0 con 0x559823031810 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.904+0000 7fee537fe640 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823030d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee53fff640 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230491f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0ba2a0 tx=0x7fee4c014d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cf93129043b65e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee397fa640 1 -- 172.21.15.79:0/1230015134 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0b4070 con 0x559823031810 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee397fa640 1 -- 172.21.15.79:0/1230015134 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c0b5260 con 0x559823031810 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee397fa640 1 -- 172.21.15.79:0/1230015134 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0145f0 con 0x559823031810 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1230015134 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 msgr2=0x5598230491f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230491f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0ba2a0 tx=0x7fee4c014d00 comp rx=0 tx=0).stop 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1230015134 shutdown_connections 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230491f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x5598230312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1230015134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823030d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1230015134 >> 172.21.15.79:0/1230015134 conn(0x559823022a40 msgr2=0x559823021b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1230015134 shutdown_connections 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.905+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1230015134 wait complete. 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 -- start start 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 Processor -- start 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823034740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x559823034c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230351c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fee541ac370 con 0x559823031810 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee541ad7e0 con 0x559823028310 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fee541aca40 con 0x559823028c30 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee5cb7f740 10 client.0 _ll_register_callbacks cb 0x7fee44005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50996/0 (socket says 172.21.15.79:50996) 2024-04-22T06:46:27.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 -- 172.21.15.79:0/1783246949 learned_addr learned my addr 172.21.15.79:0/1783246949 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee537fe640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823034740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee52ffd640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x559823034c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 -- 172.21.15.79:0/1783246949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 msgr2=0x559823034c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x559823034c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 -- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 msgr2=0x559823034740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823034740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee53fff640 1 -- 172.21.15.79:0/1783246949 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598230a7c30 con 0x559823031810 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.906+0000 7fee537fe640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823034740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee53fff640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230351c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0a0290 tx=0x7fee4c09bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ded9cb1be1a404 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee32ffd640 1 -- 172.21.15.79:0/1783246949 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0b4030 con 0x559823031810 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee4c0b5260 con 0x559823031810 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee32ffd640 1 -- 172.21.15.79:0/1783246949 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7fee4c098440 con 0x559823031810 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee32ffd640 1 -- 172.21.15.79:0/1783246949 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fee4c0850c0 con 0x559823031810 2024-04-22T06:46:27.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 10 client.16272 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5598230a7c30 con 0x559823031810 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee32ffd640 1 -- 172.21.15.79:0/1783246949 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee4c0b5260 con 0x559823031810 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee32ffd640 1 -- 172.21.15.79:0/1783246949 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fee4c0bc0b0 con 0x559823031810 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 10 client.16272 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 10 client.16272 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 20 client.16272 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5598230a91d0 0x5598230ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 4 client.16272 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "551eb3a7", "release": true, "await": true}] 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "551eb3a7", "release": true, "await": true}) -- 0x559822f9af20 con 0x5598230a91d0 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.907+0000 7fee537fe640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5598230a91d0 0x5598230ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.908+0000 7fee32ffd640 10 client.16272 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:27.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.908+0000 7fee537fe640 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5598230a91d0 0x5598230ab5c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaea3875429bf3eb server_cookie=c44473cdc30efbb6 in_seq=0 out_seq=0 2024-04-22T06:46:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.911+0000 7fee32ffd640 1 -- 172.21.15.79:0/1783246949 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x559822f9af20 con 0x5598230a91d0 2024-04-22T06:46:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.912+0000 7fee32ffd640 10 client.16272 handle_command_reply: tid=0 2024-04-22T06:46:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.912+0000 7fee5cb7f740 1 client.16272 shutdown 2024-04-22T06:46:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.912+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 msgr2=0x5598230351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.912+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230351c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0a0290 tx=0x7fee4c09bf10 comp rx=0 tx=0).stop 2024-04-22T06:46:27.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 shutdown_connections 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5598230a91d0 0x5598230ab5c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee32ffd640 0 client.16272 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559823031810 0x5598230351c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559823028c30 0x559823034c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 --2- 172.21.15.79:0/1783246949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559823028310 0x559823034740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 >> 172.21.15.79:0/1783246949 conn(0x559823022a40 msgr2=0x559823023bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 shutdown_connections 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 1 -- 172.21.15.79:0/1783246949 wait complete. 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.917+0000 7fee5cb7f740 20 client.16272 trim_cache size 0 max 16384 2024-04-22T06:46:27.916 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 31, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 8, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "551eb3a7": { 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 8, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 79.3, 2024-04-22T06:46:27.917 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:27.918 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.918+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fee3c0a3d20 msgr2=0x7fee3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.918+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fee3c0a3d20 0x7fee3c0a61e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fee541008d0 tx=0x7fee4800bad0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.918+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 msgr2=0x7fee541985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 0x7fee541985e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00b970 tx=0x7fee4c00be30 comp rx=0 tx=0).stop 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 shutdown_connections 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fee3c0a3d20 0x7fee3c0a61e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee54103af0 0x7fee5419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fee5410c510 0x7fee541985e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 --2- 172.21.15.79:0/3714106585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee541031d0 0x7fee5419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 >> 172.21.15.79:0/3714106585 conn(0x7fee54077f50 msgr2=0x7fee540fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.919+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 shutdown_connections 2024-04-22T06:46:27.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:27.920+0000 7fee59f8f640 1 -- 172.21.15.79:0/3714106585 wait complete. 2024-04-22T06:46:27.945 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 551eb3a7, seconds elapsed: 22.7 2024-04-22T06:46:27.945 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.35 2024-04-22T06:46:27.945 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5516f5d700 time 2024-04-22T06:46:27.909785+0000 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f552a5983dd] 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.190 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: clone() 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:27.909+0000 7f5516f5d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5516f5d700 time 2024-04-22T06:46:27.909785+0000 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f552a5983dd] 2024-04-22T06:46:28.191 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: clone() 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: *** Caught signal (Aborted) ** 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: in thread 7f5516f5d700 thread_name:quiesce.agt 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5529320d20] 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: gsignal() 2024-04-22T06:46:28.192 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: abort() 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f552a598437] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 13: clone() 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:27.912+0000 7f5516f5d700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: in thread 7f5516f5d700 thread_name:quiesce.agt 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.193 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5529320d20] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: gsignal() 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: abort() 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f552a598437] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 13: clone() 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.194 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: -1> 2024-04-22T06:46:27.909+0000 7f5516f5d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5516f5d700 time 2024-04-22T06:46:27.909785+0000 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f552a5983dd] 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.195 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: clone() 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 0> 2024-04-22T06:46:27.912+0000 7f5516f5d700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: in thread 7f5516f5d700 thread_name:quiesce.agt 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5529320d20] 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: gsignal() 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: abort() 2024-04-22T06:46:28.196 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f552a598437] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 13: clone() 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: -2> 2024-04-22T06:46:27.909+0000 7f5516f5d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5516f5d700 time 2024-04-22T06:46:27.909785+0000 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.197 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f552a5983dd] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: clone() 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: -1> 2024-04-22T06:46:27.912+0000 7f5516f5d700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: in thread 7f5516f5d700 thread_name:quiesce.agt 2024-04-22T06:46:28.198 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5529320d20] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2: gsignal() 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 3: abort() 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f552a598437] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f552a5985a3] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558589638e9c] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558589638f9f] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585895670d3] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55858956703c] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585897e6d3f] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55858984f560] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f55293161ca] 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 13: clone() 2024-04-22T06:46:28.199 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:28.200 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[62628]: 2024-04-22T06:46:28.200 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:27 smithi193 podman[66433]: 2024-04-22 06:46:27.967860848 +0000 UTC m=+0.020336160 container died 430ad37b311b4b74ef9d0dfeb83c4e68e03f87f133c303d8b1a590477c3718fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:46:28.204 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f887bec3700 time 2024-04-22T06:46:27.909689+0000 2024-04-22T06:46:28.204 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f888f4fe3dd] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: clone() 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:27.910+0000 7f887bec3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f887bec3700 time 2024-04-22T06:46:27.909689+0000 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.205 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f888f4fe3dd] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: clone() 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: *** Caught signal (Aborted) ** 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: in thread 7f887bec3700 thread_name:quiesce.agt 2024-04-22T06:46:28.206 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f888e286d20] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: gsignal() 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: abort() 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f888f4fe437] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 13: clone() 2024-04-22T06:46:28.207 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:27.911+0000 7f887bec3700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:28.208 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: in thread 7f887bec3700 thread_name:quiesce.agt 2024-04-22T06:46:28.208 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.208 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.208 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f888e286d20] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: gsignal() 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: abort() 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f888f4fe437] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 13: clone() 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:28.209 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: -1> 2024-04-22T06:46:27.910+0000 7f887bec3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f887bec3700 time 2024-04-22T06:46:27.909689+0000 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f888f4fe3dd] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.210 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: clone() 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 0> 2024-04-22T06:46:27.911+0000 7f887bec3700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: in thread 7f887bec3700 thread_name:quiesce.agt 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f888e286d20] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: gsignal() 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: abort() 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f888f4fe437] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.211 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 13: clone() 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: -2> 2024-04-22T06:46:27.910+0000 7f887bec3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f887bec3700 time 2024-04-22T06:46:27.909689+0000 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f888f4fe3dd] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.212 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: clone() 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: -1> 2024-04-22T06:46:27.911+0000 7f887bec3700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: in thread 7f887bec3700 thread_name:quiesce.agt 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f888e286d20] 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2: gsignal() 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 3: abort() 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f888f4fe437] 2024-04-22T06:46:28.213 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f888f4fe5a3] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x557a57532e9c] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x557a57532f9f] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x557a574610d3] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x557a5746103c] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557a576e0d3f] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x557a57749560] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f888e27c1ca] 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 13: clone() 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[61721]: 2024-04-22T06:46:28.214 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:27 smithi190 podman[65499]: 2024-04-22 06:46:27.96978071 +0000 UTC m=+0.020580105 container died 5330aa7da9cc102c4e28ac41a1489c0f32d1ba1bd658f30a29a453d85ace5731 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:46:28.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 -- 172.21.15.79:0/240594113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e410e600 msgr2=0x7f43e410e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:28.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 --2- 172.21.15.79:0/240594113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e410e600 0x7f43e410e9e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009cb0 tx=0x7f43e002bfd0 comp rx=0 tx=0).stop 2024-04-22T06:46:28.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 -- 172.21.15.79:0/240594113 shutdown_connections 2024-04-22T06:46:28.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 --2- 172.21.15.79:0/240594113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43e4108f50 0x7f43e41093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:28.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 --2- 172.21.15.79:0/240594113 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43e4109910 0x7f43e407a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:28.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 --2- 172.21.15.79:0/240594113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e410e600 0x7f43e410e9e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:28.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 -- 172.21.15.79:0/240594113 >> 172.21.15.79:0/240594113 conn(0x7f43e406e410 msgr2=0x7f43e40719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:28.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 -- 172.21.15.79:0/240594113 shutdown_connections 2024-04-22T06:46:28.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.466+0000 7f43ec64a640 1 -- 172.21.15.79:0/240594113 wait complete. 2024-04-22T06:46:28.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.468+0000 7f43ec64a640 1 Processor -- start 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.475+0000 7f43ec64a640 1 -- start start 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.476+0000 7f43ec64a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43e4108f50 0x7f43e407a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.476+0000 7f43ec64a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 0x7f43e4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.476+0000 7f43ec64a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43e410e600 0x7f43e4074770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.476+0000 7f43ec64a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43e407e680 con 0x7f43e410e600 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.476+0000 7f43ec64a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43e407e500 con 0x7f43e4108f50 2024-04-22T06:46:28.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.476+0000 7f43ec64a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43e407e800 con 0x7f43e4109910 2024-04-22T06:46:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.482+0000 7f43e9be5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 0x7f43e4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.482+0000 7f43e9be5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 0x7f43e4074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:49894/0 (socket says 172.21.15.79:49894) 2024-04-22T06:46:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.482+0000 7f43e9be5640 1 -- 172.21.15.79:0/3780392831 learned_addr learned my addr 172.21.15.79:0/3780392831 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:28.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.482+0000 7f43ea3e6640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43e4108f50 0x7f43e407a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:28.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.485+0000 7f43e9be5640 1 -- 172.21.15.79:0/3780392831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43e4108f50 msgr2=0x7f43e407a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:28.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.485+0000 7f43e9be5640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43e4108f50 0x7f43e407a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:28.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.485+0000 7f43e9be5640 1 -- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43e410e600 msgr2=0x7f43e4074770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:28.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.485+0000 7f43e9be5640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43e410e600 0x7f43e4074770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:28.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.485+0000 7f43e9be5640 1 -- 172.21.15.79:0/3780392831 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e41bc040 con 0x7f43e4109910 2024-04-22T06:46:28.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.488+0000 7f43e9be5640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 0x7f43e4074110 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00efb0 tx=0x7f43dc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:28.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.491+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc01b070 con 0x7f43e4109910 2024-04-22T06:46:28.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.491+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc00ba20 con 0x7f43e4109910 2024-04-22T06:46:28.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.491+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc016b90 con 0x7f43e4109910 2024-04-22T06:46:28.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.492+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e41bc330 con 0x7f43e4109910 2024-04-22T06:46:28.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.492+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e41bc830 con 0x7f43e4109910 2024-04-22T06:46:28.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:28 smithi079 ceph-mon[27946]: pgmap v66: 97 pgs: 97 active+clean; 2.3 GiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 59 MiB/s wr, 26 op/s 2024-04-22T06:46:28.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:28 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2246195907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:28.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.497+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c8005290 con 0x7f43e4109910 2024-04-22T06:46:28.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.498+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f43dc01e070 con 0x7f43e4109910 2024-04-22T06:46:28.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.499+0000 7f43db7fe640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d00a3ef0 0x7f43d00a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:28.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.500+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f43dc0d0770 con 0x7f43e4109910 2024-04-22T06:46:28.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.500+0000 7f43ea3e6640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d00a3ef0 0x7f43d00a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:28.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.501+0000 7f43ea3e6640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d00a3ef0 0x7f43d00a63b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009780 tx=0x7f43e0000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:28.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.509+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43dc09cd30 con 0x7f43e4109910 2024-04-22T06:46:28.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:28 smithi193 ceph-mon[31815]: pgmap v66: 97 pgs: 97 active+clean; 2.3 GiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 59 MiB/s wr, 26 op/s 2024-04-22T06:46:28.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:28 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2246195907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:28 smithi190 ceph-mon[31421]: pgmap v66: 97 pgs: 97 active+clean; 2.3 GiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 59 MiB/s wr, 26 op/s 2024-04-22T06:46:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:28 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2246195907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:28.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:28 smithi193 podman[66433]: 2024-04-22 06:46:28.688672967 +0000 UTC m=+0.741148279 container remove 430ad37b311b4b74ef9d0dfeb83c4e68e03f87f133c303d8b1a590477c3718fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:46:28.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:28 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:46:28.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:28.999+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c8005850 con 0x7f43e4109910 2024-04-22T06:46:29.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.003+0000 7f43db7fe640 1 -- 172.21.15.79:0/3780392831 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11884 (secure 0 0 0) 0x7f43dc09c630 con 0x7f43e4109910 2024-04-22T06:46:29.004 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:46:29.004 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:45:36.000223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":24413,"mds_2":24406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24406":{"gid":24406,"name":"f","rank":2,"incarnation":24,"state":"up:active","state_seq":57,"addr":"172.21.15.193:6835/3775212644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3775212644},{"type":"v1","addr":"172.21.15.193:6835","nonce":3775212644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24413":{"gid":24413,"name":"e","rank":1,"incarnation":26,"state":"up:active","state_seq":58,"addr":"172.21.15.190:6835/2543246858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":2543246858},{"type":"v1","addr":"172.21.15.190:6835","nonce":2543246858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,24406,24413]},"id":1}]} 2024-04-22T06:46:29.004 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 31 2024-04-22T06:46:29.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.014+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d00a3ef0 msgr2=0x7f43d00a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:29.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.014+0000 7f43ec64a640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d00a3ef0 0x7f43d00a63b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009780 tx=0x7f43e0000bc0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.015+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 msgr2=0x7f43e4074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:29.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.015+0000 7f43ec64a640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 0x7f43e4074110 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00efb0 tx=0x7f43dc00c800 comp rx=0 tx=0).stop 2024-04-22T06:46:29.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.018+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 shutdown_connections 2024-04-22T06:46:29.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.018+0000 7f43ec64a640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43e410e600 0x7f43e4074770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.018+0000 7f43ec64a640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43e4109910 0x7f43e4074110 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.018+0000 7f43ec64a640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d00a3ef0 0x7f43d00a63b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.018+0000 7f43ec64a640 1 --2- 172.21.15.79:0/3780392831 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43e4108f50 0x7f43e407a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.018+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 >> 172.21.15.79:0/3780392831 conn(0x7f43e406e410 msgr2=0x7f43e407b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:29.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.021+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 shutdown_connections 2024-04-22T06:46:29.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.024+0000 7f43ec64a640 1 -- 172.21.15.79:0/3780392831 wait complete. 2024-04-22T06:46:29.070 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 79.35 --await' 2024-04-22T06:46:29.070 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 79.35 --await 2024-04-22T06:46:29.187 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:28 smithi190 podman[65499]: 2024-04-22 06:46:28.795955167 +0000 UTC m=+0.846754578 container remove 5330aa7da9cc102c4e28ac41a1489c0f32d1ba1bd658f30a29a453d85ace5731 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T06:46:29.187 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:28 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:46:29.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.263+0000 7f9158a68640 1 -- 172.21.15.79:0/3247064720 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914402cc30 con 0x7f9154102b40 2024-04-22T06:46:29.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 -- 172.21.15.79:0/3247064720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154102b40 msgr2=0x7f9154106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:29.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 --2- 172.21.15.79:0/3247064720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154102b40 0x7f9154106b80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9144009900 tx=0x7f914402bc80 comp rx=0 tx=0).stop 2024-04-22T06:46:29.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 -- 172.21.15.79:0/3247064720 shutdown_connections 2024-04-22T06:46:29.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 --2- 172.21.15.79:0/3247064720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9154107260 0x7f915410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 --2- 172.21.15.79:0/3247064720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154102b40 0x7f9154106b80 secure :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9144009900 tx=0x7f914402bc80 comp rx=0 tx=0).stop 2024-04-22T06:46:29.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 --2- 172.21.15.79:0/3247064720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9154102190 0x7f9154102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 -- 172.21.15.79:0/3247064720 >> 172.21.15.79:0/3247064720 conn(0x7f9154079310 msgr2=0x7f9154079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:29.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 -- 172.21.15.79:0/3247064720 shutdown_connections 2024-04-22T06:46:29.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.264+0000 7f915bcce640 1 -- 172.21.15.79:0/3247064720 wait complete. 2024-04-22T06:46:29.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.265+0000 7f915bcce640 1 Processor -- start 2024-04-22T06:46:29.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.266+0000 7f915bcce640 1 -- start start 2024-04-22T06:46:29.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.266+0000 7f915bcce640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9154102190 0x7f915419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.266+0000 7f915bcce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9154107260 0x7f915419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.266+0000 7f915bcce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154198970 0x7f9154198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915bcce640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9154112310 con 0x7f9154198970 2024-04-22T06:46:29.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915bcce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9154112190 con 0x7f9154107260 2024-04-22T06:46:29.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915bcce640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9154112490 con 0x7f9154102190 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154198970 0x7f9154198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154198970 0x7f9154198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51024/0 (socket says 172.21.15.79:51024) 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 -- 172.21.15.79:0/832519564 learned_addr learned my addr 172.21.15.79:0/832519564 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 -- 172.21.15.79:0/832519564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9154102190 msgr2=0x7f915419e7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 --2- 172.21.15.79:0/832519564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9154102190 0x7f915419e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 -- 172.21.15.79:0/832519564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9154107260 msgr2=0x7f915419ed00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 --2- 172.21.15.79:0/832519564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9154107260 0x7f915419ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.267+0000 7f915a26b640 1 -- 172.21.15.79:0/832519564 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9154199490 con 0x7f9154198970 2024-04-22T06:46:29.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.268+0000 7f915a26b640 1 --2- 172.21.15.79:0/832519564 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9154198970 0x7f9154198e20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f914c00b990 tx=0x7f914c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:29.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.268+0000 7f914affd640 1 -- 172.21.15.79:0/832519564 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c019070 con 0x7f9154198970 2024-04-22T06:46:29.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.268+0000 7f914affd640 1 -- 172.21.15.79:0/832519564 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914c00cba0 con 0x7f9154198970 2024-04-22T06:46:29.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.268+0000 7f914affd640 1 -- 172.21.15.79:0/832519564 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0075d0 con 0x7f9154198970 2024-04-22T06:46:29.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.270+0000 7f915bcce640 1 -- 172.21.15.79:0/832519564 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9154199780 con 0x7f9154198970 2024-04-22T06:46:29.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.270+0000 7f915bcce640 1 -- 172.21.15.79:0/832519564 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9154074420 con 0x7f9154198970 2024-04-22T06:46:29.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.276+0000 7f914affd640 1 -- 172.21.15.79:0/832519564 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f914c01c070 con 0x7f9154198970 2024-04-22T06:46:29.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.276+0000 7f915e995740 1 Processor -- start 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.276+0000 7f915e995740 1 -- start start 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f915e995740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848059c0 0x55c684805da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f915e995740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848062e0 0x55c68480e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f915e995740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 0x55c6848112b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f915e995740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c6847660b0 con 0x55c68480eec0 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f915e995740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c684717680 con 0x55c6848059c0 2024-04-22T06:46:29.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f915e995740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c684723c60 con 0x55c6848062e0 2024-04-22T06:46:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f9159269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 0x55c6848112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f9159269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 0x55c6848112b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51026/0 (socket says 172.21.15.79:51026) 2024-04-22T06:46:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.277+0000 7f9159269640 1 -- 172.21.15.79:0/4264476658 learned_addr learned my addr 172.21.15.79:0/4264476658 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f914affd640 1 --2- 172.21.15.79:0/832519564 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f91400a8280 0x7f91400aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f9159269640 1 -- 172.21.15.79:0/4264476658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848062e0 msgr2=0x55c68480e980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f9159269640 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848062e0 0x55c68480e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f9159269640 1 -- 172.21.15.79:0/4264476658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848059c0 msgr2=0x55c684805da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f9159269640 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848059c0 0x55c684805da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f9159269640 1 -- 172.21.15.79:0/4264476658 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c684548c90 con 0x55c68480eec0 2024-04-22T06:46:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f914affd640 1 -- 172.21.15.79:0/832519564 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f914c0cfcc0 con 0x7f9154198970 2024-04-22T06:46:29.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f9159269640 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 0x55c6848112b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f915406b670 tx=0x7f91440095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff18611f36c30968 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.278+0000 7f915a26b640 1 --2- 172.21.15.79:0/832519564 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f91400a8280 0x7f91400aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.279+0000 7f913ffff640 1 -- 172.21.15.79:0/4264476658 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144044070 con 0x55c68480eec0 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.279+0000 7f913ffff640 1 -- 172.21.15.79:0/4264476658 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9144005d10 con 0x55c68480eec0 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.279+0000 7f913ffff640 1 -- 172.21.15.79:0/4264476658 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91440055e0 con 0x55c68480eec0 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 -- 172.21.15.79:0/4264476658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 msgr2=0x55c6848112b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 0x55c6848112b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f915406b670 tx=0x7f91440095f0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915a26b640 1 --2- 172.21.15.79:0/832519564 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f91400a8280 0x7f91400aa740 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f914c098150 tx=0x7f914c00e040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 -- 172.21.15.79:0/4264476658 shutdown_connections 2024-04-22T06:46:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480eec0 0x55c6848112b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848062e0 0x55c68480e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 --2- 172.21.15.79:0/4264476658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848059c0 0x55c684805da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 -- 172.21.15.79:0/4264476658 >> 172.21.15.79:0/4264476658 conn(0x55c6848000f0 msgr2=0x55c6848004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:29.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 -- 172.21.15.79:0/4264476658 shutdown_connections 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.280+0000 7f915e995740 1 -- 172.21.15.79:0/4264476658 wait complete. 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.281+0000 7f915e995740 1 -- start start 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.281+0000 7f915e995740 1 Processor -- start 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.281+0000 7f915e995740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c68480bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f915e995740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 0x55c68480c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f915e995740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c68481ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f915e995740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c684769400 con 0x55c68480ca70 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f915e995740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c684725840 con 0x55c6848062e0 2024-04-22T06:46:29.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f915e995740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c68476e720 con 0x55c6848059c0 2024-04-22T06:46:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f915e995740 10 client.0 _ll_register_callbacks cb 0x55c684776ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f9159269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c68481ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f9159269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c68481ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51040/0 (socket says 172.21.15.79:51040) 2024-04-22T06:46:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.282+0000 7f9159269640 1 -- 172.21.15.79:0/4209649960 learned_addr learned my addr 172.21.15.79:0/4209649960 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f9159269640 1 -- 172.21.15.79:0/4209649960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 msgr2=0x55c68480bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:29.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f9159269640 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c68480bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f9159269640 1 -- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 msgr2=0x55c68480c530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f9159269640 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 0x55c68480c530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f9159269640 1 -- 172.21.15.79:0/4209649960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6845a9420 con 0x55c68480ca70 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f9159269640 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c68481ae60 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f914402c1c0 tx=0x7f914402cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87e654a4261d598 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.283+0000 7f913d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.284+0000 7f913d7fa640 1 -- 172.21.15.79:0/4209649960 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914404c020 con 0x55c68480ca70 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.284+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9144005d10 con 0x55c68480ca70 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.284+0000 7f913d7fa640 1 -- 172.21.15.79:0/4209649960 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f9144004430 con 0x55c68480ca70 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.284+0000 7f913d7fa640 1 -- 172.21.15.79:0/4209649960 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9144057050 con 0x55c68480ca70 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.284+0000 7f915e995740 10 client.16287 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:29.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.284+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c68443c580 con 0x55c68480ca70 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f913d7fa640 1 -- 172.21.15.79:0/4209649960 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9144005d10 con 0x55c68480ca70 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f913d7fa640 1 -- 172.21.15.79:0/4209649960 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f9144003710 con 0x55c68480ca70 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f915e995740 10 client.16287 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f915e995740 10 client.16287 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f915e995740 20 client.16287 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f915e995740 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68481f250 0x55c684821670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f915e995740 4 client.16287 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:29.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.285+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c684777b10 con 0x55c68481f250 2024-04-22T06:46:29.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.287+0000 7f9159a6a640 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68481f250 0x55c684821670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.289+0000 7f9159a6a640 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68481f250 0x55c684821670 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed6efe4c6e5e542 server_cookie=ab9e152e808ba82b in_seq=0 out_seq=0 2024-04-22T06:46:29.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.289+0000 7f913d7fa640 10 client.16287 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.292+0000 7f913d7fa640 1 -- 172.21.15.79:0/4209649960 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c684777b10 con 0x55c68481f250 2024-04-22T06:46:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.292+0000 7f913d7fa640 10 client.16287 handle_command_reply: tid=0 2024-04-22T06:46:29.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.292+0000 7f915e995740 1 client.16287 shutdown 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.292+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 msgr2=0x55c68481ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.292+0000 7f915e995740 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c68481ae60 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f914402c1c0 tx=0x7f914402cc00 comp rx=0 tx=0).stop 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 shutdown_connections 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68481f250 0x55c684821670 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 0x55c68480c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c68481ae60 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 --2- 172.21.15.79:0/4209649960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c68480bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 >> 172.21.15.79:0/4209649960 conn(0x55c6848000f0 msgr2=0x55c684801e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f913d7fa640 0 client.16287 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 shutdown_connections 2024-04-22T06:46:29.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 1 -- 172.21.15.79:0/4209649960 wait complete. 2024-04-22T06:46:29.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.293+0000 7f915e995740 20 client.16287 trim_cache size 0 max 16384 2024-04-22T06:46:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 Processor -- start 2024-04-22T06:46:29.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 -- start start 2024-04-22T06:46:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c684801c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 0x55c684826f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c684827480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f91541ac350 con 0x55c68480ca70 2024-04-22T06:46:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f91541aa1a0 con 0x55c6848062e0 2024-04-22T06:46:29.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.323+0000 7f915e995740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91541abc80 con 0x55c6848059c0 2024-04-22T06:46:29.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c684827480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c684827480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51044/0 (socket says 172.21.15.79:51044) 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 -- 172.21.15.79:0/3004996997 learned_addr learned my addr 172.21.15.79:0/3004996997 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 -- 172.21.15.79:0/3004996997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 msgr2=0x55c684801c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c684801c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 -- 172.21.15.79:0/3004996997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 msgr2=0x55c684826f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 0x55c684826f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.324+0000 7f9159269640 1 -- 172.21.15.79:0/3004996997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c684811620 con 0x55c68480ca70 2024-04-22T06:46:29.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.325+0000 7f9159269640 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c684827480 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9144055040 tx=0x7f9144005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d62a24f37936257 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.325+0000 7f913f7fe640 1 -- 172.21.15.79:0/3004996997 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144060070 con 0x55c68480ca70 2024-04-22T06:46:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.325+0000 7f913f7fe640 1 -- 172.21.15.79:0/3004996997 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9144005d10 con 0x55c68480ca70 2024-04-22T06:46:29.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.326+0000 7f913f7fe640 1 -- 172.21.15.79:0/3004996997 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914404c040 con 0x55c68480ca70 2024-04-22T06:46:29.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 -- 172.21.15.79:0/3004996997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 msgr2=0x55c684827480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c684827480 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9144055040 tx=0x7f9144005fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 -- 172.21.15.79:0/3004996997 shutdown_connections 2024-04-22T06:46:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6848062e0 0x55c684826f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c68480ca70 0x55c684827480 secure :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9144055040 tx=0x7f9144005fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 --2- 172.21.15.79:0/3004996997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c684801c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.327+0000 7f915e995740 1 -- 172.21.15.79:0/3004996997 >> 172.21.15.79:0/3004996997 conn(0x55c6848000f0 msgr2=0x55c684801540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.329+0000 7f915e995740 1 -- 172.21.15.79:0/3004996997 shutdown_connections 2024-04-22T06:46:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.329+0000 7f915e995740 1 -- 172.21.15.79:0/3004996997 wait complete. 2024-04-22T06:46:29.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.330+0000 7f915e995740 1 -- start start 2024-04-22T06:46:29.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.330+0000 7f915e995740 1 Processor -- start 2024-04-22T06:46:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.331+0000 7f915e995740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c684880ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.331+0000 7f915e995740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 0x55c684816dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.331+0000 7f915e995740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c684817300 0x55c684808c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.331+0000 7f915e995740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f91541aaee0 con 0x55c6848062e0 2024-04-22T06:46:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.331+0000 7f915e995740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f91541aa810 con 0x55c684817300 2024-04-22T06:46:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.331+0000 7f915e995740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91541ab5b0 con 0x55c6848059c0 2024-04-22T06:46:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f915e995740 10 client.0 _ll_register_callbacks cb 0x7f9144009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 0x55c684816dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 0x55c684816dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51054/0 (socket says 172.21.15.79:51054) 2024-04-22T06:46:29.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 -- 172.21.15.79:0/1673346037 learned_addr learned my addr 172.21.15.79:0/1673346037 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 -- 172.21.15.79:0/1673346037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 msgr2=0x55c684880ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c684880ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 -- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c684817300 msgr2=0x55c684808c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c684817300 0x55c684808c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.332+0000 7f9159a6a640 1 -- 172.21.15.79:0/1673346037 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6848091a0 con 0x55c6848062e0 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.333+0000 7f9159a6a640 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 0x55c684816dc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9150000c00 tx=0x7f9150009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf115511b9bff1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:29.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.333+0000 7f913d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.333+0000 7f913d7fa640 1 -- 172.21.15.79:0/1673346037 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150012a60 con 0x55c6848062e0 2024-04-22T06:46:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.334+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9144005d10 con 0x55c6848062e0 2024-04-22T06:46:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.334+0000 7f913d7fa640 1 -- 172.21.15.79:0/1673346037 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f9150012c00 con 0x55c6848062e0 2024-04-22T06:46:29.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.334+0000 7f913d7fa640 1 -- 172.21.15.79:0/1673346037 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9150012e00 con 0x55c6848062e0 2024-04-22T06:46:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.334+0000 7f915e995740 10 client.16293 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.334+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c68488e820 con 0x55c6848062e0 2024-04-22T06:46:29.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.334+0000 7f913d7fa640 1 -- 172.21.15.79:0/1673346037 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f915001a070 con 0x55c6848062e0 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f913d7fa640 1 -- 172.21.15.79:0/1673346037 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f915001f690 con 0x55c6848062e0 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f915e995740 10 client.16293 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f915e995740 10 client.16293 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f915e995740 20 client.16293 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f915e995740 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68488eff0 0x55c684891410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f915e995740 4 client.16293 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.35, "await": true}] 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.335+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 79.35, "await": true}) -- 0x7f9144005d10 con 0x55c68488eff0 2024-04-22T06:46:29.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.337+0000 7f9159269640 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68488eff0 0x55c684891410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:29.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.346+0000 7f9159269640 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68488eff0 0x55c684891410 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35069593b272b279 server_cookie=480439747aa089a5 in_seq=0 out_seq=0 2024-04-22T06:46:29.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:29.346+0000 7f913d7fa640 10 client.16293 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:29.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:29 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3780392831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:29.440 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:29 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-22T06:46:29.440 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:29 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.129s CPU time. 2024-04-22T06:46:29.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:29 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-22T06:46:29.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:29 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.116s CPU time. 2024-04-22T06:46:29.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:29 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3780392831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:29.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:29 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3780392831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:30.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:30 smithi079 ceph-mon[27946]: pgmap v67: 97 pgs: 97 active+clean; 2.3 GiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 41 MiB/s wr, 18 op/s 2024-04-22T06:46:30.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:30 smithi193 ceph-mon[31815]: pgmap v67: 97 pgs: 97 active+clean; 2.3 GiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 41 MiB/s wr, 18 op/s 2024-04-22T06:46:30.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:30 smithi190 ceph-mon[31421]: pgmap v67: 97 pgs: 97 active+clean; 2.3 GiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 41 MiB/s wr, 18 op/s 2024-04-22T06:46:31.412 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:31 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:31.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:31 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:31.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:31 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:32.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:32 smithi079 ceph-mon[27946]: pgmap v68: 97 pgs: 97 active+clean; 4.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 241 MiB/s wr, 193 op/s 2024-04-22T06:46:32.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:32 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:32 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:32 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:32 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 -- 172.21.15.79:0/2836759747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 msgr2=0x7f2e54099fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 --2- 172.21.15.79:0/2836759747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54099fd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009900 tx=0x7f2e4c02bc60 comp rx=0 tx=0).stop 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 -- 172.21.15.79:0/2836759747 shutdown_connections 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 --2- 172.21.15.79:0/2836759747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 0x7f2e540952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 --2- 172.21.15.79:0/2836759747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e54095820 0x7f2e540a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 --2- 172.21.15.79:0/2836759747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54099fd0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.651+0000 7f2e60c01640 1 -- 172.21.15.79:0/2836759747 >> 172.21.15.79:0/2836759747 conn(0x7f2e5400a770 msgr2=0x7f2e5400ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.652+0000 7f2e60c01640 1 -- 172.21.15.79:0/2836759747 shutdown_connections 2024-04-22T06:46:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.652+0000 7f2e60c01640 1 -- 172.21.15.79:0/2836759747 wait complete. 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.653+0000 7f2e60c01640 1 Processor -- start 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.653+0000 7f2e60c01640 1 -- start start 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.653+0000 7f2e60c01640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 0x7f2e5412e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.653+0000 7f2e60c01640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e54095820 0x7f2e5412eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.653+0000 7f2e60c01640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54132ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54132ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e60c01640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540a4fc0 con 0x7f2e54099bf0 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e60c01640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540a4e40 con 0x7f2e54094e80 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e60c01640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540a5140 con 0x7f2e54095820 2024-04-22T06:46:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54132ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58846/0 (socket says 172.21.15.79:58846) 2024-04-22T06:46:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/237566019 learned_addr learned my addr 172.21.15.79:0/237566019 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 0x7f2e5412e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/237566019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e54095820 msgr2=0x7f2e5412eba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e54095820 0x7f2e5412eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/237566019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 msgr2=0x7f2e5412e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 0x7f2e5412e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/237566019 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e54133600 con 0x7f2e54099bf0 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.654+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 0x7f2e5412e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.655+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54132ed0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c02db50 tx=0x7f2e5c0785f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.655+0000 7f2e4b7fe640 1 -- 172.21.15.79:0/237566019 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c07d420 con 0x7f2e54099bf0 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.655+0000 7f2e4b7fe640 1 -- 172.21.15.79:0/237566019 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5c075050 con 0x7f2e54099bf0 2024-04-22T06:46:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.655+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541338f0 con 0x7f2e54099bf0 2024-04-22T06:46:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.656+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5413b180 con 0x7f2e54099bf0 2024-04-22T06:46:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.656+0000 7f2e4b7fe640 1 -- 172.21.15.79:0/237566019 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c07c720 con 0x7f2e54099bf0 2024-04-22T06:46:32.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.660+0000 7f2e4b7fe640 1 -- 172.21.15.79:0/237566019 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2e5c07db00 con 0x7f2e54099bf0 2024-04-22T06:46:32.660 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.662+0000 7f2e4b7fe640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2e440a3e20 0x7f2e440a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.661 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.662+0000 7f2e637ea740 1 Processor -- start 2024-04-22T06:46:32.661 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.662+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2e440a3e20 0x7f2e440a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.661 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e4b7fe640 1 -- 172.21.15.79:0/237566019 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2e5c13cda0 con 0x7f2e54099bf0 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 -- start start 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 0x5600134a7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56001340c8d0 con 0x5600134b12d0 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5600133a9e80 con 0x5600134a7d50 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.663+0000 7f2e637ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560013408890 con 0x5600134a8ac0 2024-04-22T06:46:32.662 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.663 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58848/0 (socket says 172.21.15.79:58848) 2024-04-22T06:46:32.663 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2751515422 learned_addr learned my addr 172.21.15.79:0/2751515422 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:32.663 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2e440a3e20 0x7f2e440a62e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0096c0 tx=0x7f2e4c000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:32.663 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2751515422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 msgr2=0x5600134a7810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:32.663 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 0x5600134a7810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2751515422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 msgr2=0x5600134a81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2751515422 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600133cb320 con 0x5600134b12d0 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.664+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134b36c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c105630 tx=0x7f2e5c109360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84401707067b5cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.665+0000 7f2e48ff9640 1 -- 172.21.15.79:0/2751515422 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c1118d0 con 0x5600134b12d0 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.665+0000 7f2e48ff9640 1 -- 172.21.15.79:0/2751515422 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5c106f60 con 0x5600134b12d0 2024-04-22T06:46:32.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.665+0000 7f2e48ff9640 1 -- 172.21.15.79:0/2751515422 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c13a890 con 0x5600134b12d0 2024-04-22T06:46:32.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 -- 172.21.15.79:0/2751515422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 msgr2=0x5600134b36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:32.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134b36c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c105630 tx=0x7f2e5c109360 comp rx=0 tx=0).stop 2024-04-22T06:46:32.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 -- 172.21.15.79:0/2751515422 shutdown_connections 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134b36c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2751515422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 0x5600134a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 -- 172.21.15.79:0/2751515422 >> 172.21.15.79:0/2751515422 conn(0x56001346ec90 msgr2=0x5600134a2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 -- 172.21.15.79:0/2751515422 shutdown_connections 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.666+0000 7f2e637ea740 1 -- 172.21.15.79:0/2751515422 wait complete. 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.667+0000 7f2e637ea740 1 -- start start 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.667+0000 7f2e637ea740 1 Processor -- start 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.667+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.667+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 0x5600134ae030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.667+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134ae640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58852/0 (socket says 172.21.15.79:58852) 2024-04-22T06:46:32.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3025502943 learned_addr learned my addr 172.21.15.79:0/3025502943 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5600134b3fa0 con 0x5600134b12d0 2024-04-22T06:46:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560013394eb0 con 0x5600134a7d50 2024-04-22T06:46:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600134b4120 con 0x5600134a8ac0 2024-04-22T06:46:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e637ea740 10 client.0 _ll_register_callbacks cb 0x5600133c1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3025502943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 msgr2=0x5600134ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 0x5600134ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 msgr2=0x5600134b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.668+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56001311f210 con 0x5600134b12d0 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134ae640 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c02db80 tx=0x7f2e5c0503e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f13b1c50fcc321 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e41ffb640 1 -- 172.21.15.79:0/3025502943 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c1118d0 con 0x5600134b12d0 2024-04-22T06:46:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e5c106f60 con 0x5600134b12d0 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e41ffb640 1 -- 172.21.15.79:0/3025502943 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2e5c108ec0 con 0x5600134b12d0 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e41ffb640 1 -- 172.21.15.79:0/3025502943 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2e5c1090c0 con 0x5600134b12d0 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e637ea740 10 client.16305 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.669+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5600131eb620 con 0x5600134b12d0 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e41ffb640 1 -- 172.21.15.79:0/3025502943 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5c106f60 con 0x5600134b12d0 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e41ffb640 1 -- 172.21.15.79:0/3025502943 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2e5c1049b0 con 0x5600134b12d0 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e637ea740 10 client.16305 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e637ea740 10 client.16305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e637ea740 20 client.16305 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b8120 0x5600134ba510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e637ea740 4 client.16305 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.670+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560013420880 con 0x5600134b8120 2024-04-22T06:46:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.671+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b8120 0x5600134ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.672+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b8120 0x5600134ba510 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c82b1869ea0cb84 server_cookie=b4ce5a945c02f314 in_seq=0 out_seq=0 2024-04-22T06:46:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.672+0000 7f2e41ffb640 10 client.16305 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e41ffb640 1 -- 172.21.15.79:0/3025502943 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560013420880 con 0x5600134b8120 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e41ffb640 10 client.16305 handle_command_reply: tid=0 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 client.16305 shutdown 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 msgr2=0x5600134ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134ae640 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c02db80 tx=0x7f2e5c0503e0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 shutdown_connections 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b8120 0x5600134ba510 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134b12d0 0x5600134ae640 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a8ac0 0x5600134ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3025502943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a7d50 0x5600134b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 >> 172.21.15.79:0/3025502943 conn(0x56001346ec90 msgr2=0x5600134a4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e41ffb640 0 client.16305 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 shutdown_connections 2024-04-22T06:46:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.675+0000 7f2e637ea740 1 -- 172.21.15.79:0/3025502943 wait complete. 2024-04-22T06:46:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.676+0000 7f2e637ea740 20 client.16305 trim_cache size 0 max 16384 2024-04-22T06:46:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:32 smithi193 ceph-mon[31815]: pgmap v68: 97 pgs: 97 active+clean; 4.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 241 MiB/s wr, 193 op/s 2024-04-22T06:46:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:32 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:32 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:32 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:32 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:32 smithi190 ceph-mon[31421]: pgmap v68: 97 pgs: 97 active+clean; 4.7 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 241 MiB/s wr, 193 op/s 2024-04-22T06:46:32.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:32 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:32 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:32 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:32 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.708+0000 7f2e637ea740 1 Processor -- start 2024-04-22T06:46:32.707 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.708+0000 7f2e637ea740 1 -- start start 2024-04-22T06:46:32.707 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.709+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56001346ec90 0x5600134ad190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.709+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134a7d50 0x5600134a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e637ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54142000 con 0x5600134a7d50 2024-04-22T06:46:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e5a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56001346ec90 0x5600134ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e5a59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56001346ec90 0x5600134ad190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:41682/0 (socket says 172.21.15.79:41682) 2024-04-22T06:46:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e5a59c640 1 -- 172.21.15.79:0/3824067690 learned_addr learned my addr 172.21.15.79:0/3824067690 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5413f8d0 con 0x5600134a8ac0 2024-04-22T06:46:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.710+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5413d760 con 0x56001346ec90 2024-04-22T06:46:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.711+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.715+0000 7f2e59d9b640 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134a7d50 0x5600134a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.716+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3824067690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56001346ec90 msgr2=0x5600134ad190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.716+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56001346ec90 0x5600134ad190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.716+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3824067690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134a7d50 msgr2=0x5600134a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.716+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134a7d50 0x5600134a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.716+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/3824067690 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600134a7020 con 0x5600134a8ac0 2024-04-22T06:46:32.726 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.727+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134b0760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c067c00 tx=0x7f2e5c109360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae3f2cce32331d78 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.743+0000 7f2e43fff640 1 -- 172.21.15.79:0/3824067690 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c10d010 con 0x5600134a8ac0 2024-04-22T06:46:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.743+0000 7f2e43fff640 1 -- 172.21.15.79:0/3824067690 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5c108ec0 con 0x5600134a8ac0 2024-04-22T06:46:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.743+0000 7f2e43fff640 1 -- 172.21.15.79:0/3824067690 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c07d5c0 con 0x5600134a8ac0 2024-04-22T06:46:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.744+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 msgr2=0x5600134b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.744+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134b0760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c067c00 tx=0x7f2e5c109360 comp rx=0 tx=0).stop 2024-04-22T06:46:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.746+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 shutdown_connections 2024-04-22T06:46:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.746+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134b0760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.746+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5600134a7d50 0x5600134a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.746+0000 7f2e637ea740 1 --2- 172.21.15.79:0/3824067690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56001346ec90 0x5600134ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.746+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 >> 172.21.15.79:0/3824067690 conn(0x5600134a9a20 msgr2=0x5600134a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.746+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 shutdown_connections 2024-04-22T06:46:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.750+0000 7f2e637ea740 1 -- 172.21.15.79:0/3824067690 wait complete. 2024-04-22T06:46:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.750+0000 7f2e637ea740 1 -- start start 2024-04-22T06:46:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 Processor -- start 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56001346ec90 0x5600134bf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a7d50 0x5600134bfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134bcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5413ddb0 con 0x56001346ec90 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5413f200 con 0x5600134a8ac0 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5413e460 con 0x5600134a7d50 2024-04-22T06:46:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.751+0000 7f2e637ea740 10 client.0 _ll_register_callbacks cb 0x7f2e5c108870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.753+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134bcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.753+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134bcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58500/0 (socket says 172.21.15.79:58500) 2024-04-22T06:46:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.753+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2095182023 learned_addr learned my addr 172.21.15.79:0/2095182023 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.756+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2095182023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a7d50 msgr2=0x5600134bfc90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.756+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a7d50 0x5600134bfc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.756+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2095182023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56001346ec90 msgr2=0x5600134bf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.756+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56001346ec90 0x5600134bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.756+0000 7f2e5ad9d640 1 -- 172.21.15.79:0/2095182023 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600134bd3c0 con 0x5600134a8ac0 2024-04-22T06:46:32.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.773+0000 7f2e5ad9d640 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134bcc80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c064fa0 tx=0x7f2e5c07c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9898fea36689add5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:32.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.774+0000 7f2e41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:46:32.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.777+0000 7f2e41ffb640 1 -- 172.21.15.79:0/2095182023 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c13ad60 con 0x5600134a8ac0 2024-04-22T06:46:32.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.777+0000 7f2e41ffb640 1 -- 172.21.15.79:0/2095182023 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f2e5c1080b0 con 0x5600134a8ac0 2024-04-22T06:46:32.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.777+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e5c108ec0 con 0x5600134a8ac0 2024-04-22T06:46:32.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.782+0000 7f2e41ffb640 1 -- 172.21.15.79:0/2095182023 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2e5c111650 con 0x5600134a8ac0 2024-04-22T06:46:32.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.784+0000 7f2e637ea740 10 client.25538 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:32.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.784+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5600134bd3c0 con 0x5600134a8ac0 2024-04-22T06:46:32.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.788+0000 7f2e41ffb640 1 -- 172.21.15.79:0/2095182023 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e5c108ec0 con 0x5600134a8ac0 2024-04-22T06:46:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.788+0000 7f2e41ffb640 1 -- 172.21.15.79:0/2095182023 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2e5c107b20 con 0x5600134a8ac0 2024-04-22T06:46:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.789+0000 7f2e637ea740 10 client.25538 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.789+0000 7f2e637ea740 10 client.25538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.789+0000 7f2e637ea740 20 client.25538 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.789+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b12d0 0x5600135291a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:32.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.789+0000 7f2e637ea740 4 client.25538 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:32.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.789+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600133ac3d0 con 0x5600134b12d0 2024-04-22T06:46:32.793 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.794+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b12d0 0x5600135291a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:32.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.806+0000 7f2e5a59c640 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b12d0 0x5600135291a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=996a3c39b5897fad server_cookie=636af861fa07a08d in_seq=0 out_seq=0 2024-04-22T06:46:32.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:32.806+0000 7f2e41ffb640 10 client.25538 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.651+0000 7f2e41ffb640 1 -- 172.21.15.79:0/2095182023 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600133ac3d0 con 0x5600134b12d0 2024-04-22T06:46:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.651+0000 7f2e41ffb640 10 client.25538 handle_command_reply: tid=0 2024-04-22T06:46:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.652+0000 7f2e637ea740 1 client.25538 shutdown 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.652+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 msgr2=0x5600134bcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.652+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134bcc80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c064fa0 tx=0x7f2e5c07c8c0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 shutdown_connections 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5600134b12d0 0x5600135291a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5600134a8ac0 0x5600134bcc80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600134a7d50 0x5600134bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 --2- 172.21.15.79:0/2095182023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56001346ec90 0x5600134bf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 >> 172.21.15.79:0/2095182023 conn(0x5600134a9a20 msgr2=0x5600134a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e41ffb640 0 client.25538 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 shutdown_connections 2024-04-22T06:46:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.653+0000 7f2e637ea740 1 -- 172.21.15.79:0/2095182023 wait complete. 2024-04-22T06:46:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.654+0000 7f2e637ea740 20 client.25538 trim_cache size 0 max 16384 2024-04-22T06:46:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:46:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:46:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:46:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:46:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2e440a3e20 msgr2=0x7f2e440a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2e440a3e20 0x7f2e440a62e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0096c0 tx=0x7f2e4c000bc0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 msgr2=0x7f2e54132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54132ed0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c02db50 tx=0x7f2e5c0785f0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 shutdown_connections 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e54099bf0 0x7f2e54132ed0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e54095820 0x7f2e5412eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2e440a3e20 0x7f2e440a62e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 --2- 172.21.15.79:0/237566019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e54094e80 0x7f2e5412e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 >> 172.21.15.79:0/237566019 conn(0x7f2e5400a770 msgr2=0x7f2e54004cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 shutdown_connections 2024-04-22T06:46:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:34.666+0000 7f2e60c01640 1 -- 172.21.15.79:0/237566019 wait complete. 2024-04-22T06:46:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:34 smithi193 ceph-mon[31815]: pgmap v69: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 253 MiB/s wr, 190 op/s 2024-04-22T06:46:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:34 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:34 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:34 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:34.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:34 smithi190 ceph-mon[31421]: pgmap v69: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 253 MiB/s wr, 190 op/s 2024-04-22T06:46:34.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:34 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:34.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:34 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:34.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:34 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:34.744 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:34 smithi079 ceph-mon[27946]: pgmap v69: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 253 MiB/s wr, 190 op/s 2024-04-22T06:46:34.744 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:34 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:34.745 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:34 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:34.745 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:34 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:46:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:36 smithi193 ceph-mon[31815]: pgmap v70: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 253 MiB/s wr, 190 op/s 2024-04-22T06:46:36.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:36 smithi190 ceph-mon[31421]: pgmap v70: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 253 MiB/s wr, 190 op/s 2024-04-22T06:46:36.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:36 smithi079 ceph-mon[27946]: pgmap v70: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 253 MiB/s wr, 190 op/s 2024-04-22T06:46:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:38 smithi193 ceph-mon[31815]: pgmap v71: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 MiB/s wr, 912 op/s 2024-04-22T06:46:38.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:38 smithi190 ceph-mon[31421]: pgmap v71: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 MiB/s wr, 912 op/s 2024-04-22T06:46:38.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:38 smithi079 ceph-mon[27946]: pgmap v71: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 MiB/s wr, 912 op/s 2024-04-22T06:46:39.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:39.256 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:39.442 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:39 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:46:39.442 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:39 smithi193 systemd[1]: Stopped Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:39.443 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:39 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.129s CPU time. 2024-04-22T06:46:39.443 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:39 smithi193 systemd[1]: Starting Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:46:39.517 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:39 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:46:39.517 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:39 smithi190 systemd[1]: Stopped Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:39.517 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:39 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.116s CPU time. 2024-04-22T06:46:39.517 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:39 smithi190 systemd[1]: Starting Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:46:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.581+0000 7ff8333e8640 1 -- 172.21.15.79:0/2613688038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10e390 msgr2=0x7ff82c10e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.581+0000 7ff8333e8640 1 --2- 172.21.15.79:0/2613688038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10e390 0x7ff82c10e770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff824009ca0 tx=0x7ff82402c1c0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.583+0000 7ff8333e8640 1 -- 172.21.15.79:0/2613688038 shutdown_connections 2024-04-22T06:46:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.583+0000 7ff8333e8640 1 --2- 172.21.15.79:0/2613688038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c10bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.583+0000 7ff8333e8640 1 --2- 172.21.15.79:0/2613688038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff82c10c690 0x7ff82c07a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.583+0000 7ff8333e8640 1 --2- 172.21.15.79:0/2613688038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10e390 0x7ff82c10e770 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.583+0000 7ff8333e8640 1 -- 172.21.15.79:0/2613688038 >> 172.21.15.79:0/2613688038 conn(0x7ff82c06ced0 msgr2=0x7ff82c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:39.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.586+0000 7ff8333e8640 1 -- 172.21.15.79:0/2613688038 shutdown_connections 2024-04-22T06:46:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.586+0000 7ff8333e8640 1 -- 172.21.15.79:0/2613688038 wait complete. 2024-04-22T06:46:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.587+0000 7ff8333e8640 1 Processor -- start 2024-04-22T06:46:39.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 -- start start 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10c690 0x7ff82c07edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff82c10e390 0x7ff82c07f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c07f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c07d680 con 0x7ff82c10e390 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c07d500 con 0x7ff82c10ecb0 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.588+0000 7ff8333e8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c07d800 con 0x7ff82c10c690 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.589+0000 7ff832be7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c07f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.589+0000 7ff832be7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c07f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58524/0 (socket says 172.21.15.79:58524) 2024-04-22T06:46:39.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.589+0000 7ff832be7640 1 -- 172.21.15.79:0/1200622905 learned_addr learned my addr 172.21.15.79:0/1200622905 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:39.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.589+0000 7ff8323e6640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10c690 0x7ff82c07edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.591+0000 7ff831be5640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff82c10e390 0x7ff82c07f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.592+0000 7ff832be7640 1 -- 172.21.15.79:0/1200622905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10c690 msgr2=0x7ff82c07edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.592+0000 7ff832be7640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10c690 0x7ff82c07edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.592+0000 7ff832be7640 1 -- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff82c10e390 msgr2=0x7ff82c07f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.592+0000 7ff832be7640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff82c10e390 0x7ff82c07f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.592+0000 7ff832be7640 1 -- 172.21.15.79:0/1200622905 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c1bc0e0 con 0x7ff82c10ecb0 2024-04-22T06:46:39.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.593+0000 7ff832be7640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c07f840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff828009fd0 tx=0x7ff82800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:39.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.594+0000 7ff8237fe640 1 -- 172.21.15.79:0/1200622905 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828011830 con 0x7ff82c10ecb0 2024-04-22T06:46:39.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.594+0000 7ff8237fe640 1 -- 172.21.15.79:0/1200622905 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff828016070 con 0x7ff82c10ecb0 2024-04-22T06:46:39.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.594+0000 7ff8237fe640 1 -- 172.21.15.79:0/1200622905 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828010de0 con 0x7ff82c10ecb0 2024-04-22T06:46:39.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.595+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82c1bc3d0 con 0x7ff82c10ecb0 2024-04-22T06:46:39.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.595+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c1bc8d0 con 0x7ff82c10ecb0 2024-04-22T06:46:39.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.601+0000 7ff83723e740 1 Processor -- start 2024-04-22T06:46:39.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.606+0000 7ff83723e740 1 -- start start 2024-04-22T06:46:39.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.606+0000 7ff83723e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b44ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff83723e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b4d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff83723e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff83723e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8aa91e0 con 0x55e9f8b4dc00 2024-04-22T06:46:39.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff83723e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8a46390 con 0x55e9f8b44700 2024-04-22T06:46:39.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff83723e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8aa51a0 con 0x55e9f8b45020 2024-04-22T06:46:39.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b4d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b4d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:41718/0 (socket says 172.21.15.79:41718) 2024-04-22T06:46:39.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.607+0000 7ff8323e6640 1 -- 172.21.15.79:0/3085035387 learned_addr learned my addr 172.21.15.79:0/3085035387 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:39.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.608+0000 7ff831be5640 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b44ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.610+0000 7ff8323e6640 1 -- 172.21.15.79:0/3085035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 msgr2=0x55e9f8b44ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.610+0000 7ff8323e6640 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b44ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.610+0000 7ff8323e6640 1 -- 172.21.15.79:0/3085035387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 msgr2=0x55e9f8b4fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.610+0000 7ff8323e6640 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.610+0000 7ff8323e6640 1 -- 172.21.15.79:0/3085035387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f88857d0 con 0x55e9f8b45020 2024-04-22T06:46:39.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.611+0000 7ff8323e6640 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b4d6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff824009ca0 tx=0x7ff824005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8580ca8581a793 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:39.613 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.615+0000 7ff8237fe640 1 -- 172.21.15.79:0/1200622905 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff8280119d0 con 0x7ff82c10ecb0 2024-04-22T06:46:39.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.616+0000 7ff820bfe640 1 -- 172.21.15.79:0/3085035387 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824039040 con 0x55e9f8b45020 2024-04-22T06:46:39.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.616+0000 7ff820bfe640 1 -- 172.21.15.79:0/3085035387 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff824003b70 con 0x55e9f8b45020 2024-04-22T06:46:39.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.616+0000 7ff820bfe640 1 -- 172.21.15.79:0/3085035387 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8240328b0 con 0x55e9f8b45020 2024-04-22T06:46:39.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.616+0000 7ff8237fe640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff8180a3f50 0x7ff8180a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.616+0000 7ff8237fe640 1 -- 172.21.15.79:0/1200622905 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7ff8280cf6f0 con 0x7ff82c10ecb0 2024-04-22T06:46:39.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.620+0000 7ff83723e740 1 -- 172.21.15.79:0/3085035387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 msgr2=0x55e9f8b4d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.620+0000 7ff83723e740 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b4d6c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff824009ca0 tx=0x7ff824005fa0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.623+0000 7ff831be5640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff8180a3f50 0x7ff8180a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.631+0000 7ff83723e740 1 -- 172.21.15.79:0/3085035387 shutdown_connections 2024-04-22T06:46:39.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.631+0000 7ff83723e740 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.631+0000 7ff83723e740 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b4d6c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.631+0000 7ff83723e740 1 --2- 172.21.15.79:0/3085035387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b44ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.631+0000 7ff83723e740 1 -- 172.21.15.79:0/3085035387 >> 172.21.15.79:0/3085035387 conn(0x55e9f8b3ee30 msgr2=0x55e9f8b3f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:39.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.640+0000 7ff83723e740 1 -- 172.21.15.79:0/3085035387 shutdown_connections 2024-04-22T06:46:39.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.640+0000 7ff831be5640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff8180a3f50 0x7ff8180a6410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55e9f8b3ffc0 tx=0x7ff81c008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:39.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.644+0000 7ff83723e740 1 -- 172.21.15.79:0/3085035387 wait complete. 2024-04-22T06:46:39.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.644+0000 7ff83723e740 1 -- start start 2024-04-22T06:46:39.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.645+0000 7ff83723e740 1 Processor -- start 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b45020 0x55e9f8b4b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8b50990 con 0x55e9f8b44700 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8a622f0 con 0x55e9f8b45020 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8b50b10 con 0x55e9f8b4dc00 2024-04-22T06:46:39.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.647+0000 7ff83723e740 10 client.0 _ll_register_callbacks cb 0x55e9f8a6f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:39.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.652+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.652+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58898/0 (socket says 172.21.15.79:58898) 2024-04-22T06:46:39.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.652+0000 7ff8323e6640 1 -- 172.21.15.79:0/141741186 learned_addr learned my addr 172.21.15.79:0/141741186 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:39.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.653+0000 7ff831be5640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b45020 0x55e9f8b4b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.653+0000 7ff832be7640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.655+0000 7ff8323e6640 1 -- 172.21.15.79:0/141741186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b4dc00 msgr2=0x55e9f8b4b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.655+0000 7ff8323e6640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.655+0000 7ff8323e6640 1 -- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b45020 msgr2=0x55e9f8b4b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.655+0000 7ff8323e6640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b45020 0x55e9f8b4b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.655+0000 7ff8323e6640 1 -- 172.21.15.79:0/141741186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f87bb3f0 con 0x55e9f8b44700 2024-04-22T06:46:39.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.656+0000 7ff8323e6640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4ac00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff824009bf0 tx=0x7ff824004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a604971457fa2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:39.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.659+0000 7ff815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:39.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.659+0000 7ff815ffb640 1 -- 172.21.15.79:0/141741186 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82404c020 con 0x55e9f8b44700 2024-04-22T06:46:39.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.659+0000 7ff815ffb640 1 -- 172.21.15.79:0/141741186 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7ff824032740 con 0x55e9f8b44700 2024-04-22T06:46:39.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.659+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff824003b70 con 0x55e9f8b44700 2024-04-22T06:46:39.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.672+0000 7ff815ffb640 1 -- 172.21.15.79:0/141741186 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff824057050 con 0x55e9f8b44700 2024-04-22T06:46:39.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.672+0000 7ff83723e740 10 client.16314 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:39.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.672+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e9f877b0f0 con 0x55e9f8b44700 2024-04-22T06:46:39.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff815ffb640 1 -- 172.21.15.79:0/141741186 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff824003b70 con 0x55e9f8b44700 2024-04-22T06:46:39.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff815ffb640 1 -- 172.21.15.79:0/141741186 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff82403fb70 con 0x55e9f8b44700 2024-04-22T06:46:39.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff83723e740 10 client.16314 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:39.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff83723e740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:39.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff83723e740 20 client.16314 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:39.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff83723e740 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8b5daf0 0x55e9f8b5ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff83723e740 4 client.16314 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:39.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.674+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f8ab6c00 con 0x55e9f8b5daf0 2024-04-22T06:46:39.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.676+0000 7ff831be5640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8b5daf0 0x55e9f8b5ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.696+0000 7ff831be5640 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8b5daf0 0x55e9f8b5ffb0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c627998d8b2390f server_cookie=6f8676a88b9840d9 in_seq=0 out_seq=0 2024-04-22T06:46:39.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.699+0000 7ff815ffb640 10 client.16314 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:39.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.699+0000 7ff815ffb640 1 -- 172.21.15.79:0/141741186 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9f8ab6c00 con 0x55e9f8b5daf0 2024-04-22T06:46:39.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.699+0000 7ff815ffb640 10 client.16314 handle_command_reply: tid=0 2024-04-22T06:46:39.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.699+0000 7ff83723e740 1 client.16314 shutdown 2024-04-22T06:46:39.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.706+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 msgr2=0x55e9f8b4ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.706+0000 7ff83723e740 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4ac00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff824009bf0 tx=0x7ff824004470 comp rx=0 tx=0).stop 2024-04-22T06:46:39.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 shutdown_connections 2024-04-22T06:46:39.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff83723e740 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8b5daf0 0x55e9f8b5ffb0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff83723e740 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b45020 0x55e9f8b4b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff83723e740 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b4b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff83723e740 1 --2- 172.21.15.79:0/141741186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4ac00 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 >> 172.21.15.79:0/141741186 conn(0x55e9f8b3ee30 msgr2=0x55e9f8b40b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:39.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.709+0000 7ff815ffb640 0 client.16314 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.726+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 shutdown_connections 2024-04-22T06:46:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.726+0000 7ff83723e740 1 -- 172.21.15.79:0/141741186 wait complete. 2024-04-22T06:46:39.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.726+0000 7ff83723e740 20 client.16314 trim_cache size 0 max 16384 2024-04-22T06:46:39.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.790+0000 7ff83723e740 1 Processor -- start 2024-04-22T06:46:39.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.792+0000 7ff83723e740 1 -- start start 2024-04-22T06:46:39.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.792+0000 7ff83723e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.793+0000 7ff83723e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.793+0000 7ff83723e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b65880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.793+0000 7ff83723e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff8100111e0 con 0x55e9f8b44700 2024-04-22T06:46:39.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.793+0000 7ff83723e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff81000eab0 con 0x55e9f8b4dc00 2024-04-22T06:46:39.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.793+0000 7ff83723e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff81000c900 con 0x55e9f8b45020 2024-04-22T06:46:39.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.794+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.794+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58900/0 (socket says 172.21.15.79:58900) 2024-04-22T06:46:39.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.794+0000 7ff8323e6640 1 -- 172.21.15.79:0/1488135188 learned_addr learned my addr 172.21.15.79:0/1488135188 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:39.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.794+0000 7ff831be5640 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.795+0000 7ff832be7640 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b65880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.796+0000 7ff8323e6640 1 -- 172.21.15.79:0/1488135188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 msgr2=0x55e9f8b598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.796+0000 7ff8323e6640 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.796+0000 7ff8323e6640 1 -- 172.21.15.79:0/1488135188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b4dc00 msgr2=0x55e9f8b65880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.796+0000 7ff8323e6640 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b65880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.796+0000 7ff8323e6640 1 -- 172.21.15.79:0/1488135188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f8b50b10 con 0x55e9f8b44700 2024-04-22T06:46:39.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.796+0000 7ff8323e6640 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4d330 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff824002230 tx=0x7ff824005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9cf1d2751efa20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:39.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.797+0000 7ff817fff640 1 -- 172.21.15.79:0/1488135188 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82404c070 con 0x55e9f8b44700 2024-04-22T06:46:39.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.798+0000 7ff817fff640 1 -- 172.21.15.79:0/1488135188 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff824004030 con 0x55e9f8b44700 2024-04-22T06:46:39.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.799+0000 7ff817fff640 1 -- 172.21.15.79:0/1488135188 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824039070 con 0x55e9f8b44700 2024-04-22T06:46:39.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.799+0000 7ff83723e740 1 -- 172.21.15.79:0/1488135188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 msgr2=0x55e9f8b4d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.799+0000 7ff83723e740 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4d330 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff824002230 tx=0x7ff824005bd0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.800+0000 7ff83723e740 1 -- 172.21.15.79:0/1488135188 shutdown_connections 2024-04-22T06:46:39.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.800+0000 7ff83723e740 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b65880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.800+0000 7ff83723e740 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.800+0000 7ff83723e740 1 --2- 172.21.15.79:0/1488135188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b44700 0x55e9f8b4d330 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.800+0000 7ff83723e740 1 -- 172.21.15.79:0/1488135188 >> 172.21.15.79:0/1488135188 conn(0x55e9f8b3ee30 msgr2=0x55e9f8b3d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:39.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.804+0000 7ff83723e740 1 -- 172.21.15.79:0/1488135188 shutdown_connections 2024-04-22T06:46:39.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.804+0000 7ff83723e740 1 -- 172.21.15.79:0/1488135188 wait complete. 2024-04-22T06:46:39.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.804+0000 7ff83723e740 1 -- start start 2024-04-22T06:46:39.803 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.805+0000 7ff83723e740 1 Processor -- start 2024-04-22T06:46:39.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.806+0000 7ff83723e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b55d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.806+0000 7ff83723e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b562d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.806+0000 7ff83723e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b51290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.806+0000 7ff83723e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff81000cf70 con 0x55e9f8b4dc00 2024-04-22T06:46:39.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.807+0000 7ff83723e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff81000e3e0 con 0x55e9f8b44700 2024-04-22T06:46:39.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.807+0000 7ff83723e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff81000d640 con 0x55e9f8b45020 2024-04-22T06:46:39.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.807+0000 7ff83723e740 10 client.0 _ll_register_callbacks cb 0x7ff824005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:39.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.807+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b55d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.807+0000 7ff8323e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b55d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58578/0 (socket says 172.21.15.79:58578) 2024-04-22T06:46:39.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.807+0000 7ff8323e6640 1 -- 172.21.15.79:0/4100124162 learned_addr learned my addr 172.21.15.79:0/4100124162 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:39.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.808+0000 7ff832be7640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b51290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.808+0000 7ff831be5640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.810+0000 7ff8323e6640 1 -- 172.21.15.79:0/4100124162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 msgr2=0x55e9f8b562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.810+0000 7ff8323e6640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.810+0000 7ff8323e6640 1 -- 172.21.15.79:0/4100124162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 msgr2=0x55e9f8b51290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:39.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.810+0000 7ff8323e6640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b51290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.810+0000 7ff8323e6640 1 -- 172.21.15.79:0/4100124162 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f8b51ac0 con 0x55e9f8b44700 2024-04-22T06:46:39.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.813+0000 7ff8323e6640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b55d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8240049d0 tx=0x7ff824003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=325b07c8234c4ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:39.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.813+0000 7ff815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:46:39.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.818+0000 7ff815ffb640 1 -- 172.21.15.79:0/4100124162 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82404c070 con 0x55e9f8b44700 2024-04-22T06:46:39.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.818+0000 7ff815ffb640 1 -- 172.21.15.79:0/4100124162 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7ff824034c60 con 0x55e9f8b44700 2024-04-22T06:46:39.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.818+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff824004030 con 0x55e9f8b44700 2024-04-22T06:46:39.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.823+0000 7ff815ffb640 1 -- 172.21.15.79:0/4100124162 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff824069050 con 0x55e9f8b44700 2024-04-22T06:46:39.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.823+0000 7ff83723e740 10 client.25550 fetch_fsmap learned FSMap version 31 2024-04-22T06:46:39.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.823+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e9f8bc3d00 con 0x55e9f8b44700 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.825+0000 7ff815ffb640 1 -- 172.21.15.79:0/4100124162 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff824060070 con 0x55e9f8b44700 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.825+0000 7ff815ffb640 1 -- 172.21.15.79:0/4100124162 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff82403f440 con 0x55e9f8b44700 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.826+0000 7ff83723e740 10 client.25550 fetch_fsmap finished waiting for FSMap version 31 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.826+0000 7ff83723e740 10 client.25550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.826+0000 7ff83723e740 20 client.25550 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.826+0000 7ff83723e740 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8bc4240 0x55e9f8bc6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.826+0000 7ff83723e740 4 client.25550 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:39.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.826+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7ff824004030 con 0x55e9f8bc4240 2024-04-22T06:46:39.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.828+0000 7ff831be5640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8bc4240 0x55e9f8bc6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:39.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.838+0000 7ff815ffb640 10 client.25550 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:39.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.838+0000 7ff831be5640 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8bc4240 0x55e9f8bc6660 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=793d3013db4be663 server_cookie=a9375cc63e7cddff in_seq=0 out_seq=0 2024-04-22T06:46:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.844+0000 7ff815ffb640 1 -- 172.21.15.79:0/4100124162 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7ff824004030 con 0x55e9f8bc4240 2024-04-22T06:46:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.844+0000 7ff815ffb640 10 client.25550 handle_command_reply: tid=0 2024-04-22T06:46:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.844+0000 7ff83723e740 1 client.25550 shutdown 2024-04-22T06:46:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 msgr2=0x55e9f8b55d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b55d90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8240049d0 tx=0x7ff824003ba0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 shutdown_connections 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e9f8bc4240 0x55e9f8bc6660 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e9f8b4dc00 0x55e9f8b51290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9f8b45020 0x55e9f8b562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 --2- 172.21.15.79:0/4100124162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e9f8b44700 0x55e9f8b55d90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 >> 172.21.15.79:0/4100124162 conn(0x55e9f8b3ee30 msgr2=0x55e9f8b3d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.845+0000 7ff815ffb640 0 client.25550 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.846+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 shutdown_connections 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.846+0000 7ff83723e740 1 -- 172.21.15.79:0/4100124162 wait complete. 2024-04-22T06:46:39.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.846+0000 7ff83723e740 20 client.25550 trim_cache size 0 max 16384 2024-04-22T06:46:39.850 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:39.850 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:46:39.850 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:46:39.850 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:39.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.847+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff8180a3f50 msgr2=0x7ff8180a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.847+0000 7ff8333e8640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff8180a3f50 0x7ff8180a6410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55e9f8b3ffc0 tx=0x7ff81c008040 comp rx=0 tx=0).stop 2024-04-22T06:46:39.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 msgr2=0x7ff82c07f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:39.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c07f840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff828009fd0 tx=0x7ff82800b8e0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 shutdown_connections 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff8180a3f50 0x7ff8180a6410 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff82c10ecb0 0x7ff82c07f840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff82c10e390 0x7ff82c07f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 --2- 172.21.15.79:0/1200622905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff82c10c690 0x7ff82c07edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.848+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 >> 172.21.15.79:0/1200622905 conn(0x7ff82c06ced0 msgr2=0x7ff82c072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.849+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 shutdown_connections 2024-04-22T06:46:39.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:39.849+0000 7ff8333e8640 1 -- 172.21.15.79:0/1200622905 wait complete. 2024-04-22T06:46:39.897 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:46:39.897 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:39.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:39 smithi193 podman[66751]: 2024-04-22 06:46:39.349208101 +0000 UTC m=+0.023488049 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:46:39.952 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:39 smithi190 podman[65796]: 2024-04-22 06:46:39.425025009 +0000 UTC m=+0.037450024 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.144+0000 7f520fbc7640 1 -- 172.21.15.79:0/2831056608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5208110550 msgr2=0x7f5208110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.144+0000 7f520fbc7640 1 --2- 172.21.15.79:0/2831056608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5208110550 0x7f5208110930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5200009ca0 tx=0x7f520002c1c0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.145+0000 7f520fbc7640 1 -- 172.21.15.79:0/2831056608 shutdown_connections 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.145+0000 7f520fbc7640 1 --2- 172.21.15.79:0/2831056608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520810eb40 0x7f520811d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.145+0000 7f520fbc7640 1 --2- 172.21.15.79:0/2831056608 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f520810e0c0 0x7f520810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.145+0000 7f520fbc7640 1 --2- 172.21.15.79:0/2831056608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5208110550 0x7f5208110930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.144 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.145+0000 7f520fbc7640 1 -- 172.21.15.79:0/2831056608 >> 172.21.15.79:0/2831056608 conn(0x7f520806ce80 msgr2=0x7f520806d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:40.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.151+0000 7f520fbc7640 1 -- 172.21.15.79:0/2831056608 shutdown_connections 2024-04-22T06:46:40.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.152+0000 7f520fbc7640 1 -- 172.21.15.79:0/2831056608 wait complete. 2024-04-22T06:46:40.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.153+0000 7f520fbc7640 1 Processor -- start 2024-04-22T06:46:40.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 -- start start 2024-04-22T06:46:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520810e0c0 0x7f520811ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f520810eb40 0x7f5208115f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 0x7f5208116440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5208120ea0 con 0x7f5208110550 2024-04-22T06:46:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5208120d20 con 0x7f520810e0c0 2024-04-22T06:46:40.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.154+0000 7f520fbc7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5208121020 con 0x7f520810eb40 2024-04-22T06:46:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.158+0000 7f520e164640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 0x7f5208116440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.158+0000 7f520e164640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 0x7f5208116440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58920/0 (socket says 172.21.15.79:58920) 2024-04-22T06:46:40.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.158+0000 7f520e164640 1 -- 172.21.15.79:0/4094362163 learned_addr learned my addr 172.21.15.79:0/4094362163 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:40.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.158+0000 7f520d963640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520810e0c0 0x7f520811ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:40.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.158+0000 7f520d162640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f520810eb40 0x7f5208115f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:40.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.161+0000 7f520e164640 1 -- 172.21.15.79:0/4094362163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f520810eb40 msgr2=0x7f5208115f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:40.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.161+0000 7f520e164640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f520810eb40 0x7f5208115f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.161+0000 7f520e164640 1 -- 172.21.15.79:0/4094362163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520810e0c0 msgr2=0x7f520811ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:40.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.161+0000 7f520e164640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520810e0c0 0x7f520811ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.161+0000 7f520e164640 1 -- 172.21.15.79:0/4094362163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5208116a80 con 0x7f5208110550 2024-04-22T06:46:40.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.166+0000 7f520e164640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 0x7f5208116440 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f520400e970 tx=0x7f520400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:40.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.182+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520400ccc0 con 0x7f5208110550 2024-04-22T06:46:40.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.182+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f520400ce60 con 0x7f5208110550 2024-04-22T06:46:40.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.182+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204010430 con 0x7f5208110550 2024-04-22T06:46:40.181 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.183+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520811a950 con 0x7f5208110550 2024-04-22T06:46:40.182 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.183+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520811ae50 con 0x7f5208110550 2024-04-22T06:46:40.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.193+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5208116d10 con 0x7f5208110550 2024-04-22T06:46:40.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.200+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5204004f20 con 0x7f5208110550 2024-04-22T06:46:40.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.202+0000 7f51feffd640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f51f40a3ea0 0x7f51f40a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:40.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.202+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8641+0+0 (secure 0 0 0) 0x7f52040d0d40 con 0x7f5208110550 2024-04-22T06:46:40.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.205+0000 7f520d963640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f51f40a3ea0 0x7f51f40a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:40.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.213+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f520409d3e0 con 0x7f5208110550 2024-04-22T06:46:40.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.217+0000 7f520d963640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f51f40a3ea0 0x7f51f40a6360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5200032500 tx=0x7f5200009840 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:40.252 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:39 smithi193 podman[66751]: 2024-04-22 06:46:39.999658612 +0000 UTC m=+0.673938542 container create 1da2865844d1d4a696b6bf4c5b5585a01d42d09aa5ab98368a242ed9564703f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.305+0000 7f913d7fa640 1 -- 172.21.15.79:0/1673346037 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: -125 ) ==== 8+0+591 (crc 0 0 0) 0x7f9144005d10 con 0x55c68488eff0 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.305+0000 7f913d7fa640 10 client.16293 handle_command_reply: tid=0 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.305+0000 7f915e995740 1 client.16293 shutdown 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.305+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 msgr2=0x55c684816dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.305+0000 7f915e995740 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 0x55c684816dc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9150000c00 tx=0x7f9150009e70 comp rx=0 tx=0).stop 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 shutdown_connections 2024-04-22T06:46:40.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f915e995740 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c68488eff0 0x55c684891410 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f915e995740 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c684817300 0x55c684808c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f915e995740 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c6848062e0 0x55c684816dc0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f915e995740 1 --2- 172.21.15.79:0/1673346037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6848059c0 0x55c684880ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 >> 172.21.15.79:0/1673346037 conn(0x55c6848000f0 msgr2=0x55c684801db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:40.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.306+0000 7f913d7fa640 0 client.16293 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:40.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.307+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 shutdown_connections 2024-04-22T06:46:40.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.308+0000 7f915e995740 1 -- 172.21.15.79:0/1673346037 wait complete. 2024-04-22T06:46:40.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.308+0000 7f915e995740 20 client.16293 trim_cache size 0 max 16384 2024-04-22T06:46:40.308 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:40.308 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 32, 2024-04-22T06:46:40.308 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:46:40.308 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 10, 2024-04-22T06:46:40.308 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "ffd2eb3c": { 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 10, 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "CANCELED", 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 79.3, 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": true, 2024-04-22T06:46:40.309 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCING", 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 11.0 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:40.310 INFO:teuthology.orchestra.run.smithi079.stderr:Error ECANCELED: 2024-04-22T06:46:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-22T06:46:40.315 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 10).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-04-22T06:46:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: pgmap v72: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 595 MiB/s wr, 904 op/s 2024-04-22T06:46:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: daemon mds.f restarted 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:boot 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,1=e=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:40 smithi190 podman[65796]: 2024-04-22 06:46:40.288551126 +0000 UTC m=+0.900976135 container create 17390753be9b4590b8249ba47681bd45deb4776fcdbfadcc7e5913db8fa40eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:46:40.557 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:40 smithi190 podman[65796]: 2024-04-22 06:46:40.558672551 +0000 UTC m=+1.171097553 container init 17390753be9b4590b8249ba47681bd45deb4776fcdbfadcc7e5913db8fa40eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:46:40.689 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:40 smithi193 podman[66751]: 2024-04-22 06:46:40.254776607 +0000 UTC m=+0.929056538 container init 1da2865844d1d4a696b6bf4c5b5585a01d42d09aa5ab98368a242ed9564703f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:40 smithi193 podman[66751]: 2024-04-22 06:46:40.258702251 +0000 UTC m=+0.932982186 container start 1da2865844d1d4a696b6bf4c5b5585a01d42d09aa5ab98368a242ed9564703f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mds[66770]: -- 172.21.15.193:0/2856703944 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654546eba40 con 0x5654546bec00 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: starting mds.f at 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:40 smithi193 bash[66751]: 1da2865844d1d4a696b6bf4c5b5585a01d42d09aa5ab98368a242ed9564703f0 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:46:40 smithi193 systemd[1]: Started Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: pgmap v72: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 595 MiB/s wr, 904 op/s 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: daemon mds.f restarted 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:46:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:46:40.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:boot 2024-04-22T06:46:40.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:46:40.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,1=e=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:46:40.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:46:40.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:40 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:46:40.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: pgmap v72: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 595 MiB/s wr, 904 op/s 2024-04-22T06:46:40.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: daemon mds.f restarted 2024-04-22T06:46:40.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:46:40.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:46:40.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:boot 2024-04-22T06:46:40.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:46:40.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,1=e=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:46:40.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:46:40.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:40 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:46:40.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.908+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f520806e120 con 0x7f5208110550 2024-04-22T06:46:40.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.910+0000 7f51feffd640 1 -- 172.21.15.79:0/4094362163 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11855 (secure 0 0 0) 0x7f520409cce0 con 0x7f5208110550 2024-04-22T06:46:40.909 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:46:40.909 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":25552,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":25559,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:46:40.667104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25265,"mds_2":25261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":33,"state":"up:resolve","state_seq":19,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467]},"id":1}]} 2024-04-22T06:46:40.909 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 35 2024-04-22T06:46:40.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f51f40a3ea0 msgr2=0x7f51f40a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:40.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f51f40a3ea0 0x7f51f40a6360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5200032500 tx=0x7f5200009840 comp rx=0 tx=0).stop 2024-04-22T06:46:40.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 msgr2=0x7f5208116440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:40.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 0x7f5208116440 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f520400e970 tx=0x7f520400ed70 comp rx=0 tx=0).stop 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 shutdown_connections 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5208110550 0x7f5208116440 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f520810eb40 0x7f5208115f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f51f40a3ea0 0x7f51f40a6360 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 --2- 172.21.15.79:0/4094362163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f520810e0c0 0x7f520811ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.916+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 >> 172.21.15.79:0/4094362163 conn(0x7f520806ce80 msgr2=0x7f5208071360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.917+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 shutdown_connections 2024-04-22T06:46:40.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:40.917+0000 7f520fbc7640 1 -- 172.21.15.79:0/4094362163 wait complete. 2024-04-22T06:46:40.940 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:46:40.952 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:40 smithi190 podman[65796]: 2024-04-22 06:46:40.563109996 +0000 UTC m=+1.175535000 container start 17390753be9b4590b8249ba47681bd45deb4776fcdbfadcc7e5913db8fa40eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T06:46:40.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: starting mds.e at 2024-04-22T06:46:40.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:40 smithi190 bash[65796]: 17390753be9b4590b8249ba47681bd45deb4776fcdbfadcc7e5913db8fa40eea 2024-04-22T06:46:40.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:46:40 smithi190 systemd[1]: Started Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: daemon mds.e restarted 2024-04-22T06:46:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:46:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:resolve 2024-04-22T06:46:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:boot 2024-04-22T06:46:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:46:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:resolve} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:46:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:46:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:41 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4094362163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:41.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.396+0000 7feca6d96640 1 -- 172.21.15.79:0/771820861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca01071f0 msgr2=0x7feca010edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.396+0000 7feca6d96640 1 --2- 172.21.15.79:0/771820861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca01071f0 0x7feca010edc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fec98009ca0 tx=0x7fec9802c120 comp rx=0 tx=0).stop 2024-04-22T06:46:41.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.397+0000 7feca6d96640 1 -- 172.21.15.79:0/771820861 shutdown_connections 2024-04-22T06:46:41.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.397+0000 7feca6d96640 1 --2- 172.21.15.79:0/771820861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feca0102ad0 0x7feca0106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.397+0000 7feca6d96640 1 --2- 172.21.15.79:0/771820861 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca01071f0 0x7feca010edc0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.397+0000 7feca6d96640 1 --2- 172.21.15.79:0/771820861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feca0102120 0x7feca0102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.397+0000 7feca6d96640 1 -- 172.21.15.79:0/771820861 >> 172.21.15.79:0/771820861 conn(0x7feca00fd9c0 msgr2=0x7feca00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:41.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.420+0000 7feca6d96640 1 -- 172.21.15.79:0/771820861 shutdown_connections 2024-04-22T06:46:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.420+0000 7feca6d96640 1 -- 172.21.15.79:0/771820861 wait complete. 2024-04-22T06:46:41.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.421+0000 7feca6d96640 1 Processor -- start 2024-04-22T06:46:41.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 -- start start 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 0x7feca019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feca01071f0 0x7feca010e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feca010e5f0 0x7feca010eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7feca01121b0 con 0x7feca0102120 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feca0112030 con 0x7feca01071f0 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.429+0000 7feca6d96640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feca0112330 con 0x7feca010e5f0 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.430+0000 7feca4b32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 0x7feca019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.430+0000 7feca4b32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 0x7feca019f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58936/0 (socket says 172.21.15.79:58936) 2024-04-22T06:46:41.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.430+0000 7feca4b32640 1 -- 172.21.15.79:0/1662750300 learned_addr learned my addr 172.21.15.79:0/1662750300 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.439+0000 7feca5333640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feca010e5f0 0x7feca010eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.439+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feca01071f0 0x7feca010e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.441+0000 7feca4b32640 1 -- 172.21.15.79:0/1662750300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feca010e5f0 msgr2=0x7feca010eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.441+0000 7feca4b32640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feca010e5f0 0x7feca010eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.441+0000 7feca4b32640 1 -- 172.21.15.79:0/1662750300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feca01071f0 msgr2=0x7feca010e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.441+0000 7feca4b32640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feca01071f0 0x7feca010e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.441+0000 7feca4b32640 1 -- 172.21.15.79:0/1662750300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca019aea0 con 0x7feca0102120 2024-04-22T06:46:41.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.443+0000 7feca4b32640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 0x7feca019f450 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fec9000c890 tx=0x7fec9000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:41.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.453+0000 7fec9dffb640 1 -- 172.21.15.79:0/1662750300 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90004220 con 0x7feca0102120 2024-04-22T06:46:41.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.453+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca019b190 con 0x7feca0102120 2024-04-22T06:46:41.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.454+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca019b6e0 con 0x7feca0102120 2024-04-22T06:46:41.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.455+0000 7fec9dffb640 1 -- 172.21.15.79:0/1662750300 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec900043c0 con 0x7feca0102120 2024-04-22T06:46:41.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.455+0000 7fec9dffb640 1 -- 172.21.15.79:0/1662750300 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90004a00 con 0x7feca0102120 2024-04-22T06:46:41.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.462+0000 7fec9dffb640 1 -- 172.21.15.79:0/1662750300 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fec90004ba0 con 0x7feca0102120 2024-04-22T06:46:41.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.463+0000 7fec9dffb640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fec880a3f20 0x7fec880a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.464+0000 7fec9dffb640 1 -- 172.21.15.79:0/1662750300 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fec900d04d0 con 0x7feca0102120 2024-04-22T06:46:41.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.469+0000 7feca9984740 1 Processor -- start 2024-04-22T06:46:41.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.471+0000 7feca9984740 1 -- start start 2024-04-22T06:46:41.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.471+0000 7feca9984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 0x55e7cd08eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.471+0000 7feca9984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd097630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca9984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd09a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca9984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e7ccff31f0 con 0x55e7cd08f080 2024-04-22T06:46:41.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca9984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e7ccf904b0 con 0x55e7cd097c00 2024-04-22T06:46:41.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca9984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e7ccfef1b0 con 0x55e7cd08e6d0 2024-04-22T06:46:41.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca4b32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 0x55e7cd08eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca4b32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 0x55e7cd08eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:41806/0 (socket says 172.21.15.79:41806) 2024-04-22T06:46:41.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.472+0000 7feca4b32640 1 -- 172.21.15.79:0/1030201552 learned_addr learned my addr 172.21.15.79:0/1030201552 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:41.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.474+0000 7feca4b32640 1 -- 172.21.15.79:0/1030201552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 msgr2=0x55e7cd09a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.474+0000 7feca4b32640 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd09a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.474+0000 7feca4b32640 1 -- 172.21.15.79:0/1030201552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 msgr2=0x55e7cd097630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:41.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.474+0000 7feca4b32640 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd097630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.474+0000 7feca4b32640 1 -- 172.21.15.79:0/1030201552 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ccf821b0 con 0x55e7cd08e6d0 2024-04-22T06:46:41.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.476+0000 7feca4b32640 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 0x55e7cd08eab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec90098cc0 tx=0x7fec9009c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c09a29ce92d4bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:41.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.484+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fec880a3f20 0x7fec880a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.484+0000 7fec86ffd640 1 -- 172.21.15.79:0/1030201552 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec900b5070 con 0x55e7cd08e6d0 2024-04-22T06:46:41.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.485+0000 7fec86ffd640 1 -- 172.21.15.79:0/1030201552 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec900a06a0 con 0x55e7cd08e6d0 2024-04-22T06:46:41.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 -- 172.21.15.79:0/1030201552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 msgr2=0x55e7cd08eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 0x55e7cd08eab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec90098cc0 tx=0x7fec9009c9f0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 -- 172.21.15.79:0/1030201552 shutdown_connections 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd09a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd097630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 --2- 172.21.15.79:0/1030201552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08e6d0 0x55e7cd08eab0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 -- 172.21.15.79:0/1030201552 >> 172.21.15.79:0/1030201552 conn(0x55e7cd088da0 msgr2=0x55e7cd089180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:41.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.486+0000 7feca9984740 1 -- 172.21.15.79:0/1030201552 shutdown_connections 2024-04-22T06:46:41.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.487+0000 7feca9984740 1 -- 172.21.15.79:0/1030201552 wait complete. 2024-04-22T06:46:41.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.487+0000 7feca9984740 1 -- start start 2024-04-22T06:46:41.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.488+0000 7feca9984740 1 Processor -- start 2024-04-22T06:46:41.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.488+0000 7feca9984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08f080 0x55e7cd097260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.488+0000 7feca9984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd094550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.488+0000 7feca9984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd094f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.489+0000 7feca9984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e7cd09abc0 con 0x55e7cd094a90 2024-04-22T06:46:41.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.489+0000 7feca9984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e7ccfac500 con 0x55e7cd097c00 2024-04-22T06:46:41.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.489+0000 7feca9984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e7cd09ad40 con 0x55e7cd08f080 2024-04-22T06:46:41.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.489+0000 7feca9984740 10 client.0 _ll_register_callbacks cb 0x55e7ccf7ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:41.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.490+0000 7feca5333640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd094f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.490+0000 7feca5333640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd094f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58946/0 (socket says 172.21.15.79:58946) 2024-04-22T06:46:41.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.490+0000 7feca5333640 1 -- 172.21.15.79:0/2295080319 learned_addr learned my addr 172.21.15.79:0/2295080319 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:41.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.490+0000 7fec9ffff640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd094550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.491+0000 7feca4b32640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08f080 0x55e7cd097260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.491+0000 7feca5333640 1 -- 172.21.15.79:0/2295080319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08f080 msgr2=0x55e7cd097260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.491+0000 7feca5333640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08f080 0x55e7cd097260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.491+0000 7feca5333640 1 -- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 msgr2=0x55e7cd094550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.491+0000 7feca5333640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd094550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.491+0000 7feca5333640 1 -- 172.21.15.79:0/2295080319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ccd053f0 con 0x55e7cd094a90 2024-04-22T06:46:41.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.493+0000 7feca5333640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd094f10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fec98009ca0 tx=0x7fec980327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8b03e02f2c11cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:41.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.498+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fec880a3f20 0x7fec880a63e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55e7cd095b20 tx=0x7fec9400a0d0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:41.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.523+0000 7fec7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:41.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.523+0000 7fec7ffff640 1 -- 172.21.15.79:0/2295080319 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec98043020 con 0x55e7cd094a90 2024-04-22T06:46:41.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.523+0000 7fec7ffff640 1 -- 172.21.15.79:0/2295080319 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fec98003970 con 0x55e7cd094a90 2024-04-22T06:46:41.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.526+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e7ccf92930 con 0x55e7cd094a90 2024-04-22T06:46:41.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.531+0000 7fec7ffff640 1 -- 172.21.15.79:0/2295080319 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec98039040 con 0x55e7cd094a90 2024-04-22T06:46:41.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.532+0000 7feca9984740 10 client.16332 fetch_fsmap learned FSMap version 35 2024-04-22T06:46:41.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.532+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7ccdcf7d0 con 0x55e7cd094a90 2024-04-22T06:46:41.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.538+0000 7fec7ffff640 1 -- 172.21.15.79:0/2295080319 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e7ccf92930 con 0x55e7cd094a90 2024-04-22T06:46:41.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.538+0000 7fec7ffff640 1 -- 172.21.15.79:0/2295080319 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fec9803f470 con 0x55e7cd094a90 2024-04-22T06:46:41.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.541+0000 7feca9984740 10 client.16332 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T06:46:41.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.541+0000 7feca9984740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:41.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.541+0000 7feca9984740 20 client.16332 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:41.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.541+0000 7feca9984740 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd0a8090 0x55e7cd0aa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.541+0000 7feca9984740 4 client.16332 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:41.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.541+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7cd000c10 con 0x55e7cd0a8090 2024-04-22T06:46:41.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.546+0000 7feca4b32640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd0a8090 0x55e7cd0aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.553+0000 7fec7ffff640 10 client.16332 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:41.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.553+0000 7feca4b32640 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd0a8090 0x55e7cd0aa480 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de4f6c0c674315a3 server_cookie=b7459b0c1878bea6 in_seq=0 out_seq=0 2024-04-22T06:46:41.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.564+0000 7fec7ffff640 1 -- 172.21.15.79:0/2295080319 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7cd000c10 con 0x55e7cd0a8090 2024-04-22T06:46:41.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.564+0000 7fec7ffff640 10 client.16332 handle_command_reply: tid=0 2024-04-22T06:46:41.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.564+0000 7feca9984740 1 client.16332 shutdown 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.564+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 msgr2=0x55e7cd094f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.564+0000 7feca9984740 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd094f10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fec98009ca0 tx=0x7fec980327b0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 shutdown_connections 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd0a8090 0x55e7cd0aa480 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd094550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd094f10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 --2- 172.21.15.79:0/2295080319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd08f080 0x55e7cd097260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 >> 172.21.15.79:0/2295080319 conn(0x55e7cd088da0 msgr2=0x55e7cd087590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:41.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7fec7ffff640 0 client.16332 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:41.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.565+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 shutdown_connections 2024-04-22T06:46:41.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.568+0000 7feca9984740 1 -- 172.21.15.79:0/2295080319 wait complete. 2024-04-22T06:46:41.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.568+0000 7feca9984740 20 client.16332 trim_cache size 0 max 16384 2024-04-22T06:46:41.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.621+0000 7feca9984740 1 Processor -- start 2024-04-22T06:46:41.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.621+0000 7feca9984740 1 -- start start 2024-04-22T06:46:41.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.622+0000 7feca9984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd08f080 0x55e7cd0a3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.622+0000 7feca9984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd093400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.622+0000 7feca9984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd097c00 0x55e7cd096d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.622+0000 7feca9984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00ec50 con 0x55e7cd094a90 2024-04-22T06:46:41.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.622+0000 7feca9984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00f320 con 0x55e7cd08f080 2024-04-22T06:46:41.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.622+0000 7feca9984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00caa0 con 0x55e7cd097c00 2024-04-22T06:46:41.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.625+0000 7fec9ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd093400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.625+0000 7feca4b32640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd08f080 0x55e7cd0a3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.625+0000 7fec9ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd093400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58956/0 (socket says 172.21.15.79:58956) 2024-04-22T06:46:41.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.625+0000 7fec9ffff640 1 -- 172.21.15.79:0/2985569197 learned_addr learned my addr 172.21.15.79:0/2985569197 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:41.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.626+0000 7feca5333640 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd097c00 0x55e7cd096d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.628+0000 7fec9ffff640 1 -- 172.21.15.79:0/2985569197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd097c00 msgr2=0x55e7cd096d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.628+0000 7fec9ffff640 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd097c00 0x55e7cd096d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.628+0000 7fec9ffff640 1 -- 172.21.15.79:0/2985569197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd08f080 msgr2=0x55e7cd0a3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.628+0000 7fec9ffff640 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd08f080 0x55e7cd0a3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.628+0000 7fec9ffff640 1 -- 172.21.15.79:0/2985569197 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7cd0aa7f0 con 0x55e7cd094a90 2024-04-22T06:46:41.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.630+0000 7fec9ffff640 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd093400 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fec9400efd0 tx=0x7fec9400d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c83111a6fbd53b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:41.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.630+0000 7fec867fc640 1 -- 172.21.15.79:0/2985569197 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9400f040 con 0x55e7cd094a90 2024-04-22T06:46:41.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.631+0000 7fec867fc640 1 -- 172.21.15.79:0/2985569197 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec94026070 con 0x55e7cd094a90 2024-04-22T06:46:41.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.632+0000 7feca9984740 1 -- 172.21.15.79:0/2985569197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 msgr2=0x55e7cd093400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.632+0000 7feca9984740 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd093400 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fec9400efd0 tx=0x7fec9400d000 comp rx=0 tx=0).stop 2024-04-22T06:46:41.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.634+0000 7feca9984740 1 -- 172.21.15.79:0/2985569197 shutdown_connections 2024-04-22T06:46:41.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.634+0000 7feca9984740 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd097c00 0x55e7cd096d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.634+0000 7feca9984740 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd094a90 0x55e7cd093400 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.634+0000 7feca9984740 1 --2- 172.21.15.79:0/2985569197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd08f080 0x55e7cd0a3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.634+0000 7feca9984740 1 -- 172.21.15.79:0/2985569197 >> 172.21.15.79:0/2985569197 conn(0x55e7cd088da0 msgr2=0x55e7cd092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:41.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.639+0000 7feca9984740 1 -- 172.21.15.79:0/2985569197 shutdown_connections 2024-04-22T06:46:41.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.640+0000 7feca9984740 1 -- 172.21.15.79:0/2985569197 wait complete. 2024-04-22T06:46:41.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.640+0000 7feca9984740 1 -- start start 2024-04-22T06:46:41.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.640+0000 7feca9984740 1 Processor -- start 2024-04-22T06:46:41.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd0a0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd094a90 0x55e7cd098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00d110 con 0x55e7cd08f080 2024-04-22T06:46:41.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00e580 con 0x55e7cd097c00 2024-04-22T06:46:41.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c00d7e0 con 0x55e7cd094a90 2024-04-22T06:46:41.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.641+0000 7feca9984740 10 client.0 _ll_register_callbacks cb 0x7fec98009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:41.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.642+0000 7feca4b32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd0a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.642+0000 7feca4b32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd0a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58962/0 (socket says 172.21.15.79:58962) 2024-04-22T06:46:41.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.642+0000 7feca4b32640 1 -- 172.21.15.79:0/1649129280 learned_addr learned my addr 172.21.15.79:0/1649129280 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:41.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.643+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd094a90 0x55e7cd098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.643+0000 7feca5333640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.644+0000 7fec9ffff640 1 -- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 msgr2=0x55e7cd098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.644+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.644+0000 7fec9ffff640 1 -- 172.21.15.79:0/1649129280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 msgr2=0x55e7cd0a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:41.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.644+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd0a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.644+0000 7fec9ffff640 1 -- 172.21.15.79:0/1649129280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7cd099510 con 0x55e7cd094a90 2024-04-22T06:46:41.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.645+0000 7fec9ffff640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd094a90 0x55e7cd098760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec94006f20 tx=0x7fec94011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f0bf7513905aa90 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:41.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.645+0000 7fec7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:46:41.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.647+0000 7fec7ffff640 1 -- 172.21.15.79:0/1649129280 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94030070 con 0x55e7cd094a90 2024-04-22T06:46:41.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.648+0000 7fec7ffff640 1 -- 172.21.15.79:0/1649129280 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fec94017d20 con 0x55e7cd094a90 2024-04-22T06:46:41.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.648+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec94026070 con 0x55e7cd094a90 2024-04-22T06:46:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.650+0000 7fec7ffff640 1 -- 172.21.15.79:0/1649129280 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec9403a050 con 0x55e7cd094a90 2024-04-22T06:46:41.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.650+0000 7feca9984740 10 client.25567 fetch_fsmap learned FSMap version 35 2024-04-22T06:46:41.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.650+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7cd10e640 con 0x55e7cd094a90 2024-04-22T06:46:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7fec7ffff640 1 -- 172.21.15.79:0/1649129280 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec94026070 con 0x55e7cd094a90 2024-04-22T06:46:41.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7fec7ffff640 1 -- 172.21.15.79:0/1649129280 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fec9402a410 con 0x55e7cd094a90 2024-04-22T06:46:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7feca9984740 10 client.25567 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T06:46:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7feca9984740 10 client.25567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7feca9984740 20 client.25567 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7feca9984740 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd10ed30 0x55e7cd111150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7feca9984740 4 client.25567 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:41.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.652+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7ccffaa80 con 0x55e7cd10ed30 2024-04-22T06:46:41.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.655+0000 7feca4b32640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd10ed30 0x55e7cd111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:41.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.660+0000 7fec7ffff640 10 client.25567 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:41.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.660+0000 7feca4b32640 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd10ed30 0x55e7cd111150 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c12c7ee59bcf757a server_cookie=11a61dcd1b46205 in_seq=0 out_seq=0 2024-04-22T06:46:41.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.664+0000 7fec7ffff640 1 -- 172.21.15.79:0/1649129280 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7ccffaa80 con 0x55e7cd10ed30 2024-04-22T06:46:41.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.664+0000 7fec7ffff640 10 client.25567 handle_command_reply: tid=0 2024-04-22T06:46:41.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.664+0000 7feca9984740 1 client.25567 shutdown 2024-04-22T06:46:41.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.664+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd094a90 msgr2=0x55e7cd098760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.664+0000 7feca9984740 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd094a90 0x55e7cd098760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec94006f20 tx=0x7fec94011a00 comp rx=0 tx=0).stop 2024-04-22T06:46:41.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 shutdown_connections 2024-04-22T06:46:41.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e7cd10ed30 0x55e7cd111150 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7cd097c00 0x55e7cd098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7cd094a90 0x55e7cd098760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 --2- 172.21.15.79:0/1649129280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e7cd08f080 0x55e7cd0a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 >> 172.21.15.79:0/1649129280 conn(0x55e7cd088da0 msgr2=0x55e7cd08a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7fec7ffff640 0 client.25567 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 shutdown_connections 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 1 -- 172.21.15.79:0/1649129280 wait complete. 2024-04-22T06:46:41.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.667+0000 7feca9984740 20 client.25567 trim_cache size 0 max 16384 2024-04-22T06:46:41.669 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:46:41.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.674+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fec880a3f20 msgr2=0x7fec880a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.674+0000 7feca6d96640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fec880a3f20 0x7fec880a63e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55e7cd095b20 tx=0x7fec9400a0d0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.674+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 msgr2=0x7feca019f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:41.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.674+0000 7feca6d96640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 0x7feca019f450 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fec9000c890 tx=0x7fec9000cd50 comp rx=0 tx=0).stop 2024-04-22T06:46:41.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.676+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 shutdown_connections 2024-04-22T06:46:41.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.676+0000 7feca6d96640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fec880a3f20 0x7fec880a63e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.676+0000 7feca6d96640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feca01071f0 0x7feca010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.676+0000 7feca6d96640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feca010e5f0 0x7feca010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.676+0000 7feca6d96640 1 --2- 172.21.15.79:0/1662750300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feca0102120 0x7feca019f450 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:41.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.676+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 >> 172.21.15.79:0/1662750300 conn(0x7feca00fd9c0 msgr2=0x7feca00fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:41.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.678+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 shutdown_connections 2024-04-22T06:46:41.688 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:41.690+0000 7feca6d96640 1 -- 172.21.15.79:0/1662750300 wait complete. 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: daemon mds.e restarted 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:resolve 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:boot 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:resolve} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:46:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:46:41.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:41.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:41 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4094362163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: daemon mds.e restarted 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:resolve 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:boot 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:resolve} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:46:41.727 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:46:41.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:41.728 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:41 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4094362163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:41.729 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:46:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.129+0000 7f33938a8640 1 -- 172.21.15.79:0/3850344871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 msgr2=0x7f338c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.129+0000 7f33938a8640 1 --2- 172.21.15.79:0/3850344871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c1035b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3388009900 tx=0x7f3388034570 comp rx=0 tx=0).stop 2024-04-22T06:46:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 -- 172.21.15.79:0/3850344871 shutdown_connections 2024-04-22T06:46:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 --2- 172.21.15.79:0/3850344871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338c10c510 0x7f338c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 --2- 172.21.15.79:0/3850344871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c103af0 0x7f338c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 --2- 172.21.15.79:0/3850344871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c1035b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 -- 172.21.15.79:0/3850344871 >> 172.21.15.79:0/3850344871 conn(0x7f338c077f50 msgr2=0x7f338c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 -- 172.21.15.79:0/3850344871 shutdown_connections 2024-04-22T06:46:42.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.130+0000 7f33938a8640 1 -- 172.21.15.79:0/3850344871 wait complete. 2024-04-22T06:46:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.132+0000 7f33938a8640 1 Processor -- start 2024-04-22T06:46:42.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.132+0000 7f33938a8640 1 -- start start 2024-04-22T06:46:42.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.133+0000 7f33938a8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.133+0000 7f33938a8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338c103af0 0x7f338c196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.133+0000 7f33938a8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c10c510 0x7f338c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.133+0000 7f3391644640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.133+0000 7f3391644640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58990/0 (socket says 172.21.15.79:58990) 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f33938a8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f338c110000 con 0x7f338c1031d0 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f33938a8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f338c10fe80 con 0x7f338c103af0 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f3391644640 1 -- 172.21.15.79:0/3177994095 learned_addr learned my addr 172.21.15.79:0/3177994095 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f338c110180 con 0x7f338c10c510 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f3391644640 1 -- 172.21.15.79:0/3177994095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c10c510 msgr2=0x7f338c19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f3391644640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c10c510 0x7f338c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f3391644640 1 -- 172.21.15.79:0/3177994095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338c103af0 msgr2=0x7f338c196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f3391644640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338c103af0 0x7f338c196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.134+0000 7f3391644640 1 -- 172.21.15.79:0/3177994095 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338c19b530 con 0x7f338c1031d0 2024-04-22T06:46:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.135+0000 7f3391644640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c196540 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3388005ab0 tx=0x7f3388003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.135+0000 7f33827fc640 1 -- 172.21.15.79:0/3177994095 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388049070 con 0x7f338c1031d0 2024-04-22T06:46:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.135+0000 7f33827fc640 1 -- 172.21.15.79:0/3177994095 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f338803bb50 con 0x7f338c1031d0 2024-04-22T06:46:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.135+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f338c19b7c0 con 0x7f338c1031d0 2024-04-22T06:46:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.135+0000 7f33827fc640 1 -- 172.21.15.79:0/3177994095 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338803f030 con 0x7f338c1031d0 2024-04-22T06:46:42.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.135+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f338c1a30b0 con 0x7f338c1031d0 2024-04-22T06:46:42.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.139+0000 7f339657b740 1 Processor -- start 2024-04-22T06:46:42.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 -- start start 2024-04-22T06:46:42.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b2e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b37430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b39e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563831a931f0 con 0x563831b2e4d0 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563831a304b0 con 0x563831b2ee80 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f339657b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563831a8f1b0 con 0x563831b37a00 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f33827fc640 1 -- 172.21.15.79:0/3177994095 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f338804c070 con 0x7f338c1031d0 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f3391e45640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b2e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f3391e45640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b2e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59006/0 (socket says 172.21.15.79:59006) 2024-04-22T06:46:42.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.140+0000 7f3391e45640 1 -- 172.21.15.79:0/2464983865 learned_addr learned my addr 172.21.15.79:0/2464983865 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.141+0000 7f3391e45640 1 -- 172.21.15.79:0/2464983865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 msgr2=0x563831b39e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.141+0000 7f3391e45640 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b39e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.141+0000 7f3391e45640 1 -- 172.21.15.79:0/2464983865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 msgr2=0x563831b37430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.141+0000 7f3391e45640 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b37430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.141+0000 7f3391e45640 1 -- 172.21.15.79:0/2464983865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563831a221b0 con 0x563831b2e4d0 2024-04-22T06:46:42.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.141+0000 7f3391e45640 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b2e8b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3384009e20 tx=0x7f338400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9f95776fa0cdd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.142+0000 7f33827fc640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f33780a3da0 0x7f33780a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.142+0000 7f33827fc640 1 -- 172.21.15.79:0/3177994095 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f33880fa3f0 con 0x7f338c1031d0 2024-04-22T06:46:42.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.142+0000 7f33777fe640 1 -- 172.21.15.79:0/2464983865 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384004d20 con 0x563831b2e4d0 2024-04-22T06:46:42.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.142+0000 7f33777fe640 1 -- 172.21.15.79:0/2464983865 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f338400ee70 con 0x563831b2e4d0 2024-04-22T06:46:42.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.142+0000 7f33777fe640 1 -- 172.21.15.79:0/2464983865 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384018890 con 0x563831b2e4d0 2024-04-22T06:46:42.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.143+0000 7f3391644640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f33780a3da0 0x7f33780a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 -- 172.21.15.79:0/2464983865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 msgr2=0x563831b2e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b2e8b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3384009e20 tx=0x7f338400e720 comp rx=0 tx=0).stop 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 -- 172.21.15.79:0/2464983865 shutdown_connections 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b37430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 --2- 172.21.15.79:0/2464983865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b2e8b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.144+0000 7f339657b740 1 -- 172.21.15.79:0/2464983865 >> 172.21.15.79:0/2464983865 conn(0x563831b28ba0 msgr2=0x563831b28f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.145+0000 7f3391644640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f33780a3da0 0x7f33780a6260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f338c19bd50 tx=0x7f33880456b0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.145+0000 7f339657b740 1 -- 172.21.15.79:0/2464983865 shutdown_connections 2024-04-22T06:46:42.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.147+0000 7f339657b740 1 -- 172.21.15.79:0/2464983865 wait complete. 2024-04-22T06:46:42.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.147+0000 7f339657b740 1 -- start start 2024-04-22T06:46:42.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.148+0000 7f339657b740 1 Processor -- start 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b37370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2ee80 0x563831b34700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b37a00 0x563831b34d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563831b3a970 con 0x563831b2e4d0 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563831a4c500 con 0x563831b37a00 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563831b3aaf0 con 0x563831b2ee80 2024-04-22T06:46:42.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f339657b740 10 client.0 _ll_register_callbacks cb 0x563831a1ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f3391e45640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b37370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f3391e45640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b37370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59018/0 (socket says 172.21.15.79:59018) 2024-04-22T06:46:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f3391e45640 1 -- 172.21.15.79:0/2592495398 learned_addr learned my addr 172.21.15.79:0/2592495398 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.149+0000 7f3391e45640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b37a00 0x563831b34d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3391e45640 1 -- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2ee80 msgr2=0x563831b34700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3390e43640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2ee80 0x563831b34700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3391e45640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2ee80 0x563831b34700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3391e45640 1 -- 172.21.15.79:0/2592495398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b37a00 msgr2=0x563831b34d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3391e45640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b37a00 0x563831b34d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3391e45640 1 -- 172.21.15.79:0/2592495398 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638317a53f0 con 0x563831b2e4d0 2024-04-22T06:46:42.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.150+0000 7f3390e43640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2ee80 0x563831b34700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:42.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.152+0000 7f3391e45640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b37370 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3384009c60 tx=0x7f3384018510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcef94c536789f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.152+0000 7f3374ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:42.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.152+0000 7f3374ff9640 1 -- 172.21.15.79:0/2592495398 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384018d90 con 0x563831b2e4d0 2024-04-22T06:46:42.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.153+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f338400ee70 con 0x563831b2e4d0 2024-04-22T06:46:42.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.153+0000 7f3374ff9640 1 -- 172.21.15.79:0/2592495398 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f3384020040 con 0x563831b2e4d0 2024-04-22T06:46:42.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.153+0000 7f3374ff9640 1 -- 172.21.15.79:0/2592495398 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3384005000 con 0x563831b2e4d0 2024-04-22T06:46:42.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.153+0000 7f339657b740 10 client.16350 fetch_fsmap learned FSMap version 35 2024-04-22T06:46:42.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.153+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56383186f7d0 con 0x563831b2e4d0 2024-04-22T06:46:42.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f3374ff9640 1 -- 172.21.15.79:0/2592495398 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f338400ee70 con 0x563831b2e4d0 2024-04-22T06:46:42.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f3374ff9640 1 -- 172.21.15.79:0/2592495398 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f3384023030 con 0x563831b2e4d0 2024-04-22T06:46:42.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f339657b740 10 client.16350 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T06:46:42.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f339657b740 10 client.16350 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:46:42.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f339657b740 20 client.16350 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:42.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f339657b740 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b47f00 0x563831b4a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f339657b740 4 client.16350 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:42.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.154+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563831a32930 con 0x563831b47f00 2024-04-22T06:46:42.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.155+0000 7f3390e43640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b47f00 0x563831b4a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.157+0000 7f3390e43640 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b47f00 0x563831b4a380 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6c746dec6de9835b server_cookie=bcdcf59ba990a21 in_seq=0 out_seq=0 2024-04-22T06:46:42.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.157+0000 7f3374ff9640 10 client.16350 ms_handle_connect on v2:172.21.15.193:6838/1588799940 2024-04-22T06:46:42.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.161+0000 7f3374ff9640 1 -- 172.21.15.79:0/2592495398 <== mds.2 v2:172.21.15.193:6838/1588799940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563831a32930 con 0x563831b47f00 2024-04-22T06:46:42.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.161+0000 7f3374ff9640 10 client.16350 handle_command_reply: tid=0 2024-04-22T06:46:42.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.161+0000 7f339657b740 1 client.16350 shutdown 2024-04-22T06:46:42.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.161+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 msgr2=0x563831b37370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.161+0000 7f339657b740 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b37370 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3384009c60 tx=0x7f3384018510 comp rx=0 tx=0).stop 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.161+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 shutdown_connections 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b47f00 0x563831b4a380 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b37a00 0x563831b34d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2ee80 0x563831b34700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 --2- 172.21.15.79:0/2592495398 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2e4d0 0x563831b37370 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 >> 172.21.15.79:0/2592495398 conn(0x563831b28ba0 msgr2=0x563831b2a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f3374ff9640 0 client.16350 ms_handle_reset on v2:172.21.15.193:6838/1588799940 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 shutdown_connections 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 1 -- 172.21.15.79:0/2592495398 wait complete. 2024-04-22T06:46:42.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.162+0000 7f339657b740 20 client.16350 trim_cache size 0 max 16384 2024-04-22T06:46:42.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.228+0000 7f339657b740 1 Processor -- start 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.239+0000 7f339657b740 1 -- start start 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f339657b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2e4d0 0x563831b43830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f339657b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b346a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f339657b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b37a00 0x563831b36e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f339657b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1abff0 con 0x563831b37a00 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f339657b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1ac6c0 con 0x563831b2ee80 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f339657b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1a9e40 con 0x563831b2e4d0 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f3391e45640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b37a00 0x563831b36e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f3391e45640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b37a00 0x563831b36e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59024/0 (socket says 172.21.15.79:59024) 2024-04-22T06:46:42.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.240+0000 7f3391e45640 1 -- 172.21.15.79:0/4260006813 learned_addr learned my addr 172.21.15.79:0/4260006813 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.245+0000 7f3391e45640 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2e4d0 0x563831b43830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.245+0000 7f3390e43640 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.254+0000 7f3390e43640 1 -- 172.21.15.79:0/4260006813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2e4d0 msgr2=0x563831b43830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.254+0000 7f3390e43640 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2e4d0 0x563831b43830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.254+0000 7f3390e43640 1 -- 172.21.15.79:0/4260006813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b37a00 msgr2=0x563831b36e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.254+0000 7f3390e43640 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b37a00 0x563831b36e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.254+0000 7f3390e43640 1 -- 172.21.15.79:0/4260006813 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563831b29770 con 0x563831b2ee80 2024-04-22T06:46:42.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.266+0000 7f3390e43640 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b346a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x563831b4ac30 tx=0x7f337c005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fecaf95f8463e69 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.273+0000 7f3376ffd640 1 -- 172.21.15.79:0/4260006813 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c01c070 con 0x563831b2ee80 2024-04-22T06:46:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.273+0000 7f3376ffd640 1 -- 172.21.15.79:0/4260006813 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f337c018e10 con 0x563831b2ee80 2024-04-22T06:46:42.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.273+0000 7f3376ffd640 1 -- 172.21.15.79:0/4260006813 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c004a00 con 0x563831b2ee80 2024-04-22T06:46:42.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.278+0000 7f339657b740 1 -- 172.21.15.79:0/4260006813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 msgr2=0x563831b346a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.278+0000 7f339657b740 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b346a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x563831b4ac30 tx=0x7f337c005c70 comp rx=0 tx=0).stop 2024-04-22T06:46:42.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.283+0000 7f339657b740 1 -- 172.21.15.79:0/4260006813 shutdown_connections 2024-04-22T06:46:42.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.283+0000 7f339657b740 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2ee80 0x563831b346a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.283+0000 7f339657b740 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b37a00 0x563831b36e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.283+0000 7f339657b740 1 --2- 172.21.15.79:0/4260006813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b2e4d0 0x563831b43830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.283+0000 7f339657b740 1 -- 172.21.15.79:0/4260006813 >> 172.21.15.79:0/4260006813 conn(0x563831b28ba0 msgr2=0x563831b33f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.283+0000 7f339657b740 1 -- 172.21.15.79:0/4260006813 shutdown_connections 2024-04-22T06:46:42.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.284+0000 7f339657b740 1 -- 172.21.15.79:0/4260006813 wait complete. 2024-04-22T06:46:42.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.285+0000 7f339657b740 1 -- start start 2024-04-22T06:46:42.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.285+0000 7f339657b740 1 Processor -- start 2024-04-22T06:46:42.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.287+0000 7f339657b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2e4d0 0x563831b320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.287+0000 7f339657b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2ee80 0x563831b3fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.288+0000 7f339657b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b40280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f339657b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1aa4b0 con 0x563831b2ee80 2024-04-22T06:46:42.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f3391e45640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b40280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f3391e45640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b40280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:41902/0 (socket says 172.21.15.79:41902) 2024-04-22T06:46:42.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f3391e45640 1 -- 172.21.15.79:0/1003906467 learned_addr learned my addr 172.21.15.79:0/1003906467 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1ab920 con 0x563831b2e4d0 2024-04-22T06:46:42.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1aab80 con 0x563831b37a00 2024-04-22T06:46:42.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.289+0000 7f339657b740 10 client.0 _ll_register_callbacks cb 0x7f3384018860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:42.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.290+0000 7f3391e45640 1 -- 172.21.15.79:0/1003906467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2e4d0 msgr2=0x563831b320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.290+0000 7f3391e45640 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2e4d0 0x563831b320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.290+0000 7f3391e45640 1 -- 172.21.15.79:0/1003906467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2ee80 msgr2=0x563831b3fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.290+0000 7f3391e45640 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2ee80 0x563831b3fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.290+0000 7f3391e45640 1 -- 172.21.15.79:0/1003906467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563831ba9590 con 0x563831b37a00 2024-04-22T06:46:42.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.292+0000 7f3391e45640 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b40280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33840278c0 tx=0x7f338400efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65bb2f70b163f89d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.301+0000 7f3374ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:46:42.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.301+0000 7f3374ff9640 1 -- 172.21.15.79:0/1003906467 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384027a70 con 0x563831b37a00 2024-04-22T06:46:42.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.301+0000 7f3374ff9640 1 -- 172.21.15.79:0/1003906467 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f3384020070 con 0x563831b37a00 2024-04-22T06:46:42.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.302+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f337c018e10 con 0x563831b37a00 2024-04-22T06:46:42.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.302+0000 7f3374ff9640 1 -- 172.21.15.79:0/1003906467 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f33840098e0 con 0x563831b37a00 2024-04-22T06:46:42.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.303+0000 7f339657b740 10 client.25576 fetch_fsmap learned FSMap version 35 2024-04-22T06:46:42.301 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.303+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563831b30d30 con 0x563831b37a00 2024-04-22T06:46:42.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.321+0000 7f3374ff9640 1 -- 172.21.15.79:0/1003906467 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3384003e20 con 0x563831b37a00 2024-04-22T06:46:42.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.321+0000 7f3374ff9640 1 -- 172.21.15.79:0/1003906467 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f3384023040 con 0x563831b37a00 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.323+0000 7f339657b740 10 client.25576 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.323+0000 7f339657b740 10 client.25576 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.323+0000 7f339657b740 20 client.25576 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.323+0000 7f339657b740 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b31440 0x563831bc1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.323+0000 7f339657b740 4 client.25576 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.323+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f337c018e10 con 0x563831b31440 2024-04-22T06:46:42.322 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.324+0000 7f3390e43640 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b31440 0x563831bc1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.325+0000 7f3390e43640 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b31440 0x563831bc1520 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b030ed78f7ac51e5 server_cookie=7c408d8df31a5cbe in_seq=0 out_seq=0 2024-04-22T06:46:42.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.325+0000 7f3374ff9640 10 client.25576 ms_handle_connect on v2:172.21.15.193:6838/1588799940 2024-04-22T06:46:42.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.326+0000 7f3374ff9640 1 -- 172.21.15.79:0/1003906467 <== mds.2 v2:172.21.15.193:6838/1588799940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f337c018e10 con 0x563831b31440 2024-04-22T06:46:42.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.326+0000 7f3374ff9640 10 client.25576 handle_command_reply: tid=0 2024-04-22T06:46:42.324 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.326+0000 7f339657b740 1 client.25576 shutdown 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.338+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 msgr2=0x563831b40280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.338+0000 7f339657b740 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b40280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33840278c0 tx=0x7f338400efd0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 shutdown_connections 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f339657b740 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x563831b31440 0x563831bc1520 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f339657b740 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563831b37a00 0x563831b40280 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f339657b740 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563831b2ee80 0x563831b3fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f339657b740 1 --2- 172.21.15.79:0/1003906467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563831b2e4d0 0x563831b320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 >> 172.21.15.79:0/1003906467 conn(0x563831b28ba0 msgr2=0x563831b29fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.339+0000 7f3374ff9640 0 client.25576 ms_handle_reset on v2:172.21.15.193:6838/1588799940 2024-04-22T06:46:42.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.342+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 shutdown_connections 2024-04-22T06:46:42.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.344+0000 7f339657b740 1 -- 172.21.15.79:0/1003906467 wait complete. 2024-04-22T06:46:42.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.347+0000 7f339657b740 20 client.25576 trim_cache size 0 max 16384 2024-04-22T06:46:42.348 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f33780a3da0 msgr2=0x7f33780a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f33780a3da0 0x7f33780a6260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f338c19bd50 tx=0x7f33880456b0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 msgr2=0x7f338c196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c196540 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3388005ab0 tx=0x7f3388003a70 comp rx=0 tx=0).stop 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 shutdown_connections 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f33780a3da0 0x7f33780a6260 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f338c19bd50 tx=0x7f33880456b0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338c103af0 0x7f338c196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c10c510 0x7f338c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 --2- 172.21.15.79:0/3177994095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f338c1031d0 0x7f338c196540 secure :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3388005ab0 tx=0x7f3388003a70 comp rx=0 tx=0).stop 2024-04-22T06:46:42.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.358+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 >> 172.21.15.79:0/3177994095 conn(0x7f338c077f50 msgr2=0x7f338c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.360+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 shutdown_connections 2024-04-22T06:46:42.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.361+0000 7f33938a8640 1 -- 172.21.15.79:0/3177994095 wait complete. 2024-04-22T06:46:42.431 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:46:42.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:42 smithi079 ceph-mon[27946]: pgmap v75: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 864 MiB/s wr, 1.95k op/s 2024-04-22T06:46:42.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.844+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/2425658895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481099d0 msgr2=0x7fbb4811d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.844+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/2425658895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481099d0 0x7fbb4811d740 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009be0 tx=0x7fbb4002bfa0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.847+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/2425658895 shutdown_connections 2024-04-22T06:46:42.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.847+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/2425658895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481099d0 0x7fbb4811d740 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.847+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/2425658895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb48108f50 0x7fbb481093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.847+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/2425658895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb481105f0 0x7fbb481109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.847+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/2425658895 >> 172.21.15.79:0/2425658895 conn(0x7fbb4806ced0 msgr2=0x7fbb4806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.851+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/2425658895 shutdown_connections 2024-04-22T06:46:42.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.852+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/2425658895 wait complete. 2024-04-22T06:46:42.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.853+0000 7fbb4e4a4640 1 Processor -- start 2024-04-22T06:46:42.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 -- start start 2024-04-22T06:46:42.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb48108f50 0x7fbb481a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 0x7fbb481a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481105f0 0x7fbb481abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbb48120ee0 con 0x7fbb481099d0 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb48120d60 con 0x7fbb48108f50 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.854+0000 7fbb4e4a4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbb48121060 con 0x7fbb481105f0 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.855+0000 7fbb4cca1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 0x7fbb481a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.855+0000 7fbb4cca1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 0x7fbb481a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60264/0 (socket says 172.21.15.79:60264) 2024-04-22T06:46:42.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.855+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1478414222 learned_addr learned my addr 172.21.15.79:0/1478414222 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.856+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1478414222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481105f0 msgr2=0x7fbb481abc50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.856+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481105f0 0x7fbb481abc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.856+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1478414222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb48108f50 msgr2=0x7fbb481a7390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:46:42.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.856+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb48108f50 0x7fbb481a7390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.856+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1478414222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb481ac350 con 0x7fbb481099d0 2024-04-22T06:46:42.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.857+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 0x7fbb481a78d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800d950 tx=0x7fbb3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.857+0000 7fbb3e7fc640 1 -- 172.21.15.79:0/1478414222 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38017070 con 0x7fbb481099d0 2024-04-22T06:46:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.858+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb481ac640 con 0x7fbb481099d0 2024-04-22T06:46:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.858+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb48110f10 con 0x7fbb481099d0 2024-04-22T06:46:42.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.858+0000 7fbb3e7fc640 1 -- 172.21.15.79:0/1478414222 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb38009c30 con 0x7fbb481099d0 2024-04-22T06:46:42.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.858+0000 7fbb3e7fc640 1 -- 172.21.15.79:0/1478414222 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800f620 con 0x7fbb481099d0 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.868+0000 7fbb523cc740 1 Processor -- start 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.870+0000 7fbb523cc740 1 -- start start 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 0x5614f6f5fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f60080 0x5614f6f68630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb3e7fc640 1 -- 172.21.15.79:0/1478414222 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbb38010460 con 0x7fbb481099d0 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f6b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb523cc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5614f6ec41f0 con 0x5614f6f68c00 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb523cc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5614f6e614b0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.871+0000 7fbb523cc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5614f6ec01b0 con 0x5614f6f60080 2024-04-22T06:46:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.872+0000 7fbb4dca3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 0x5614f6f5fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.872+0000 7fbb4dca3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 0x5614f6f5fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:49172/0 (socket says 172.21.15.79:49172) 2024-04-22T06:46:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.872+0000 7fbb4dca3640 1 -- 172.21.15.79:0/3432191010 learned_addr learned my addr 172.21.15.79:0/3432191010 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.873+0000 7fbb3e7fc640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbb340a3f20 0x7fbb340a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.873+0000 7fbb3e7fc640 1 -- 172.21.15.79:0/1478414222 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fbb380d04f0 con 0x7fbb481099d0 2024-04-22T06:46:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.874+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f6b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.874+0000 7fbb4dca3640 1 -- 172.21.15.79:0/3432191010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f60080 msgr2=0x5614f6f68630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.874+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f60080 0x5614f6f68630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.874+0000 7fbb4dca3640 1 -- 172.21.15.79:0/3432191010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 msgr2=0x5614f6f6b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.874+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f6b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.874+0000 7fbb4dca3640 1 -- 172.21.15.79:0/3432191010 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f6e531b0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.887+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbb340a3f20 0x7fbb340a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.887+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 0x5614f6f5fab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb40006010 tx=0x7fbb4002c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c8b2b5f377143f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.892+0000 7fbb337fe640 1 -- 172.21.15.79:0/3432191010 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb400349c0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.892+0000 7fbb337fe640 1 -- 172.21.15.79:0/3432191010 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb40034b60 con 0x5614f6f5f6d0 2024-04-22T06:46:42.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.892+0000 7fbb337fe640 1 -- 172.21.15.79:0/3432191010 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40041420 con 0x5614f6f5f6d0 2024-04-22T06:46:42.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.895+0000 7fbb523cc740 1 -- 172.21.15.79:0/3432191010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 msgr2=0x5614f6f5fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.895+0000 7fbb523cc740 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 0x5614f6f5fab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb40006010 tx=0x7fbb4002c940 comp rx=0 tx=0).stop 2024-04-22T06:46:42.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.902+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbb340a3f20 0x7fbb340a63e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb40048000 tx=0x7fbb40049000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.904+0000 7fbb523cc740 1 -- 172.21.15.79:0/3432191010 shutdown_connections 2024-04-22T06:46:42.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.904+0000 7fbb523cc740 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f6b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.903 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.904+0000 7fbb523cc740 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f60080 0x5614f6f68630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.904+0000 7fbb523cc740 1 --2- 172.21.15.79:0/3432191010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f5f6d0 0x5614f6f5fab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.904+0000 7fbb523cc740 1 -- 172.21.15.79:0/3432191010 >> 172.21.15.79:0/3432191010 conn(0x5614f6f59da0 msgr2=0x5614f6f5a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.915+0000 7fbb523cc740 1 -- 172.21.15.79:0/3432191010 shutdown_connections 2024-04-22T06:46:42.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.916+0000 7fbb523cc740 1 -- 172.21.15.79:0/3432191010 wait complete. 2024-04-22T06:46:42.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.916+0000 7fbb523cc740 1 -- start start 2024-04-22T06:46:42.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.917+0000 7fbb523cc740 1 Processor -- start 2024-04-22T06:46:42.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.917+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f68540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.918+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f60080 0x5614f6f658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.918+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f65ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.918+0000 7fbb523cc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5614f6f6bb70 con 0x5614f6f68c00 2024-04-22T06:46:42.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.918+0000 7fbb523cc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5614f6e7d500 con 0x5614f6f60080 2024-04-22T06:46:42.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.918+0000 7fbb523cc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5614f6f6bcf0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.918+0000 7fbb523cc740 10 client.0 _ll_register_callbacks cb 0x5614f6e4bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:42.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.919+0000 7fbb4d4a2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f68540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.919+0000 7fbb4d4a2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f68540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:45558/0 (socket says 172.21.15.79:45558) 2024-04-22T06:46:42.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.919+0000 7fbb4d4a2640 1 -- 172.21.15.79:0/1313973695 learned_addr learned my addr 172.21.15.79:0/1313973695 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:42.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.919+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f65ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.923+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f60080 0x5614f6f658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.928+0000 7fbb4d4a2640 1 -- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f60080 msgr2=0x5614f6f658d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.928+0000 7fbb4d4a2640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f60080 0x5614f6f658d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.928+0000 7fbb4d4a2640 1 -- 172.21.15.79:0/1313973695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 msgr2=0x5614f6f65ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.928+0000 7fbb4d4a2640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f65ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.928+0000 7fbb4d4a2640 1 -- 172.21.15.79:0/1313973695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f6bd63f0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.931+0000 7fbb4d4a2640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f68540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400cf50 tx=0x7fbb4400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b405a2aaebe14c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.933+0000 7fbb30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:46:42.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.938+0000 7fbb30ff9640 1 -- 172.21.15.79:0/1313973695 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400f650 con 0x5614f6f5f6d0 2024-04-22T06:46:42.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.938+0000 7fbb30ff9640 1 -- 172.21.15.79:0/1313973695 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fbb4400f7f0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.940+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb40034b60 con 0x5614f6f5f6d0 2024-04-22T06:46:42.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.941+0000 7fbb30ff9640 1 -- 172.21.15.79:0/1313973695 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbb4400f9f0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.941+0000 7fbb523cc740 10 client.25582 fetch_fsmap learned FSMap version 36 2024-04-22T06:46:42.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.942+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5614f6ca07d0 con 0x5614f6f5f6d0 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb30ff9640 1 -- 172.21.15.79:0/1313973695 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb40034b60 con 0x5614f6f5f6d0 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb30ff9640 1 -- 172.21.15.79:0/1313973695 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fbb44017400 con 0x5614f6f5f6d0 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb523cc740 10 client.25582 fetch_fsmap finished waiting for FSMap version 36 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb523cc740 10 client.25582 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb523cc740 20 client.25582 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6f78fc0 0x5614f6f7b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb523cc740 4 client.25582 mds_command: new command op to 25265 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:42.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.944+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 --> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614f6e63930 con 0x5614f6f78fc0 2024-04-22T06:46:42.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.952+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6f78fc0 0x5614f6f7b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:42 smithi190 ceph-mon[31421]: pgmap v75: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 864 MiB/s wr, 1.95k op/s 2024-04-22T06:46:42.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:42 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:resolve 2024-04-22T06:46:42.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:42 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:42.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.965+0000 7fbb30ff9640 10 client.25582 ms_handle_connect on v2:172.21.15.190:6836/1635214680 2024-04-22T06:46:42.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.965+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6f78fc0 0x5614f6f7b440 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e1a1c25b7f3aba87 server_cookie=5399b30fd1104efa in_seq=0 out_seq=0 2024-04-22T06:46:42.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.971+0000 7fbb30ff9640 1 -- 172.21.15.79:0/1313973695 <== mds.1 v2:172.21.15.190:6836/1635214680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614f6e63930 con 0x5614f6f78fc0 2024-04-22T06:46:42.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.971+0000 7fbb30ff9640 10 client.25582 handle_command_reply: tid=0 2024-04-22T06:46:42.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.971+0000 7fbb523cc740 1 client.25582 shutdown 2024-04-22T06:46:42.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.974+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 msgr2=0x5614f6f68540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.974+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f68540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400cf50 tx=0x7fbb4400a620 comp rx=0 tx=0).stop 2024-04-22T06:46:42.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 shutdown_connections 2024-04-22T06:46:42.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6f78fc0 0x5614f6f7b440 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f60080 0x5614f6f658d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f68c00 0x5614f6f65ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1313973695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f68540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb30ff9640 0 client.25582 ms_handle_reset on v2:172.21.15.190:6836/1635214680 2024-04-22T06:46:42.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.975+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 >> 172.21.15.79:0/1313973695 conn(0x5614f6f59da0 msgr2=0x5614f6f5baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.976+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 shutdown_connections 2024-04-22T06:46:42.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.976+0000 7fbb523cc740 1 -- 172.21.15.79:0/1313973695 wait complete. 2024-04-22T06:46:42.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:42.976+0000 7fbb523cc740 20 client.25582 trim_cache size 0 max 16384 2024-04-22T06:46:42.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:42 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:resolve 2024-04-22T06:46:42.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:42 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:43.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.015+0000 7fbb523cc740 1 Processor -- start 2024-04-22T06:46:43.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.015+0000 7fbb523cc740 1 -- start start 2024-04-22T06:46:43.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.015+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f74ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.016+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f68000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.016+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6f68540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.016+0000 7fbb523cc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c00eae0 con 0x5614f6f60080 2024-04-22T06:46:43.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.016+0000 7fbb523cc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c00f1b0 con 0x5614f6f68c00 2024-04-22T06:46:43.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.016+0000 7fbb523cc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c00c930 con 0x5614f6f5f6d0 2024-04-22T06:46:43.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.016+0000 7fbb4cca1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f68000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.017+0000 7fbb4cca1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f68000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60290/0 (socket says 172.21.15.79:60290) 2024-04-22T06:46:43.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.017+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1262288342 learned_addr learned my addr 172.21.15.79:0/1262288342 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:43.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.017+0000 7fbb4dca3640 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6f68540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.018+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1262288342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 msgr2=0x5614f6f74ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:43.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.018+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f74ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.018+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1262288342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 msgr2=0x5614f6f68540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:43.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.018+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6f68540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.018+0000 7fbb4cca1640 1 -- 172.21.15.79:0/1262288342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f6f63e20 con 0x5614f6f60080 2024-04-22T06:46:43.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.018+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f68000 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809bde0 tx=0x7fbb380d28c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ba7cb3ebbb92ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:43.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.019+0000 7fbb32ffd640 1 -- 172.21.15.79:0/1262288342 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb380b4030 con 0x5614f6f60080 2024-04-22T06:46:43.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.019+0000 7fbb32ffd640 1 -- 172.21.15.79:0/1262288342 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb38015020 con 0x5614f6f60080 2024-04-22T06:46:43.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.019+0000 7fbb32ffd640 1 -- 172.21.15.79:0/1262288342 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38020040 con 0x5614f6f60080 2024-04-22T06:46:43.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.020+0000 7fbb523cc740 1 -- 172.21.15.79:0/1262288342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 msgr2=0x5614f6f68000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.020+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f68000 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809bde0 tx=0x7fbb380d28c0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 -- 172.21.15.79:0/1262288342 shutdown_connections 2024-04-22T06:46:43.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6f68540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f68000 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 --2- 172.21.15.79:0/1262288342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f74ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 -- 172.21.15.79:0/1262288342 >> 172.21.15.79:0/1262288342 conn(0x5614f6f59da0 msgr2=0x5614f6f60e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:43.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 -- 172.21.15.79:0/1262288342 shutdown_connections 2024-04-22T06:46:43.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.021+0000 7fbb523cc740 1 -- 172.21.15.79:0/1262288342 wait complete. 2024-04-22T06:46:43.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 -- start start 2024-04-22T06:46:43.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 Processor -- start 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f6cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f6d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6fde580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c00cfa0 con 0x5614f6f60080 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c00e410 con 0x5614f6f68c00 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.022+0000 7fbb523cc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c00d670 con 0x5614f6f5f6d0 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.023+0000 7fbb523cc740 10 client.0 _ll_register_callbacks cb 0x7fbb4400cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.023+0000 7fbb4cca1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f6d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.023+0000 7fbb4cca1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f6d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60306/0 (socket says 172.21.15.79:60306) 2024-04-22T06:46:43.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.023+0000 7fbb4cca1640 1 -- 172.21.15.79:0/103692865 learned_addr learned my addr 172.21.15.79:0/103692865 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:43.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.024+0000 7fbb4cca1640 1 -- 172.21.15.79:0/103692865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 msgr2=0x5614f6f6cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:43.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.024+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f6cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.024+0000 7fbb4cca1640 1 -- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 msgr2=0x5614f6fde580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:43.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.024+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6fde580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.024+0000 7fbb4cca1640 1 -- 172.21.15.79:0/103692865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f6fdeac0 con 0x5614f6f60080 2024-04-22T06:46:43.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.026+0000 7fbb4cca1640 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f6d310 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809c010 tx=0x7fbb380cdf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d12a282109009e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:43.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.027+0000 7fbb30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:43.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.027+0000 7fbb30ff9640 1 -- 172.21.15.79:0/103692865 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb380b4070 con 0x5614f6f60080 2024-04-22T06:46:43.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.028+0000 7fbb30ff9640 1 -- 172.21.15.79:0/103692865 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fbb3809b760 con 0x5614f6f60080 2024-04-22T06:46:43.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.028+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb38015020 con 0x5614f6f60080 2024-04-22T06:46:43.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.028+0000 7fbb30ff9640 1 -- 172.21.15.79:0/103692865 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbb38020040 con 0x5614f6f60080 2024-04-22T06:46:43.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.028+0000 7fbb523cc740 10 client.16371 fetch_fsmap learned FSMap version 36 2024-04-22T06:46:43.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.028+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5614f6fdfbf0 con 0x5614f6f60080 2024-04-22T06:46:43.027 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.029+0000 7fbb30ff9640 1 -- 172.21.15.79:0/103692865 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb38015020 con 0x5614f6f60080 2024-04-22T06:46:43.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb30ff9640 1 -- 172.21.15.79:0/103692865 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fbb380abce0 con 0x5614f6f60080 2024-04-22T06:46:43.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb523cc740 10 client.16371 fetch_fsmap finished waiting for FSMap version 36 2024-04-22T06:46:43.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb523cc740 10 client.16371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:46:43.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb523cc740 20 client.16371 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:43.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb523cc740 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6fe0270 0x5614f6fe2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb523cc740 4 client.16371 mds_command: new command op to 25265 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:43.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.030+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 --> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5614f6ed1c10 con 0x5614f6fe0270 2024-04-22T06:46:43.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.033+0000 7fbb4d4a2640 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6fe0270 0x5614f6fe2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.038+0000 7fbb4d4a2640 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6fe0270 0x5614f6fe2690 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15340781c0e208fb server_cookie=a2a21f2cf8d3be05 in_seq=0 out_seq=0 2024-04-22T06:46:43.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.038+0000 7fbb30ff9640 10 client.16371 ms_handle_connect on v2:172.21.15.190:6836/1635214680 2024-04-22T06:46:43.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.039+0000 7fbb30ff9640 1 -- 172.21.15.79:0/103692865 <== mds.1 v2:172.21.15.190:6836/1635214680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5614f6ed1c10 con 0x5614f6fe0270 2024-04-22T06:46:43.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.039+0000 7fbb30ff9640 10 client.16371 handle_command_reply: tid=0 2024-04-22T06:46:43.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.039+0000 7fbb523cc740 1 client.16371 shutdown 2024-04-22T06:46:43.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.040+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 msgr2=0x5614f6f6d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.040+0000 7fbb523cc740 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f6d310 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809c010 tx=0x7fbb380cdf70 comp rx=0 tx=0).stop 2024-04-22T06:46:43.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.041+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 shutdown_connections 2024-04-22T06:46:43.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.041+0000 7fbb523cc740 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5614f6fe0270 0x5614f6fe2690 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.042+0000 7fbb523cc740 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614f6f68c00 0x5614f6fde580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.042+0000 7fbb523cc740 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5614f6f60080 0x5614f6f6d310 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.042+0000 7fbb523cc740 1 --2- 172.21.15.79:0/103692865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614f6f5f6d0 0x5614f6f6cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.042+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 >> 172.21.15.79:0/103692865 conn(0x5614f6f59da0 msgr2=0x5614f6f58590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:43.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.042+0000 7fbb30ff9640 0 client.16371 ms_handle_reset on v2:172.21.15.190:6836/1635214680 2024-04-22T06:46:43.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.042+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 shutdown_connections 2024-04-22T06:46:43.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.043+0000 7fbb523cc740 1 -- 172.21.15.79:0/103692865 wait complete. 2024-04-22T06:46:43.041 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.043+0000 7fbb523cc740 20 client.16371 trim_cache size 0 max 16384 2024-04-22T06:46:43.042 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:46:43.054 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.055+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbb340a3f20 msgr2=0x7fbb340a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.055+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbb340a3f20 0x7fbb340a63e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb40048000 tx=0x7fbb40049000 comp rx=0 tx=0).stop 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.055+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 msgr2=0x7fbb481a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.055+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 0x7fbb481a78d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800d950 tx=0x7fbb3800de10 comp rx=0 tx=0).stop 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 shutdown_connections 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb481105f0 0x7fbb481abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbb481099d0 0x7fbb481a78d0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fbb340a3f20 0x7fbb340a63e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 --2- 172.21.15.79:0/1478414222 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb48108f50 0x7fbb481a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.055 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 >> 172.21.15.79:0/1478414222 conn(0x7fbb4806ced0 msgr2=0x7fbb48070bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:43.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 shutdown_connections 2024-04-22T06:46:43.056 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:43.057+0000 7fbb4e4a4640 1 -- 172.21.15.79:0/1478414222 wait complete. 2024-04-22T06:46:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:42 smithi193 ceph-mon[31815]: pgmap v75: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 864 MiB/s wr, 1.95k op/s 2024-04-22T06:46:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:42 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:resolve 2024-04-22T06:46:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:42 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:44.088 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:46:44.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:46:44.089 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:46:44.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.393+0000 7fb734914640 1 -- 172.21.15.79:0/2335024430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb730108f50 msgr2=0x7fb7301093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.393+0000 7fb734914640 1 --2- 172.21.15.79:0/2335024430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb730108f50 0x7fb7301093b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb720009900 tx=0x7fb72002bd10 comp rx=0 tx=0).stop 2024-04-22T06:46:44.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.395+0000 7fb734914640 1 -- 172.21.15.79:0/2335024430 shutdown_connections 2024-04-22T06:46:44.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.395+0000 7fb734914640 1 --2- 172.21.15.79:0/2335024430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7301099d0 0x7fb73011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.395+0000 7fb734914640 1 --2- 172.21.15.79:0/2335024430 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb730108f50 0x7fb7301093b0 secure :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb720009900 tx=0x7fb72002bd10 comp rx=0 tx=0).stop 2024-04-22T06:46:44.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.395+0000 7fb734914640 1 --2- 172.21.15.79:0/2335024430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301105f0 0x7fb7301109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.395+0000 7fb734914640 1 -- 172.21.15.79:0/2335024430 >> 172.21.15.79:0/2335024430 conn(0x7fb73006ced0 msgr2=0x7fb73006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.404+0000 7fb734914640 1 -- 172.21.15.79:0/2335024430 shutdown_connections 2024-04-22T06:46:44.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.404+0000 7fb734914640 1 -- 172.21.15.79:0/2335024430 wait complete. 2024-04-22T06:46:44.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 Processor -- start 2024-04-22T06:46:44.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 -- start start 2024-04-22T06:46:44.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7301099d0 0x7fb73011d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301167d0 0x7fb730116c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb730117170 0x7fb73011b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb730120f20 con 0x7fb7301099d0 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb730120da0 con 0x7fb730117170 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.406+0000 7fb734914640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301210a0 con 0x7fb7301167d0 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.408+0000 7fb72ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7301099d0 0x7fb73011d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.408+0000 7fb72ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7301099d0 0x7fb73011d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60326/0 (socket says 172.21.15.79:60326) 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.408+0000 7fb72ed9d640 1 -- 172.21.15.79:0/1066457233 learned_addr learned my addr 172.21.15.79:0/1066457233 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:44.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.408+0000 7fb72e59c640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301167d0 0x7fb730116c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.409+0000 7fb72f59e640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb730117170 0x7fb73011b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.409+0000 7fb72e59c640 1 -- 172.21.15.79:0/1066457233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb730117170 msgr2=0x7fb73011b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.409+0000 7fb72e59c640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb730117170 0x7fb73011b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.409+0000 7fb72e59c640 1 -- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7301099d0 msgr2=0x7fb73011d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.409+0000 7fb72e59c640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7301099d0 0x7fb73011d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.409+0000 7fb72e59c640 1 -- 172.21.15.79:0/1066457233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73011b950 con 0x7fb7301167d0 2024-04-22T06:46:44.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.412+0000 7fb72e59c640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301167d0 0x7fb730116c30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb720008000 tx=0x7fb72002db20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.413+0000 7fb717fff640 1 -- 172.21.15.79:0/1066457233 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb720043020 con 0x7fb7301167d0 2024-04-22T06:46:44.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.413+0000 7fb717fff640 1 -- 172.21.15.79:0/1066457233 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7200052b0 con 0x7fb7301167d0 2024-04-22T06:46:44.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.414+0000 7fb717fff640 1 -- 172.21.15.79:0/1066457233 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb720003ad0 con 0x7fb7301167d0 2024-04-22T06:46:44.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.414+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73011bb80 con 0x7fb7301167d0 2024-04-22T06:46:44.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.414+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73011c050 con 0x7fb7301167d0 2024-04-22T06:46:44.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.419+0000 7fb737de5740 1 Processor -- start 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 -- start start 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c5430 0x55e5a98cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 0x55e5a98d0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9828f00 con 0x55e5a98c5430 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e5a97c6590 con 0x55e5a98c4a80 2024-04-22T06:46:44.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.420+0000 7fb737de5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9824ec0 con 0x55e5a98cdfb0 2024-04-22T06:46:44.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.424+0000 7fb717fff640 1 -- 172.21.15.79:0/1066457233 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb720046020 con 0x7fb7301167d0 2024-04-22T06:46:44.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.425+0000 7fb717fff640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7180a3dd0 0x7fb7180a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.425+0000 7fb717fff640 1 -- 172.21.15.79:0/1066457233 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fb7200f21e0 con 0x7fb7301167d0 2024-04-22T06:46:44.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.427+0000 7fb72e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 0x55e5a98d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.427+0000 7fb72f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.427+0000 7fb72f59e640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7180a3dd0 0x7fb7180a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.428+0000 7fb72e59c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 0x55e5a98d0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:45586/0 (socket says 172.21.15.79:45586) 2024-04-22T06:46:44.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.428+0000 7fb72e59c640 1 -- 172.21.15.79:0/3672874783 learned_addr learned my addr 172.21.15.79:0/3672874783 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:44.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.428+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c5430 0x55e5a98cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.428 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.430+0000 7fb72f59e640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7180a3dd0 0x7fb7180a6290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb728005c40 tx=0x7fb72800a5b0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.430+0000 7fb72e59c640 1 -- 172.21.15.79:0/3672874783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 msgr2=0x55e5a98c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.430+0000 7fb72e59c640 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.430+0000 7fb72e59c640 1 -- 172.21.15.79:0/3672874783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c5430 msgr2=0x55e5a98cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.430+0000 7fb72e59c640 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c5430 0x55e5a98cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.430+0000 7fb72e59c640 1 -- 172.21.15.79:0/3672874783 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5a96037e0 con 0x55e5a98cdfb0 2024-04-22T06:46:44.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.434+0000 7fb72e59c640 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 0x55e5a98d0400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb720006010 tx=0x7fb7200bdee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db728da40d9b4efe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.436+0000 7fb7157fa640 1 -- 172.21.15.79:0/3672874783 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7200be810 con 0x55e5a98cdfb0 2024-04-22T06:46:44.434 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.436+0000 7fb7157fa640 1 -- 172.21.15.79:0/3672874783 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7200be110 con 0x55e5a98cdfb0 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.437+0000 7fb737de5740 1 -- 172.21.15.79:0/3672874783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 msgr2=0x55e5a98d0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.437+0000 7fb737de5740 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 0x55e5a98d0400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb720006010 tx=0x7fb7200bdee0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.437+0000 7fb737de5740 1 -- 172.21.15.79:0/3672874783 shutdown_connections 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.438+0000 7fb737de5740 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98cdfb0 0x55e5a98d0400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.438+0000 7fb737de5740 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c5430 0x55e5a98cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.438+0000 7fb737de5740 1 --2- 172.21.15.79:0/3672874783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.438+0000 7fb737de5740 1 -- 172.21.15.79:0/3672874783 >> 172.21.15.79:0/3672874783 conn(0x55e5a98bf150 msgr2=0x55e5a98bf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.440+0000 7fb737de5740 1 -- 172.21.15.79:0/3672874783 shutdown_connections 2024-04-22T06:46:44.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.440+0000 7fb737de5740 1 -- 172.21.15.79:0/3672874783 wait complete. 2024-04-22T06:46:44.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.440+0000 7fb737de5740 1 -- start start 2024-04-22T06:46:44.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.440+0000 7fb737de5740 1 Processor -- start 2024-04-22T06:46:44.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 0x55e5a98da0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e5a98d0ce0 con 0x55e5a98cb2f0 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e5a97e2580 con 0x55e5a98c4a80 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5a98d0e60 con 0x55e5a98c5430 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.441+0000 7fb737de5740 10 client.0 _ll_register_callbacks cb 0x55e5a97e8650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.442+0000 7fb72f59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 0x55e5a98da0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.442+0000 7fb72f59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 0x55e5a98da0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60350/0 (socket says 172.21.15.79:60350) 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.442+0000 7fb72f59e640 1 -- 172.21.15.79:0/650960455 learned_addr learned my addr 172.21.15.79:0/650960455 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:44.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.442+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.443+0000 7fb72f59e640 1 -- 172.21.15.79:0/650960455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 msgr2=0x55e5a98cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.443+0000 7fb72f59e640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.443+0000 7fb72f59e640 1 -- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 msgr2=0x55e5a98cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.443+0000 7fb72f59e640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.443+0000 7fb72f59e640 1 -- 172.21.15.79:0/650960455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5a9663f70 con 0x55e5a98cb2f0 2024-04-22T06:46:44.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.444+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:44.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.444+0000 7fb72f59e640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 0x55e5a98da0d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb7280108c0 tx=0x7fb72800a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dae4d81e620e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.444+0000 7fb712ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:44.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.445+0000 7fb712ff5640 1 -- 172.21.15.79:0/650960455 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728029060 con 0x55e5a98cb2f0 2024-04-22T06:46:44.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.445+0000 7fb712ff5640 1 -- 172.21.15.79:0/650960455 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fb7280054c0 con 0x55e5a98cb2f0 2024-04-22T06:46:44.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.445+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e5a9836920 con 0x55e5a98cb2f0 2024-04-22T06:46:44.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.447+0000 7fb712ff5640 1 -- 172.21.15.79:0/650960455 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb7280056c0 con 0x55e5a98cb2f0 2024-04-22T06:46:44.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.447+0000 7fb737de5740 10 client.16383 fetch_fsmap learned FSMap version 38 2024-04-22T06:46:44.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.447+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e5a94fb170 con 0x55e5a98cb2f0 2024-04-22T06:46:44.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.453+0000 7fb712ff5640 1 -- 172.21.15.79:0/650960455 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e5a9836920 con 0x55e5a98cb2f0 2024-04-22T06:46:44.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb712ff5640 1 -- 172.21.15.79:0/650960455 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fb72801b070 con 0x55e5a98cb2f0 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb737de5740 10 client.16383 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb737de5740 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb737de5740 20 client.16383 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb737de5740 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a98de410 0x55e5a98e0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb737de5740 4 client.16383 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.454+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5a9830790 con 0x55e5a98de410 2024-04-22T06:46:44.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.455+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a98de410 0x55e5a98e0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.465+0000 7fb712ff5640 10 client.16383 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:44.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.465+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a98de410 0x55e5a98e0870 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdad0ba260fc5439 server_cookie=8b5e7b2b2258f5ee in_seq=0 out_seq=0 2024-04-22T06:46:44.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.471+0000 7fb712ff5640 1 -- 172.21.15.79:0/650960455 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5a9830790 con 0x55e5a98de410 2024-04-22T06:46:44.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.471+0000 7fb712ff5640 10 client.16383 handle_command_reply: tid=0 2024-04-22T06:46:44.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.472+0000 7fb737de5740 1 client.16383 shutdown 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 msgr2=0x55e5a98da0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 0x55e5a98da0d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb7280108c0 tx=0x7fb72800a320 comp rx=0 tx=0).stop 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 shutdown_connections 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a98de410 0x55e5a98e0870 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98cb2f0 0x55e5a98da0d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 --2- 172.21.15.79:0/650960455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98c4a80 0x55e5a98cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 >> 172.21.15.79:0/650960455 conn(0x55e5a98bf150 msgr2=0x55e5a98c0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb712ff5640 0 client.16383 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 shutdown_connections 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 1 -- 172.21.15.79:0/650960455 wait complete. 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.473+0000 7fb737de5740 20 client.16383 trim_cache size 0 max 16384 2024-04-22T06:46:44.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 Processor -- start 2024-04-22T06:46:44.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 -- start start 2024-04-22T06:46:44.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98e6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 0x55e5a98e6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301b4b00 con 0x55e5a98c4a80 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301b51d0 con 0x55e5a98cb2f0 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.506+0000 7fb737de5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301b2950 con 0x55e5a98c5430 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60358/0 (socket says 172.21.15.79:60358) 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 -- 172.21.15.79:0/3394946403 learned_addr learned my addr 172.21.15.79:0/3394946403 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 -- 172.21.15.79:0/3394946403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 msgr2=0x55e5a98e6140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98e6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 -- 172.21.15.79:0/3394946403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 msgr2=0x55e5a98e6680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 0x55e5a98e6680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 -- 172.21.15.79:0/3394946403 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5a98c9bd0 con 0x55e5a98c4a80 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d9ba0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x55e5a98c5830 tx=0x7fb724008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08be5eb1f1a7936 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb714ff9640 1 -- 172.21.15.79:0/3394946403 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb724017870 con 0x55e5a98c4a80 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.507+0000 7fb714ff9640 1 -- 172.21.15.79:0/3394946403 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb72401d250 con 0x55e5a98c4a80 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.508+0000 7fb714ff9640 1 -- 172.21.15.79:0/3394946403 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb724015600 con 0x55e5a98c4a80 2024-04-22T06:46:44.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.508+0000 7fb737de5740 1 -- 172.21.15.79:0/3394946403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 msgr2=0x55e5a98d9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.508+0000 7fb737de5740 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d9ba0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x55e5a98c5830 tx=0x7fb724008040 comp rx=0 tx=0).stop 2024-04-22T06:46:44.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.508+0000 7fb737de5740 1 -- 172.21.15.79:0/3394946403 shutdown_connections 2024-04-22T06:46:44.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.508+0000 7fb737de5740 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 0x55e5a98e6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.508+0000 7fb737de5740 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98e6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 --2- 172.21.15.79:0/3394946403 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d9ba0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 -- 172.21.15.79:0/3394946403 >> 172.21.15.79:0/3394946403 conn(0x55e5a98bf150 msgr2=0x55e5a98be140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 -- 172.21.15.79:0/3394946403 shutdown_connections 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 -- 172.21.15.79:0/3394946403 wait complete. 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 -- start start 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 Processor -- start 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98d35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.509+0000 7fb737de5740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 0x55e5a98d3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb737de5740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301b2fc0 con 0x55e5a98c4a80 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb737de5740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301b4430 con 0x55e5a98cb2f0 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb737de5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7301b3690 con 0x55e5a98c5430 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb737de5740 10 client.0 _ll_register_callbacks cb 0x7fb728004d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60372/0 (socket says 172.21.15.79:60372) 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 -- 172.21.15.79:0/2876778309 learned_addr learned my addr 172.21.15.79:0/2876778309 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 -- 172.21.15.79:0/2876778309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 msgr2=0x55e5a98d35c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98d35c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 -- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 msgr2=0x55e5a98d3b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 0x55e5a98d3b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.510+0000 7fb72ed9d640 1 -- 172.21.15.79:0/2876778309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5a99449c0 con 0x55e5a98c4a80 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.511+0000 7fb72ed9d640 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d3080 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb724005990 tx=0x7fb7240059c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5b3753e9aebc84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.511+0000 7fb712ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.511+0000 7fb712ff5640 1 -- 172.21.15.79:0/2876778309 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72401d600 con 0x55e5a98c4a80 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.511+0000 7fb712ff5640 1 -- 172.21.15.79:0/2876778309 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fb72401d7a0 con 0x55e5a98c4a80 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.511+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb72401d250 con 0x55e5a98c4a80 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.512+0000 7fb712ff5640 1 -- 172.21.15.79:0/2876778309 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb724017870 con 0x55e5a98c4a80 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.512+0000 7fb737de5740 10 client.16389 fetch_fsmap learned FSMap version 38 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.512+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e5a9945ce0 con 0x55e5a98c4a80 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.515+0000 7fb712ff5640 1 -- 172.21.15.79:0/2876778309 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb724036480 con 0x55e5a98c4a80 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.516+0000 7fb712ff5640 1 -- 172.21.15.79:0/2876778309 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fb72402d020 con 0x55e5a98c4a80 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.516+0000 7fb737de5740 10 client.16389 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.516+0000 7fb737de5740 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.516+0000 7fb737de5740 20 client.16389 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.516+0000 7fb737de5740 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a9946370 0x55e5a9948790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.516+0000 7fb737de5740 4 client.16389 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:46:44.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.517+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fb72401d250 con 0x55e5a9946370 2024-04-22T06:46:44.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.519+0000 7fb72e59c640 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a9946370 0x55e5a9948790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.522+0000 7fb72e59c640 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a9946370 0x55e5a9948790 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84e7709959872b18 server_cookie=45660213428da046 in_seq=0 out_seq=0 2024-04-22T06:46:44.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.522+0000 7fb712ff5640 10 client.16389 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:44.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 GiB/s wr, 2.56k op/s 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00399989 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0109998 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0149998 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:reconnect 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:reconnect 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0269992 2024-04-22T06:46:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:44 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:44.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 GiB/s wr, 2.56k op/s 2024-04-22T06:46:44.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00399989 2024-04-22T06:46:44.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0109998 2024-04-22T06:46:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0149998 2024-04-22T06:46:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:reconnect 2024-04-22T06:46:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:reconnect 2024-04-22T06:46:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0269992 2024-04-22T06:46:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:44 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:44.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb712ff5640 1 -- 172.21.15.79:0/2876778309 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fb72401d250 con 0x55e5a9946370 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb712ff5640 10 client.16389 handle_command_reply: tid=0 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 client.16389 shutdown 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 msgr2=0x55e5a98d3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d3080 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb724005990 tx=0x7fb7240059c0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 shutdown_connections 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5a9946370 0x55e5a9948790 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5a98cb2f0 0x55e5a98d3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5a98c5430 0x55e5a98d35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 --2- 172.21.15.79:0/2876778309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5a98c4a80 0x55e5a98d3080 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 >> 172.21.15.79:0/2876778309 conn(0x55e5a98bf150 msgr2=0x55e5a98c0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.722+0000 7fb712ff5640 0 client.16389 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:44.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.723+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 shutdown_connections 2024-04-22T06:46:44.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.723+0000 7fb737de5740 1 -- 172.21.15.79:0/2876778309 wait complete. 2024-04-22T06:46:44.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.723+0000 7fb737de5740 20 client.16389 trim_cache size 0 max 16384 2024-04-22T06:46:44.723 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:44.723 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:46:44.723 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "91e0fc9c-9e49-4860-8ef6-0d7b27003d10", 2024-04-22T06:46:44.723 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:46:44.723 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:44.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.737+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7180a3dd0 msgr2=0x7fb7180a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.737+0000 7fb734914640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7180a3dd0 0x7fb7180a6290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb728005c40 tx=0x7fb72800a5b0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.737+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301167d0 msgr2=0x7fb730116c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.737+0000 7fb734914640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301167d0 0x7fb730116c30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb720008000 tx=0x7fb72002db20 comp rx=0 tx=0).stop 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 GiB/s wr, 2.56k op/s 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00399989 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0109998 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0149998 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:reconnect 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:reconnect 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0269992 2024-04-22T06:46:44.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:44 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.747+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 shutdown_connections 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.747+0000 7fb734914640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb7180a3dd0 0x7fb7180a6290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.747+0000 7fb734914640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb730117170 0x7fb73011b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.747+0000 7fb734914640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7301167d0 0x7fb730116c30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.747+0000 7fb734914640 1 --2- 172.21.15.79:0/1066457233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7301099d0 0x7fb73011d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.747+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 >> 172.21.15.79:0/1066457233 conn(0x7fb73006ced0 msgr2=0x7fb730070e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.748+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 shutdown_connections 2024-04-22T06:46:44.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:44.748+0000 7fb734914640 1 -- 172.21.15.79:0/1066457233 wait complete. 2024-04-22T06:46:44.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:44.822 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:45.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 -- 172.21.15.79:0/1034967597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc106660 msgr2=0x7efcbc106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 --2- 172.21.15.79:0/1034967597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc106660 0x7efcbc106a40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efcac009930 tx=0x7efcac02bc50 comp rx=0 tx=0).stop 2024-04-22T06:46:45.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 -- 172.21.15.79:0/1034967597 shutdown_connections 2024-04-22T06:46:45.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 --2- 172.21.15.79:0/1034967597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc100f80 0x7efcbc10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 --2- 172.21.15.79:0/1034967597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc1005c0 0x7efcbc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 --2- 172.21.15.79:0/1034967597 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc106660 0x7efcbc106a40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.096+0000 7efcc084e640 1 -- 172.21.15.79:0/1034967597 >> 172.21.15.79:0/1034967597 conn(0x7efcbc0773a0 msgr2=0x7efcbc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:45.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.097+0000 7efcc084e640 1 -- 172.21.15.79:0/1034967597 shutdown_connections 2024-04-22T06:46:45.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.097+0000 7efcc084e640 1 -- 172.21.15.79:0/1034967597 wait complete. 2024-04-22T06:46:45.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 Processor -- start 2024-04-22T06:46:45.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 -- start start 2024-04-22T06:46:45.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 0x7efcbc197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc100f80 0x7efcbc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc106660 0x7efcbc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc110930 con 0x7efcbc1005c0 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc1107b0 con 0x7efcbc106660 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcc084e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc110ab0 con 0x7efcbc100f80 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcba59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 0x7efcbc197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.098+0000 7efcba59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 0x7efcbc197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60390/0 (socket says 172.21.15.79:60390) 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.099+0000 7efcba59c640 1 -- 172.21.15.79:0/575950508 learned_addr learned my addr 172.21.15.79:0/575950508 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.099+0000 7efcba59c640 1 -- 172.21.15.79:0/575950508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc100f80 msgr2=0x7efcbc197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.099+0000 7efcba59c640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc100f80 0x7efcbc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.099+0000 7efcba59c640 1 -- 172.21.15.79:0/575950508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc106660 msgr2=0x7efcbc197fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.099+0000 7efcba59c640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc106660 0x7efcbc197fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.099+0000 7efcba59c640 1 -- 172.21.15.79:0/575950508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc1a6d90 con 0x7efcbc1005c0 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.100+0000 7efcba59c640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 0x7efcbc197520 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efcac005ec0 tx=0x7efcac02ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.100+0000 7efcab7fe640 1 -- 172.21.15.79:0/575950508 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac043020 con 0x7efcbc1005c0 2024-04-22T06:46:45.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.100+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc1a7020 con 0x7efcbc1005c0 2024-04-22T06:46:45.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.100+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc1a75c0 con 0x7efcbc1005c0 2024-04-22T06:46:45.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.100+0000 7efcab7fe640 1 -- 172.21.15.79:0/575950508 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcac0052b0 con 0x7efcbc1005c0 2024-04-22T06:46:45.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.100+0000 7efcab7fe640 1 -- 172.21.15.79:0/575950508 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac003d40 con 0x7efcbc1005c0 2024-04-22T06:46:45.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.103+0000 7efcc3440740 1 Processor -- start 2024-04-22T06:46:45.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 -- start start 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef792cd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792cd6b0 0x55ef792d5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ef792291e0 con 0x55ef792d6290 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ef791c6390 con 0x55ef792ccd90 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcc3440740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ef792251a0 con 0x55ef792cd6b0 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60392/0 (socket says 172.21.15.79:60392) 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 -- 172.21.15.79:0/1046175710 learned_addr learned my addr 172.21.15.79:0/1046175710 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 -- 172.21.15.79:0/1046175710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792cd6b0 msgr2=0x55ef792d5d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792cd6b0 0x55ef792d5d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 -- 172.21.15.79:0/1046175710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 msgr2=0x55ef792cd170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.104+0000 7efcb9d9b640 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef792cd170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.105+0000 7efcb9d9b640 1 -- 172.21.15.79:0/1046175710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef790057d0 con 0x55ef792d6290 2024-04-22T06:46:45.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.105+0000 7efcb9d9b640 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d8680 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7efcb00097b0 tx=0x7efcb0009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71a533d8637e4ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.105+0000 7efca8ff9640 1 -- 172.21.15.79:0/1046175710 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0026020 con 0x55ef792d6290 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.105+0000 7efca8ff9640 1 -- 172.21.15.79:0/1046175710 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb000bd80 con 0x55ef792d6290 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.106+0000 7efca8ff9640 1 -- 172.21.15.79:0/1046175710 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0022e50 con 0x55ef792d6290 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 -- 172.21.15.79:0/1046175710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 msgr2=0x55ef792d8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d8680 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7efcb00097b0 tx=0x7efcb0009c70 comp rx=0 tx=0).stop 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 -- 172.21.15.79:0/1046175710 shutdown_connections 2024-04-22T06:46:45.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d8680 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792cd6b0 0x55ef792d5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 --2- 172.21.15.79:0/1046175710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef792cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcc3440740 1 -- 172.21.15.79:0/1046175710 >> 172.21.15.79:0/1046175710 conn(0x55ef792c74c0 msgr2=0x55ef792c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:45.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.107+0000 7efcab7fe640 1 -- 172.21.15.79:0/575950508 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efcac046020 con 0x7efcbc1005c0 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.108+0000 7efcc3440740 1 -- 172.21.15.79:0/1046175710 shutdown_connections 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.108+0000 7efcc3440740 1 -- 172.21.15.79:0/1046175710 wait complete. 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.108+0000 7efcc3440740 1 -- start start 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.108+0000 7efcc3440740 1 Processor -- start 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.108+0000 7efcab7fe640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efca40a3cb0 0x7efca40a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcbad9d640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efca40a3cb0 0x7efca40a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef792d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d6290 0x55ef792d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 0x55ef792e2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ef792d9080 con 0x55ef792d3cf0 2024-04-22T06:46:45.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ef791e22f0 con 0x55ef792ccd90 2024-04-22T06:46:45.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ef792d9200 con 0x55ef792d6290 2024-04-22T06:46:45.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcc3440740 10 client.0 _ll_register_callbacks cb 0x55ef791ef580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.109+0000 7efcab7fe640 1 -- 172.21.15.79:0/575950508 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7efcac0f29c0 con 0x7efcbc1005c0 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.110+0000 7efcbad9d640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efca40a3cb0 0x7efca40a6170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0ff940 tx=0x7efcb4009250 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.110+0000 7efcbad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 0x55ef792e2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.110+0000 7efcbad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 0x55ef792e2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60404/0 (socket says 172.21.15.79:60404) 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.110+0000 7efcbad9d640 1 -- 172.21.15.79:0/2437488506 learned_addr learned my addr 172.21.15.79:0/2437488506 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcb9d9b640 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d6290 0x55ef792d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcbad9d640 1 -- 172.21.15.79:0/2437488506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d6290 msgr2=0x55ef792d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:45.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcbad9d640 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d6290 0x55ef792d37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcbad9d640 1 -- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 msgr2=0x55ef792d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:45.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcbad9d640 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef792d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcbad9d640 1 -- 172.21.15.79:0/2437488506 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef78f3b3f0 con 0x55ef792d3cf0 2024-04-22T06:46:45.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efcbad9d640 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 0x55ef792e2580 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efcb400da00 tx=0x7efcb4004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b91daea9b30ed75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:45.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.111+0000 7efca1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:45.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efca1ffb640 1 -- 172.21.15.79:0/2437488506 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4018d10 con 0x55ef792d3cf0 2024-04-22T06:46:45.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb000bd80 con 0x55ef792d3cf0 2024-04-22T06:46:45.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efca1ffb640 1 -- 172.21.15.79:0/2437488506 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7efcb4023480 con 0x55ef792d3cf0 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efca1ffb640 1 -- 172.21.15.79:0/2437488506 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7efcb4023680 con 0x55ef792d3cf0 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 10 client.16401 fetch_fsmap learned FSMap version 38 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ef78efb0f0 con 0x55ef792d3cf0 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efca1ffb640 1 -- 172.21.15.79:0/2437488506 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb000bd80 con 0x55ef792d3cf0 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efca1ffb640 1 -- 172.21.15.79:0/2437488506 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7efcb401a940 con 0x55ef792d3cf0 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 10 client.16401 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 20 client.16401 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef792e9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.112+0000 7efcc3440740 4 client.16401 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:45.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.113+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef79236c00 con 0x55ef792e6c00 2024-04-22T06:46:45.113 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.115+0000 7efcba59c640 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef792e9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.121+0000 7efca1ffb640 10 client.16401 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:45.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.121+0000 7efcba59c640 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef792e9080 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6754c6cc406a6db server_cookie=2e50f6e951c18ece in_seq=0 out_seq=0 2024-04-22T06:46:45.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.125+0000 7efca1ffb640 1 -- 172.21.15.79:0/2437488506 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef79236c00 con 0x55ef792e6c00 2024-04-22T06:46:45.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.125+0000 7efca1ffb640 10 client.16401 handle_command_reply: tid=0 2024-04-22T06:46:45.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.125+0000 7efcc3440740 1 client.16401 shutdown 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.125+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 msgr2=0x55ef792e2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.125+0000 7efcc3440740 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 0x55ef792e2580 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efcb400da00 tx=0x7efcb4004b50 comp rx=0 tx=0).stop 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 shutdown_connections 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef792e9080 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d3cf0 0x55ef792e2580 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efca1ffb640 0 client.16401 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:45.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d6290 0x55ef792d37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 --2- 172.21.15.79:0/2437488506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef792d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 >> 172.21.15.79:0/2437488506 conn(0x55ef792c74c0 msgr2=0x55ef792c9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:45.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 shutdown_connections 2024-04-22T06:46:45.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 1 -- 172.21.15.79:0/2437488506 wait complete. 2024-04-22T06:46:45.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.126+0000 7efcc3440740 20 client.16401 trim_cache size 0 max 16384 2024-04-22T06:46:45.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.156+0000 7efcc3440740 1 Processor -- start 2024-04-22T06:46:45.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.156+0000 7efcc3440740 1 -- start start 2024-04-22T06:46:45.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcc3440740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792ccd90 0x55ef792c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcc3440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792d3cf0 0x55ef792ee990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcc3440740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcc3440740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c00eae0 con 0x55ef792d6290 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcbad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcbad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792eeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60408/0 (socket says 172.21.15.79:60408) 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcbad9d640 1 -- 172.21.15.79:0/1073744374 learned_addr learned my addr 172.21.15.79:0/1073744374 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c00f1b0 con 0x55ef792d3cf0 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.157+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c00c930 con 0x55ef792ccd90 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efcbad9d640 1 -- 172.21.15.79:0/1073744374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792ccd90 msgr2=0x55ef792c78a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efcbad9d640 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792ccd90 0x55ef792c78a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efcbad9d640 1 -- 172.21.15.79:0/1073744374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792d3cf0 msgr2=0x55ef792ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:45.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efcbad9d640 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792d3cf0 0x55ef792ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efcbad9d640 1 -- 172.21.15.79:0/1073744374 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef792e5350 con 0x55ef792d6290 2024-04-22T06:46:45.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efcbad9d640 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792eeed0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efcb400c8a0 tx=0x7efcb4018fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffbaa6286f152ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:45.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.158+0000 7efca3fff640 1 -- 172.21.15.79:0/1073744374 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4018d10 con 0x55ef792d6290 2024-04-22T06:46:45.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.159+0000 7efca3fff640 1 -- 172.21.15.79:0/1073744374 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb4023cd0 con 0x55ef792d6290 2024-04-22T06:46:45.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.159+0000 7efca3fff640 1 -- 172.21.15.79:0/1073744374 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb402a480 con 0x55ef792d6290 2024-04-22T06:46:45.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 msgr2=0x55ef792eeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792eeed0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7efcb400c8a0 tx=0x7efcb4018fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 shutdown_connections 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792d3cf0 0x55ef792ee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792eeed0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 --2- 172.21.15.79:0/1073744374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792ccd90 0x55ef792c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 >> 172.21.15.79:0/1073744374 conn(0x55ef792c74c0 msgr2=0x55ef792d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.160+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 shutdown_connections 2024-04-22T06:46:45.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.161+0000 7efcc3440740 1 -- 172.21.15.79:0/1073744374 wait complete. 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.161+0000 7efcc3440740 1 -- start start 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.161+0000 7efcc3440740 1 Processor -- start 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef79345500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d3cf0 0x55ef79345a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c00cfa0 con 0x55ef792d6290 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c00e410 con 0x55ef792ccd90 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c00d670 con 0x55ef792d3cf0 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcc3440740 10 client.0 _ll_register_callbacks cb 0x7efcb400a7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcbad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcbad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60422/0 (socket says 172.21.15.79:60422) 2024-04-22T06:46:45.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcbad9d640 1 -- 172.21.15.79:0/295799392 learned_addr learned my addr 172.21.15.79:0/295799392 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcbad9d640 1 -- 172.21.15.79:0/295799392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d3cf0 msgr2=0x55ef79345a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.162+0000 7efcbad9d640 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d3cf0 0x55ef79345a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcbad9d640 1 -- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 msgr2=0x55ef79345500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcbad9d640 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef79345500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcbad9d640 1 -- 172.21.15.79:0/295799392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef792d9fa0 con 0x55ef792d6290 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcbad9d640 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d98b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7efcb400da00 tx=0x7efcb400a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1952076a4c3c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:45.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efca1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:45.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efca1ffb640 1 -- 172.21.15.79:0/295799392 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4044060 con 0x55ef792d6290 2024-04-22T06:46:45.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efca1ffb640 1 -- 172.21.15.79:0/295799392 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7efcb4023a30 con 0x55ef792d6290 2024-04-22T06:46:45.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb4023cd0 con 0x55ef792d6290 2024-04-22T06:46:45.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efca1ffb640 1 -- 172.21.15.79:0/295799392 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7efcb404b050 con 0x55ef792d6290 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcc3440740 10 client.16407 fetch_fsmap learned FSMap version 38 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.163+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ef7934ccd0 con 0x55ef792d6290 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.164+0000 7efca1ffb640 1 -- 172.21.15.79:0/295799392 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb4023cd0 con 0x55ef792d6290 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efca1ffb640 1 -- 172.21.15.79:0/295799392 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7efcb402ac00 con 0x55ef792d6290 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efcc3440740 10 client.16407 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efcc3440740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efcc3440740 20 client.16407 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efcc3440740 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef7934f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efcc3440740 4 client.16407 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:45.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.165+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ef79230a70 con 0x55ef792e6c00 2024-04-22T06:46:45.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.167+0000 7efcba59c640 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef7934f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.173+0000 7efca1ffb640 10 client.16407 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:45.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.173+0000 7efcba59c640 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef7934f340 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=876e8330b31613fb server_cookie=3d278a41692e3d76 in_seq=0 out_seq=0 2024-04-22T06:46:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.176+0000 7efca1ffb640 1 -- 172.21.15.79:0/295799392 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55ef79230a70 con 0x55ef792e6c00 2024-04-22T06:46:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.176+0000 7efca1ffb640 10 client.16407 handle_command_reply: tid=0 2024-04-22T06:46:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.176+0000 7efcc3440740 1 client.16407 shutdown 2024-04-22T06:46:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.176+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 msgr2=0x55ef792d98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.176+0000 7efcc3440740 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d98b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7efcb400da00 tx=0x7efcb400a910 comp rx=0 tx=0).stop 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 shutdown_connections 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ef792e6c00 0x55ef7934f340 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ef792d6290 0x55ef792d98b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ef792d3cf0 0x55ef79345a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 --2- 172.21.15.79:0/295799392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ef792ccd90 0x55ef79345500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 >> 172.21.15.79:0/295799392 conn(0x55ef792c74c0 msgr2=0x55ef792bd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efca1ffb640 0 client.16407 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 shutdown_connections 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 1 -- 172.21.15.79:0/295799392 wait complete. 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.177+0000 7efcc3440740 20 client.16407 trim_cache size 0 max 16384 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:46:45.176 INFO:teuthology.orchestra.run.smithi079.stdout: "91e0fc9c-9e49-4860-8ef6-0d7b27003d10": { 2024-04-22T06:46:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:46:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "91e0fc9c-9e49-4860-8ef6-0d7b27003d10", 2024-04-22T06:46:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:46:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:45.177 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:46:45.177 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efca40a3cb0 msgr2=0x7efca40a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efca40a3cb0 0x7efca40a6170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0ff940 tx=0x7efcb4009250 comp rx=0 tx=0).stop 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 msgr2=0x7efcbc197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 0x7efcbc197520 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efcac005ec0 tx=0x7efcac02ca30 comp rx=0 tx=0).stop 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 shutdown_connections 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efca40a3cb0 0x7efca40a6170 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc106660 0x7efcbc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.178+0000 7efcc084e640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc100f80 0x7efcbc197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.179+0000 7efcc084e640 1 --2- 172.21.15.79:0/575950508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efcbc1005c0 0x7efcbc197520 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.179+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 >> 172.21.15.79:0/575950508 conn(0x7efcbc0773a0 msgr2=0x7efcbc102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.179+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 shutdown_connections 2024-04-22T06:46:45.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:46:45.179+0000 7efcc084e640 1 -- 172.21.15.79:0/575950508 wait complete. 2024-04-22T06:46:45.196 INFO:tasks.cephfs.filesystem:scrub status for tag:91e0fc9c-9e49-4860-8ef6-0d7b27003d10 - {'path': '/', 'tag': '91e0fc9c-9e49-4860-8ef6-0d7b27003d10', 'options': 'recursive,force'} 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:rejoin 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:rejoin 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:46:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:active 2024-04-22T06:46:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:active 2024-04-22T06:46:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:45.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:45 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:rejoin 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:rejoin 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:46:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:active 2024-04-22T06:46:45.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:active 2024-04-22T06:46:45.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:45.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:45 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:rejoin 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:rejoin 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] up:active 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] up:active 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:46:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:45 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:46:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:46 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:46:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:46 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:46 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:46:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:46 smithi193 ceph-mon[31815]: pgmap v77: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 544 MiB/s wr, 1.48k op/s 2024-04-22T06:46:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:46 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:46:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:46 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:46.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:46 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:46:46.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:46 smithi190 ceph-mon[31421]: pgmap v77: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 544 MiB/s wr, 1.48k op/s 2024-04-22T06:46:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:46 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:46:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:46 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:46 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:46:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:46 smithi079 ceph-mon[27946]: pgmap v77: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 544 MiB/s wr, 1.48k op/s 2024-04-22T06:46:47.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 -- 172.21.15.79:0/1627907741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e08100d60 msgr2=0x7f2e08101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/1627907741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e08100d60 0x7f2e08101140 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009900 tx=0x7f2dfc02bc80 comp rx=0 tx=0).stop 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 -- 172.21.15.79:0/1627907741 shutdown_connections 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/1627907741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e0810e910 0x7f2e08110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/1627907741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 0x7f2e0810e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/1627907741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e08100d60 0x7f2e08101140 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 -- 172.21.15.79:0/1627907741 >> 172.21.15.79:0/1627907741 conn(0x7f2e08077fd0 msgr2=0x7f2e080783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 -- 172.21.15.79:0/1627907741 shutdown_connections 2024-04-22T06:46:47.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.953+0000 7f2e0edbd640 1 -- 172.21.15.79:0/1627907741 wait complete. 2024-04-22T06:46:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 Processor -- start 2024-04-22T06:46:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 -- start start 2024-04-22T06:46:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 0x7f2e08109010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 0x7f2e08109550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e08109a90 0x7f2e081ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08114540 con 0x7f2e0810e910 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081143c0 con 0x7f2e08109a90 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e0edbd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081146c0 con 0x7f2e08101680 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 0x7f2e08109550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 0x7f2e08109550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60446/0 (socket says 172.21.15.79:60446) 2024-04-22T06:46:47.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.954+0000 7f2e07fff640 1 -- 172.21.15.79:0/3357846648 learned_addr learned my addr 172.21.15.79:0/3357846648 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:47.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 0x7f2e08109010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e07fff640 1 -- 172.21.15.79:0/3357846648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 msgr2=0x7f2e08109010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 0x7f2e08109010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e07fff640 1 -- 172.21.15.79:0/3357846648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e08109a90 msgr2=0x7f2e081ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e08109a90 0x7f2e081ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e07fff640 1 -- 172.21.15.79:0/3357846648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e081ab7c0 con 0x7f2e0810e910 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.956+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 0x7f2e08109010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.957+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 0x7f2e08109550 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b450 tx=0x7f2df400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.957+0000 7f2e05ffb640 1 -- 172.21.15.79:0/3357846648 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4015020 con 0x7f2e0810e910 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.957+0000 7f2e05ffb640 1 -- 172.21.15.79:0/3357846648 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df4011d40 con 0x7f2e0810e910 2024-04-22T06:46:47.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.957+0000 7f2e05ffb640 1 -- 172.21.15.79:0/3357846648 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df400f340 con 0x7f2e0810e910 2024-04-22T06:46:47.957 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.957+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e081aba50 con 0x7f2e0810e910 2024-04-22T06:46:47.957 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.958+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e081abf20 con 0x7f2e0810e910 2024-04-22T06:46:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.963+0000 7f2e05ffb640 1 -- 172.21.15.79:0/3357846648 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2df4018020 con 0x7f2e0810e910 2024-04-22T06:46:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.965+0000 7f2e119b4740 1 Processor -- start 2024-04-22T06:46:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.965+0000 7f2e119b4740 1 -- start start 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.965+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33903620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.965+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b33903fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.966+0000 7f2e0cb59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b33903fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.966+0000 7f2e0cb59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b33903fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60454/0 (socket says 172.21.15.79:60454) 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3975819023 learned_addr learned my addr 172.21.15.79:0/3975819023 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560b338688d0 con 0x560b33903b60 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560b33805e80 con 0x560b339048d0 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560b33864890 con 0x560b3390d0e0 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e0d35a640 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33903620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.967+0000 7f2e05ffb640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2df80a3ec0 0x7f2df80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3975819023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 msgr2=0x560b3390f4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390f4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3975819023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 msgr2=0x560b33903620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33903620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3975819023 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b33827320 con 0x560b33903b60 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e05ffb640 1 -- 172.21.15.79:0/3357846648 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f2df40d0170 con 0x7f2e0810e910 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0d35a640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2df80a3ec0 0x7f2df80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b33903fe0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc006010 tx=0x7f2dfc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c985e293c70adb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2df2ffd640 1 -- 172.21.15.79:0/3975819023 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc043020 con 0x560b33903b60 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2df2ffd640 1 -- 172.21.15.79:0/3975819023 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dfc02c830 con 0x560b33903b60 2024-04-22T06:46:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.968+0000 7f2df2ffd640 1 -- 172.21.15.79:0/3975819023 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0046c0 con 0x560b33903b60 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 msgr2=0x560b33903fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b33903fe0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc006010 tx=0x7f2dfc005e40 comp rx=0 tx=0).stop 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e0d35a640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2df80a3ec0 0x7f2df80a6380 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2e0806b730 tx=0x7f2e00004ad0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 shutdown_connections 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b33903fe0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3975819023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33903620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.969+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 >> 172.21.15.79:0/3975819023 conn(0x560b338cac90 msgr2=0x560b338fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.970+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 shutdown_connections 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.970+0000 7f2e119b4740 1 -- 172.21.15.79:0/3975819023 wait complete. 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.970+0000 7f2e119b4740 1 -- start start 2024-04-22T06:46:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.970+0000 7f2e119b4740 1 Processor -- start 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33909e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560b3390fdb0 con 0x560b33903b60 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560b337f0eb0 con 0x560b339048d0 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560b3390ff30 con 0x560b3390d0e0 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e0cb59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e0cb59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60468/0 (socket says 172.21.15.79:60468) 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3808287527 learned_addr learned my addr 172.21.15.79:0/3808287527 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.971+0000 7f2e119b4740 10 client.0 _ll_register_callbacks cb 0x560b3381db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3808287527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 msgr2=0x560b3390a440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:47.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390a440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 msgr2=0x560b33909e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33909e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3808287527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b3357b210 con 0x560b33903b60 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390caa0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009a20 tx=0x7f2dfc005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9698af33ac0b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2de7fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.972+0000 7f2de7fff640 1 -- 172.21.15.79:0/3808287527 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc043040 con 0x560b33903b60 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.973+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dfc02c830 con 0x560b33903b60 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.973+0000 7f2de7fff640 1 -- 172.21.15.79:0/3808287527 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f2dfc0046c0 con 0x560b33903b60 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.973+0000 7f2de7fff640 1 -- 172.21.15.79:0/3808287527 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f2dfc057050 con 0x560b33903b60 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.973+0000 7f2e119b4740 10 client.16419 fetch_fsmap learned FSMap version 40 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.973+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560b33647620 con 0x560b33903b60 2024-04-22T06:46:47.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.973+0000 7f2de7fff640 1 -- 172.21.15.79:0/3808287527 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dfc02c830 con 0x560b33903b60 2024-04-22T06:46:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2de7fff640 1 -- 172.21.15.79:0/3808287527 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f2dfc0037a0 con 0x560b33903b60 2024-04-22T06:46:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2e119b4740 10 client.16419 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:46:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2e119b4740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2e119b4740 20 client.16419 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b3391f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2e119b4740 4 client.16419 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.974+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b3387c880 con 0x560b3391cd10 2024-04-22T06:46:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.976+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b3391f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.981+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b3391f100 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98539064cb0dd3fd server_cookie=1b27c92c357249df in_seq=0 out_seq=0 2024-04-22T06:46:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.981+0000 7f2de7fff640 10 client.16419 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.984+0000 7f2de7fff640 1 -- 172.21.15.79:0/3808287527 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b3387c880 con 0x560b3391cd10 2024-04-22T06:46:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.984+0000 7f2de7fff640 10 client.16419 handle_command_reply: tid=0 2024-04-22T06:46:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.984+0000 7f2e119b4740 1 client.16419 shutdown 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 msgr2=0x560b3390caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390caa0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009a20 tx=0x7f2dfc005270 comp rx=0 tx=0).stop 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 shutdown_connections 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b3391f100 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b33909e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2de7fff640 0 client.16419 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3808287527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390caa0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 >> 172.21.15.79:0/3808287527 conn(0x560b338cac90 msgr2=0x560b339005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 shutdown_connections 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 1 -- 172.21.15.79:0/3808287527 wait complete. 2024-04-22T06:46:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:47.985+0000 7f2e119b4740 20 client.16419 trim_cache size 0 max 16384 2024-04-22T06:46:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.011+0000 7f2e119b4740 1 Processor -- start 2024-04-22T06:46:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 -- start start 2024-04-22T06:46:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b33903b60 0x560b33918cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 0x560b3390c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b3390d0e0 0x560b3390caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b7290 con 0x560b339048d0 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b4b60 con 0x560b3390d0e0 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e119b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b29b0 con 0x560b33903b60 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 0x560b3390c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 0x560b3390c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60476/0 (socket says 172.21.15.79:60476) 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e07fff640 1 -- 172.21.15.79:0/1633110100 learned_addr learned my addr 172.21.15.79:0/1633110100 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.012+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b33903b60 0x560b33918cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2e07fff640 1 -- 172.21.15.79:0/1633110100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b33903b60 msgr2=0x560b33918cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2e07fff640 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b33903b60 0x560b33918cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2e07fff640 1 -- 172.21.15.79:0/1633110100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b3390d0e0 msgr2=0x560b3390caa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2e07fff640 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b3390d0e0 0x560b3390caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2e07fff640 1 -- 172.21.15.79:0/1633110100 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b338f9f30 con 0x560b339048d0 2024-04-22T06:46:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2e07fff640 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 0x560b3390c560 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2df40093b0 tx=0x7f2df40046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0f3e7f6964b462 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.013+0000 7f2df27fc640 1 -- 172.21.15.79:0/1633110100 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df40b4070 con 0x560b339048d0 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.014+0000 7f2df27fc640 1 -- 172.21.15.79:0/1633110100 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df40a4d10 con 0x560b339048d0 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.014+0000 7f2df27fc640 1 -- 172.21.15.79:0/1633110100 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df40a0210 con 0x560b339048d0 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.014+0000 7f2e119b4740 1 -- 172.21.15.79:0/1633110100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 msgr2=0x560b3390c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.014+0000 7f2e119b4740 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 0x560b3390c560 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2df40093b0 tx=0x7f2df40046e0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 -- 172.21.15.79:0/1633110100 shutdown_connections 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b3390d0e0 0x560b3390caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b339048d0 0x560b3390c560 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 --2- 172.21.15.79:0/1633110100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b33903b60 0x560b33918cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 -- 172.21.15.79:0/1633110100 >> 172.21.15.79:0/1633110100 conn(0x560b338cac90 msgr2=0x560b338fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 -- 172.21.15.79:0/1633110100 shutdown_connections 2024-04-22T06:46:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 -- 172.21.15.79:0/1633110100 wait complete. 2024-04-22T06:46:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 -- start start 2024-04-22T06:46:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.015+0000 7f2e119b4740 1 Processor -- start 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b3390d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b3020 con 0x560b33903b60 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b4490 con 0x560b339048d0 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081b36f0 con 0x560b3390d0e0 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e119b4740 10 client.0 _ll_register_callbacks cb 0x7f2dfc005d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e0cb59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e0cb59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60488/0 (socket says 172.21.15.79:60488) 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.016+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3676467149 learned_addr learned my addr 172.21.15.79:0/3676467149 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3676467149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 msgr2=0x560b3390dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 msgr2=0x560b3390d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b3390d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e0cb59640 1 -- 172.21.15.79:0/3676467149 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b339141c0 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e0cb59640 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390ed00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc02bcc0 tx=0x7f2dfc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d61de5ed34d0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2de7fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2de7fff640 1 -- 172.21.15.79:0/3676467149 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc043070 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2de7fff640 1 -- 172.21.15.79:0/3676467149 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f2dfc003800 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.017+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2df40a4d10 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2de7fff640 1 -- 172.21.15.79:0/3676467149 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f2dfc02ccf0 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 10 client.16425 fetch_fsmap learned FSMap version 40 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560b33915430 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2de7fff640 1 -- 172.21.15.79:0/3676467149 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2df40a4d10 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2de7fff640 1 -- 172.21.15.79:0/3676467149 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f2dfc04ab50 con 0x560b33903b60 2024-04-22T06:46:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 10 client.16425 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:46:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:46:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 20 client.16425 populate_metadata read hostname 'smithi079' 2024-04-22T06:46:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b33984f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 4 client.16425 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.018+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b338083d0 con 0x560b3391cd10 2024-04-22T06:46:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.021+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b33984f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.027+0000 7f2de7fff640 10 client.16425 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:48.027+0000 7f2e07fff640 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b33984f50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0d06c25c39c8b6 server_cookie=439391918c26d479 in_seq=0 out_seq=0 2024-04-22T06:46:48.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:48 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:48.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:48 smithi193 ceph-mon[31815]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 GiB/s wr, 2.61k op/s 2024-04-22T06:46:48.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:48 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:48.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:48 smithi190 ceph-mon[31421]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 GiB/s wr, 2.61k op/s 2024-04-22T06:46:48.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:48 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:48 smithi079 ceph-mon[27946]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 GiB/s wr, 2.61k op/s 2024-04-22T06:46:49.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:49 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:46:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:49 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:46:49.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:49 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:46:50.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:50 smithi190 ceph-mon[31421]: pgmap v79: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 956 MiB/s wr, 2.40k op/s 2024-04-22T06:46:50.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:50 smithi079 ceph-mon[27946]: pgmap v79: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 956 MiB/s wr, 2.40k op/s 2024-04-22T06:46:50.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:50 smithi193 ceph-mon[31815]: pgmap v79: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 956 MiB/s wr, 2.40k op/s 2024-04-22T06:46:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.600+0000 7f2de7fff640 1 -- 172.21.15.79:0/3676467149 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560b338083d0 con 0x560b3391cd10 2024-04-22T06:46:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.600+0000 7f2de7fff640 10 client.16425 handle_command_reply: tid=0 2024-04-22T06:46:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.600+0000 7f2e119b4740 1 client.16425 shutdown 2024-04-22T06:46:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 msgr2=0x560b3390ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390ed00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc02bcc0 tx=0x7f2dfc005780 comp rx=0 tx=0).stop 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 shutdown_connections 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560b3391cd10 0x560b33984f50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560b339048d0 0x560b3390d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b3390d0e0 0x560b3390dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 --2- 172.21.15.79:0/3676467149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560b33903b60 0x560b3390ed00 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 >> 172.21.15.79:0/3676467149 conn(0x560b338cac90 msgr2=0x560b339085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2de7fff640 0 client.16425 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:46:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 shutdown_connections 2024-04-22T06:46:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.601+0000 7f2e119b4740 1 -- 172.21.15.79:0/3676467149 wait complete. 2024-04-22T06:46:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.602+0000 7f2e119b4740 20 client.16425 trim_cache size 0 max 16384 2024-04-22T06:46:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:46:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:46:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:46:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2df80a3ec0 msgr2=0x7f2df80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2df80a3ec0 0x7f2df80a6380 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2e0806b730 tx=0x7f2e00004ad0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 msgr2=0x7f2e08109550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 0x7f2e08109550 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2df400b450 tx=0x7f2df400b910 comp rx=0 tx=0).stop 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 shutdown_connections 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2df80a3ec0 0x7f2df80a6380 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e08109a90 0x7f2e081ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.606+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e0810e910 0x7f2e08109550 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.607+0000 7f2e0edbd640 1 --2- 172.21.15.79:0/3357846648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e08101680 0x7f2e08109010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.607+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 >> 172.21.15.79:0/3357846648 conn(0x7f2e08077fd0 msgr2=0x7f2e080ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.609+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 shutdown_connections 2024-04-22T06:46:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:46:52.609+0000 7f2e0edbd640 1 -- 172.21.15.79:0/3357846648 wait complete. 2024-04-22T06:46:52.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:52 smithi193 ceph-mon[31815]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 988 MiB/s wr, 2.38k op/s 2024-04-22T06:46:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:52 smithi190 ceph-mon[31421]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 988 MiB/s wr, 2.38k op/s 2024-04-22T06:46:52.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:52 smithi079 ceph-mon[27946]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 988 MiB/s wr, 2.38k op/s 2024-04-22T06:46:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:54 smithi079 ceph-mon[27946]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 752 MiB/s wr, 1.69k op/s 2024-04-22T06:46:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:54 smithi193 ceph-mon[31815]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 752 MiB/s wr, 1.69k op/s 2024-04-22T06:46:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:54 smithi190 ceph-mon[31421]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 752 MiB/s wr, 1.69k op/s 2024-04-22T06:46:56.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:56 smithi193 ceph-mon[31815]: pgmap v82: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 603 MiB/s wr, 1.28k op/s 2024-04-22T06:46:56.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:56 smithi190 ceph-mon[31421]: pgmap v82: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 603 MiB/s wr, 1.28k op/s 2024-04-22T06:46:56.998 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:56 smithi079 ceph-mon[27946]: pgmap v82: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 603 MiB/s wr, 1.28k op/s 2024-04-22T06:46:58.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:46:58 smithi193 ceph-mon[31815]: pgmap v83: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 893 MiB/s wr, 1.49k op/s 2024-04-22T06:46:58.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:46:58 smithi190 ceph-mon[31421]: pgmap v83: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 893 MiB/s wr, 1.49k op/s 2024-04-22T06:46:58.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:46:58 smithi079 ceph-mon[27946]: pgmap v83: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 893 MiB/s wr, 1.49k op/s 2024-04-22T06:47:00.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:00 smithi193 ceph-mon[31815]: pgmap v84: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 562 MiB/s wr, 736 op/s 2024-04-22T06:47:00.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:00 smithi190 ceph-mon[31421]: pgmap v84: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 562 MiB/s wr, 736 op/s 2024-04-22T06:47:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:00 smithi079 ceph-mon[27946]: pgmap v84: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 562 MiB/s wr, 736 op/s 2024-04-22T06:47:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:02 smithi193 ceph-mon[31815]: pgmap v85: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 743 MiB/s wr, 912 op/s 2024-04-22T06:47:02.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:02 smithi190 ceph-mon[31421]: pgmap v85: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 743 MiB/s wr, 912 op/s 2024-04-22T06:47:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:02 smithi079 ceph-mon[27946]: pgmap v85: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 743 MiB/s wr, 912 op/s 2024-04-22T06:47:04.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:04 smithi193 ceph-mon[31815]: pgmap v86: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 686 MiB/s wr, 708 op/s 2024-04-22T06:47:04.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:04 smithi190 ceph-mon[31421]: pgmap v86: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 686 MiB/s wr, 708 op/s 2024-04-22T06:47:04.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:04 smithi079 ceph-mon[27946]: pgmap v86: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 686 MiB/s wr, 708 op/s 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 -- 172.21.15.79:0/16563812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c810c520 msgr2=0x7f56c810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/16563812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c810c520 0x7f56c810e9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009ca0 tx=0x7f56c002c250 comp rx=0 tx=0).stop 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 -- 172.21.15.79:0/16563812 shutdown_connections 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/16563812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c8103b00 0x7f56c810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/16563812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c810c520 0x7f56c810e9c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/16563812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c81031e0 0x7f56c81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 -- 172.21.15.79:0/16563812 >> 172.21.15.79:0/16563812 conn(0x7f56c8077fd0 msgr2=0x7f56c80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 -- 172.21.15.79:0/16563812 shutdown_connections 2024-04-22T06:47:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.799+0000 7f56ccc2c640 1 -- 172.21.15.79:0/16563812 wait complete. 2024-04-22T06:47:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 Processor -- start 2024-04-22T06:47:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 -- start start 2024-04-22T06:47:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 0x7f56c819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c8103b00 0x7f56c819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c810c520 0x7f56c8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81122d0 con 0x7f56c81031e0 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56c8112150 con 0x7f56c8103b00 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.800+0000 7f56ccc2c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56c8112450 con 0x7f56c810c520 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 0x7f56c819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 0x7f56c819e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46948/0 (socket says 172.21.15.79:46948) 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 -- 172.21.15.79:0/2081783587 learned_addr learned my addr 172.21.15.79:0/2081783587 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 -- 172.21.15.79:0/2081783587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c810c520 msgr2=0x7f56c8198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c810c520 0x7f56c8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 -- 172.21.15.79:0/2081783587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c8103b00 msgr2=0x7f56c819eaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c8103b00 0x7f56c819eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 -- 172.21.15.79:0/2081783587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c8198f30 con 0x7f56c81031e0 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.801+0000 7f56c659c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 0x7f56c819e5b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b450 tx=0x7f56b800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.802+0000 7f56bf7fe640 1 -- 172.21.15.79:0/2081783587 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8015020 con 0x7f56c81031e0 2024-04-22T06:47:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.802+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c8199220 con 0x7f56c81031e0 2024-04-22T06:47:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.802+0000 7f56bf7fe640 1 -- 172.21.15.79:0/2081783587 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56b8011d40 con 0x7f56c81031e0 2024-04-22T06:47:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.802+0000 7f56bf7fe640 1 -- 172.21.15.79:0/2081783587 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b800f340 con 0x7f56c81031e0 2024-04-22T06:47:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.802+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c81a5530 con 0x7f56c81031e0 2024-04-22T06:47:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 Processor -- start 2024-04-22T06:47:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 -- start start 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 0x555ee1d57620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 0x555ee1d57fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d654f0 0x555ee1d678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x555ee1cbc8d0 con 0x555ee1d57b60 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555ee1c59e80 con 0x555ee1d588d0 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.805+0000 7f56cf818740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555ee1cb8890 con 0x555ee1d654f0 2024-04-22T06:47:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56bf7fe640 1 -- 172.21.15.79:0/2081783587 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f56b8018020 con 0x7f56c81031e0 2024-04-22T06:47:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 0x555ee1d57fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 0x555ee1d57fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46962/0 (socket says 172.21.15.79:46962) 2024-04-22T06:47:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 -- 172.21.15.79:0/2077034570 learned_addr learned my addr 172.21.15.79:0/2077034570 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 -- 172.21.15.79:0/2077034570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d654f0 msgr2=0x555ee1d678e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d654f0 0x555ee1d678e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56bf7fe640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b40a3dc0 0x7f56b40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 -- 172.21.15.79:0/2077034570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 msgr2=0x555ee1d57620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.806+0000 7f56c659c640 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 0x555ee1d57620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56c659c640 1 -- 172.21.15.79:0/2077034570 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ee1c7b320 con 0x555ee1d57b60 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56bf7fe640 1 -- 172.21.15.79:0/2081783587 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f56b80d0360 con 0x7f56c81031e0 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b40a3dc0 0x7f56b40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56c659c640 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 0x555ee1d57fe0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f56b80cda10 tx=0x7f56b80668f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a0adcb91621587 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56bcff9640 1 -- 172.21.15.79:0/2077034570 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8096e60 con 0x555ee1d57b60 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56bcff9640 1 -- 172.21.15.79:0/2077034570 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56b8066e00 con 0x555ee1d57b60 2024-04-22T06:47:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.807+0000 7f56bcff9640 1 -- 172.21.15.79:0/2077034570 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b809c0c0 con 0x555ee1d57b60 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 -- 172.21.15.79:0/2077034570 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 msgr2=0x555ee1d57fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b40a3dc0 0x7f56b40a6280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f56c002c860 tx=0x7f56c0005c50 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 0x555ee1d57fe0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f56b80cda10 tx=0x7f56b80668f0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 -- 172.21.15.79:0/2077034570 shutdown_connections 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d654f0 0x555ee1d678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d57b60 0x555ee1d57fe0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 --2- 172.21.15.79:0/2077034570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 0x555ee1d57620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 -- 172.21.15.79:0/2077034570 >> 172.21.15.79:0/2077034570 conn(0x555ee1d1ec90 msgr2=0x555ee1d52850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 -- 172.21.15.79:0/2077034570 shutdown_connections 2024-04-22T06:47:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.808+0000 7f56cf818740 1 -- 172.21.15.79:0/2077034570 wait complete. 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 -- start start 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 Processor -- start 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 0x555ee1d64e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 0x555ee1d62220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d627c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x555ee1d681c0 con 0x555ee1d654f0 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555ee1c44eb0 con 0x555ee1d588d0 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.809+0000 7f56cf818740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555ee1d68340 con 0x555ee1d57b60 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56cf818740 10 client.0 _ll_register_callbacks cb 0x555ee1c71b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d627c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46974/0 (socket says 172.21.15.79:46974) 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 -- 172.21.15.79:0/1448744867 learned_addr learned my addr 172.21.15.79:0/1448744867 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 -- 172.21.15.79:0/1448744867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 msgr2=0x555ee1d64e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 0x555ee1d64e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 -- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 msgr2=0x555ee1d62220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 0x555ee1d62220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 -- 172.21.15.79:0/1448744867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ee19cf210 con 0x555ee1d654f0 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d627c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f56c0003de0 tx=0x7f56c0003e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47da22e564ead724 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.810+0000 7f56adffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56adffb640 1 -- 172.21.15.79:0/1448744867 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c004f060 con 0x555ee1d654f0 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56adffb640 1 -- 172.21.15.79:0/1448744867 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f56c0004250 con 0x555ee1d654f0 2024-04-22T06:47:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56b8066e00 con 0x555ee1d654f0 2024-04-22T06:47:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56adffb640 1 -- 172.21.15.79:0/1448744867 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f56c0056050 con 0x555ee1d654f0 2024-04-22T06:47:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56cf818740 10 client.16437 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555ee1a9b620 con 0x555ee1d654f0 2024-04-22T06:47:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.811+0000 7f56adffb640 1 -- 172.21.15.79:0/1448744867 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c0032660 con 0x555ee1d654f0 2024-04-22T06:47:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56adffb640 1 -- 172.21.15.79:0/1448744867 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f56c003c030 con 0x555ee1d654f0 2024-04-22T06:47:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56cf818740 10 client.16437 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56cf818740 10 client.16437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56cf818740 20 client.16437 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56cf818740 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1d75520 0x555ee1d77950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56cf818740 4 client.16437 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.812+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f56b8066e00 con 0x555ee1d75520 2024-04-22T06:47:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.814+0000 7f56c659c640 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1d75520 0x555ee1d77950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.817+0000 7f56adffb640 10 client.16437 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.817+0000 7f56c659c640 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1d75520 0x555ee1d77950 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=724e43571716b9a4 server_cookie=6f51ff71236c6418 in_seq=0 out_seq=0 2024-04-22T06:47:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.819+0000 7f56adffb640 1 -- 172.21.15.79:0/1448744867 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f56b8066e00 con 0x555ee1d75520 2024-04-22T06:47:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.819+0000 7f56adffb640 10 client.16437 handle_command_reply: tid=0 2024-04-22T06:47:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.819+0000 7f56cf818740 1 client.16437 shutdown 2024-04-22T06:47:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 msgr2=0x555ee1d627c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d627c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f56c0003de0 tx=0x7f56c0003e10 comp rx=0 tx=0).stop 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 shutdown_connections 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1d75520 0x555ee1d77950 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d588d0 0x555ee1d62220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d627c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 --2- 172.21.15.79:0/1448744867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 0x555ee1d64e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 >> 172.21.15.79:0/1448744867 conn(0x555ee1d1ec90 msgr2=0x555ee1d545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56adffb640 0 client.16437 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 shutdown_connections 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 1 -- 172.21.15.79:0/1448744867 wait complete. 2024-04-22T06:47:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.820+0000 7f56cf818740 20 client.16437 trim_cache size 0 max 16384 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 Processor -- start 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 -- start start 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d57b60 0x555ee1d710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d588d0 0x555ee1d64950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d64e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81ae5d0 con 0x555ee1d654f0 2024-04-22T06:47:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81abea0 con 0x555ee1d57b60 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56cf818740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81a9cf0 con 0x555ee1d588d0 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56c6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d64e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56c6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d64e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46978/0 (socket says 172.21.15.79:46978) 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56c6d9d640 1 -- 172.21.15.79:0/970409947 learned_addr learned my addr 172.21.15.79:0/970409947 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56c6d9d640 1 -- 172.21.15.79:0/970409947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d588d0 msgr2=0x555ee1d64950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d588d0 0x555ee1d64950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.840+0000 7f56c6d9d640 1 -- 172.21.15.79:0/970409947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d57b60 msgr2=0x555ee1d710f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.841+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d57b60 0x555ee1d710f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.841+0000 7f56c6d9d640 1 -- 172.21.15.79:0/970409947 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ee1d67d40 con 0x555ee1d654f0 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.841+0000 7f56c6d9d640 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d64e90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f56c003f510 tx=0x7f56c002ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11efbf5ade6cc0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.841+0000 7f56affff640 1 -- 172.21.15.79:0/970409947 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c005e020 con 0x555ee1d654f0 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.841+0000 7f56affff640 1 -- 172.21.15.79:0/970409947 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c0004700 con 0x555ee1d654f0 2024-04-22T06:47:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.841+0000 7f56affff640 1 -- 172.21.15.79:0/970409947 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c004f040 con 0x555ee1d654f0 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 -- 172.21.15.79:0/970409947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 msgr2=0x555ee1d64e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d64e90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f56c003f510 tx=0x7f56c002ce00 comp rx=0 tx=0).stop 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 -- 172.21.15.79:0/970409947 shutdown_connections 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d654f0 0x555ee1d64e90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d588d0 0x555ee1d64950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 --2- 172.21.15.79:0/970409947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d57b60 0x555ee1d710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 -- 172.21.15.79:0/970409947 >> 172.21.15.79:0/970409947 conn(0x555ee1d1ec90 msgr2=0x555ee1d5d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.843+0000 7f56cf818740 1 -- 172.21.15.79:0/970409947 shutdown_connections 2024-04-22T06:47:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.844+0000 7f56cf818740 1 -- 172.21.15.79:0/970409947 wait complete. 2024-04-22T06:47:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.844+0000 7f56cf818740 1 -- start start 2024-04-22T06:47:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.844+0000 7f56cf818740 1 Processor -- start 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 0x555ee1d6d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 0x555ee1d6db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d654f0 0x555ee1d5e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81aa360 con 0x555ee1d588d0 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81ab7d0 con 0x555ee1d654f0 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56c81aaa30 con 0x555ee1d57b60 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56cf818740 10 client.0 _ll_register_callbacks cb 0x7f56c0037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 0x555ee1d6db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 0x555ee1d6db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46994/0 (socket says 172.21.15.79:46994) 2024-04-22T06:47:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.845+0000 7f56bffff640 1 -- 172.21.15.79:0/184094353 learned_addr learned my addr 172.21.15.79:0/184094353 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56bffff640 1 -- 172.21.15.79:0/184094353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 msgr2=0x555ee1d6d600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56bffff640 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 0x555ee1d6d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56bffff640 1 -- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d654f0 msgr2=0x555ee1d5e490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56bffff640 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d654f0 0x555ee1d5e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56bffff640 1 -- 172.21.15.79:0/184094353 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ee1d5e9d0 con 0x555ee1d588d0 2024-04-22T06:47:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56bffff640 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 0x555ee1d6db40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f56c8199ec0 tx=0x7f56b0009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac4a2999c0fa1e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56adffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.846+0000 7f56adffb640 1 -- 172.21.15.79:0/184094353 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b0011440 con 0x555ee1d588d0 2024-04-22T06:47:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.847+0000 7f56adffb640 1 -- 172.21.15.79:0/184094353 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f56b00115e0 con 0x555ee1d588d0 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.847+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56c0004700 con 0x555ee1d588d0 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.847+0000 7f56adffb640 1 -- 172.21.15.79:0/184094353 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f56b001f050 con 0x555ee1d588d0 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.847+0000 7f56cf818740 10 client.16443 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.847+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555ee1ddb390 con 0x555ee1d588d0 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56adffb640 1 -- 172.21.15.79:0/184094353 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56c0004700 con 0x555ee1d588d0 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56adffb640 1 -- 172.21.15.79:0/184094353 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f56b001a6e0 con 0x555ee1d588d0 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56cf818740 10 client.16443 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56cf818740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56cf818740 20 client.16443 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56cf818740 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1ddbc10 0x555ee1dde030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56cf818740 4 client.16443 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:47:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.848+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ee1cd0880 con 0x555ee1ddbc10 2024-04-22T06:47:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.850+0000 7f56c659c640 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1ddbc10 0x555ee1dde030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.852+0000 7f56adffb640 10 client.16443 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:05.852+0000 7f56c659c640 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1ddbc10 0x555ee1dde030 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f0719ff7a5fef8 server_cookie=f3d0e0055d075e52 in_seq=0 out_seq=0 2024-04-22T06:47:06.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:06 smithi193 ceph-mon[31815]: pgmap v87: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 571 MiB/s wr, 492 op/s 2024-04-22T06:47:06.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:06 smithi190 ceph-mon[31421]: pgmap v87: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 571 MiB/s wr, 492 op/s 2024-04-22T06:47:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:06 smithi079 ceph-mon[27946]: pgmap v87: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 571 MiB/s wr, 492 op/s 2024-04-22T06:47:08.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56adffb640 1 -- 172.21.15.79:0/184094353 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ee1cd0880 con 0x555ee1ddbc10 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56adffb640 10 client.16443 handle_command_reply: tid=0 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 client.16443 shutdown 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 msgr2=0x555ee1d6db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 0x555ee1d6db40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f56c8199ec0 tx=0x7f56b0009430 comp rx=0 tx=0).stop 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 shutdown_connections 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x555ee1ddbc10 0x555ee1dde030 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555ee1d654f0 0x555ee1d5e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56adffb640 0 client.16443 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555ee1d588d0 0x555ee1d6db40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 --2- 172.21.15.79:0/184094353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555ee1d57b60 0x555ee1d6d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.463+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 >> 172.21.15.79:0/184094353 conn(0x555ee1d1ec90 msgr2=0x555ee1d52380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:08.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.464+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 shutdown_connections 2024-04-22T06:47:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.464+0000 7f56cf818740 1 -- 172.21.15.79:0/184094353 wait complete. 2024-04-22T06:47:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.464+0000 7f56cf818740 20 client.16443 trim_cache size 0 max 16384 2024-04-22T06:47:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:47:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:47:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:47:08.464 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b40a3dc0 msgr2=0x7f56b40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b40a3dc0 0x7f56b40a6280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f56c002c860 tx=0x7f56c0005c50 comp rx=0 tx=0).stop 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 msgr2=0x7f56c819e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 0x7f56c819e5b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b450 tx=0x7f56b800b910 comp rx=0 tx=0).stop 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 shutdown_connections 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f56b40a3dc0 0x7f56b40a6280 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56c8103b00 0x7f56c819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c810c520 0x7f56c8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 --2- 172.21.15.79:0/2081783587 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f56c81031e0 0x7f56c819e5b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 >> 172.21.15.79:0/2081783587 conn(0x7f56c8077fd0 msgr2=0x7f56c80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 shutdown_connections 2024-04-22T06:47:08.466 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:08.465+0000 7f56ccc2c640 1 -- 172.21.15.79:0/2081783587 wait complete. 2024-04-22T06:47:08.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:08 smithi079 ceph-mon[27946]: pgmap v88: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 859 MiB/s wr, 869 op/s 2024-04-22T06:47:08.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:08 smithi193 ceph-mon[31815]: pgmap v88: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 859 MiB/s wr, 869 op/s 2024-04-22T06:47:08.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:08 smithi190 ceph-mon[31421]: pgmap v88: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 859 MiB/s wr, 869 op/s 2024-04-22T06:47:10.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:10 smithi193 ceph-mon[31815]: pgmap v89: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 569 MiB/s wr, 657 op/s 2024-04-22T06:47:10.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:10 smithi190 ceph-mon[31421]: pgmap v89: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 569 MiB/s wr, 657 op/s 2024-04-22T06:47:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:10 smithi079 ceph-mon[27946]: pgmap v89: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 569 MiB/s wr, 657 op/s 2024-04-22T06:47:12.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:12 smithi193 ceph-mon[31815]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 MiB/s wr, 1.86k op/s 2024-04-22T06:47:12.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:12 smithi190 ceph-mon[31421]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 MiB/s wr, 1.86k op/s 2024-04-22T06:47:12.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:12 smithi079 ceph-mon[27946]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 MiB/s wr, 1.86k op/s 2024-04-22T06:47:14.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:14 smithi193 ceph-mon[31815]: pgmap v91: 97 pgs: 97 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 651 MiB/s wr, 2.63k op/s 2024-04-22T06:47:14.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:14 smithi190 ceph-mon[31421]: pgmap v91: 97 pgs: 97 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 651 MiB/s wr, 2.63k op/s 2024-04-22T06:47:14.992 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:14 smithi079 ceph-mon[27946]: pgmap v91: 97 pgs: 97 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 651 MiB/s wr, 2.63k op/s 2024-04-22T06:47:15.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:47:15.197 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:47:15.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.461+0000 7f286543a640 1 -- 172.21.15.79:0/3275366115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 msgr2=0x7f286010e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.461+0000 7f286543a640 1 --2- 172.21.15.79:0/3275366115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 0x7f286010e270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ca0 tx=0x7f285802c120 comp rx=0 tx=0).stop 2024-04-22T06:47:15.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.464+0000 7f286543a640 1 -- 172.21.15.79:0/3275366115 shutdown_connections 2024-04-22T06:47:15.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.465+0000 7f286543a640 1 --2- 172.21.15.79:0/3275366115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f286011b730 0x7f286011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.465+0000 7f286543a640 1 --2- 172.21.15.79:0/3275366115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286010e7b0 0x7f286010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.465+0000 7f286543a640 1 --2- 172.21.15.79:0/3275366115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 0x7f286010e270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.465+0000 7f286543a640 1 -- 172.21.15.79:0/3275366115 >> 172.21.15.79:0/3275366115 conn(0x7f286006d070 msgr2=0x7f286006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.470+0000 7f286543a640 1 -- 172.21.15.79:0/3275366115 shutdown_connections 2024-04-22T06:47:15.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.470+0000 7f286543a640 1 -- 172.21.15.79:0/3275366115 wait complete. 2024-04-22T06:47:15.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.471+0000 7f286543a640 1 Processor -- start 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.471+0000 7f286543a640 1 -- start start 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.472+0000 7f286543a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f286010e7b0 0x7f2860119450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.472+0000 7f286543a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 0x7f2860119990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.472+0000 7f286543a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 0x7f28601126e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.472+0000 7f286543a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2860121060 con 0x7f286011b730 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.472+0000 7f286543a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2860120ee0 con 0x7f286010e7b0 2024-04-22T06:47:15.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.472+0000 7f286543a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28601211e0 con 0x7f28601106d0 2024-04-22T06:47:15.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.474+0000 7f285f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 0x7f28601126e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.474+0000 7f285f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 0x7f28601126e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33732/0 (socket says 172.21.15.79:33732) 2024-04-22T06:47:15.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.474+0000 7f285f7fe640 1 -- 172.21.15.79:0/734508130 learned_addr learned my addr 172.21.15.79:0/734508130 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:15.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.475+0000 7f285e7fc640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 0x7f2860119990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.477+0000 7f285f7fe640 1 -- 172.21.15.79:0/734508130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 msgr2=0x7f2860119990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.477+0000 7f285f7fe640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 0x7f2860119990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.477+0000 7f285f7fe640 1 -- 172.21.15.79:0/734508130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f286010e7b0 msgr2=0x7f2860119450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.477+0000 7f285f7fe640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f286010e7b0 0x7f2860119450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.477+0000 7f285f7fe640 1 -- 172.21.15.79:0/734508130 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2860112fa0 con 0x7f286011b730 2024-04-22T06:47:15.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.484+0000 7f285f7fe640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 0x7f28601126e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f285400b3c0 tx=0x7f285400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.488+0000 7f2847fff640 1 -- 172.21.15.79:0/734508130 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854011830 con 0x7f286011b730 2024-04-22T06:47:15.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.488+0000 7f2847fff640 1 -- 172.21.15.79:0/734508130 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2854016070 con 0x7f286011b730 2024-04-22T06:47:15.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.488+0000 7f2847fff640 1 -- 172.21.15.79:0/734508130 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854010de0 con 0x7f286011b730 2024-04-22T06:47:15.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.488+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2860113290 con 0x7f286011b730 2024-04-22T06:47:15.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.488+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2860117210 con 0x7f286011b730 2024-04-22T06:47:15.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.495+0000 7f2868021740 1 Processor -- start 2024-04-22T06:47:15.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 -- start start 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb24960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb2fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558aeca891e0 con 0x558aecb2da80 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558aeca26390 con 0x558aecb24ea0 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2868021740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558aeca851a0 con 0x558aecb24580 2024-04-22T06:47:15.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.499+0000 7f2847fff640 1 -- 172.21.15.79:0/734508130 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28540119d0 con 0x7f286011b730 2024-04-22T06:47:15.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.501+0000 7f2847fff640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28480a3f20 0x7f28480a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.501+0000 7f2847fff640 1 -- 172.21.15.79:0/734508130 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f28540cfed0 con 0x7f286011b730 2024-04-22T06:47:15.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.502+0000 7f285effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.502+0000 7f285effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58604/0 (socket says 172.21.15.79:58604) 2024-04-22T06:47:15.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.502+0000 7f285effd640 1 -- 172.21.15.79:0/264612936 learned_addr learned my addr 172.21.15.79:0/264612936 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:15.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.504+0000 7f285effd640 1 -- 172.21.15.79:0/264612936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 msgr2=0x558aecb24960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:15.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.504+0000 7f285effd640 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb24960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.504+0000 7f285effd640 1 -- 172.21.15.79:0/264612936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 msgr2=0x558aecb2fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.504+0000 7f285effd640 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb2fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.504+0000 7f285effd640 1 -- 172.21.15.79:0/264612936 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aec8657d0 con 0x558aecb24ea0 2024-04-22T06:47:15.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.504+0000 7f285f7fe640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28480a3f20 0x7f28480a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.507+0000 7f285f7fe640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28480a3f20 0x7f28480a63e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f285400bd80 tx=0x7f285401b000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.510+0000 7f285effd640 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2d540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2858009dc0 tx=0x7f28580097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a36ad658e8b7991 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.511+0000 7f28457fa640 1 -- 172.21.15.79:0/264612936 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858041020 con 0x558aecb24ea0 2024-04-22T06:47:15.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.511+0000 7f28457fa640 1 -- 172.21.15.79:0/264612936 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28580329d0 con 0x558aecb24ea0 2024-04-22T06:47:15.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.511+0000 7f28457fa640 1 -- 172.21.15.79:0/264612936 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858039040 con 0x558aecb24ea0 2024-04-22T06:47:15.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.514+0000 7f2868021740 1 -- 172.21.15.79:0/264612936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 msgr2=0x558aecb2d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.514+0000 7f2868021740 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2d540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2858009dc0 tx=0x7f28580097e0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.516+0000 7f2868021740 1 -- 172.21.15.79:0/264612936 shutdown_connections 2024-04-22T06:47:15.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.516+0000 7f2868021740 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2d540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.516+0000 7f2868021740 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb2fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.516+0000 7f2868021740 1 --2- 172.21.15.79:0/264612936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb24960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.516+0000 7f2868021740 1 -- 172.21.15.79:0/264612936 >> 172.21.15.79:0/264612936 conn(0x558aecb1ecb0 msgr2=0x558aecb1f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.528 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.530+0000 7f2868021740 1 -- 172.21.15.79:0/264612936 shutdown_connections 2024-04-22T06:47:15.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.535+0000 7f2868021740 1 -- 172.21.15.79:0/264612936 wait complete. 2024-04-22T06:47:15.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.535+0000 7f2868021740 1 -- start start 2024-04-22T06:47:15.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.535+0000 7f2868021740 1 Processor -- start 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb2abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 0x558aecb2b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb2da80 0x558aecb2b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558aecb30810 con 0x558aecb24ea0 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558aeca422f0 con 0x558aecb24580 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558aecb30990 con 0x558aecb2da80 2024-04-22T06:47:15.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.536+0000 7f2868021740 10 client.0 _ll_register_callbacks cb 0x558aeca4f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:15.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.539+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb2abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.539+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb2da80 0x558aecb2b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.539+0000 7f285effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 0x558aecb2b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.539+0000 7f285effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 0x558aecb2b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33758/0 (socket says 172.21.15.79:33758) 2024-04-22T06:47:15.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.539+0000 7f285effd640 1 -- 172.21.15.79:0/2083636940 learned_addr learned my addr 172.21.15.79:0/2083636940 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:15.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.541+0000 7f285effd640 1 -- 172.21.15.79:0/2083636940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb2da80 msgr2=0x558aecb2b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.541+0000 7f285effd640 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb2da80 0x558aecb2b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.541+0000 7f285effd640 1 -- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 msgr2=0x558aecb2abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.541+0000 7f285effd640 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb2abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.541+0000 7f285effd640 1 -- 172.21.15.79:0/2083636940 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aec79b3f0 con 0x558aecb24ea0 2024-04-22T06:47:15.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.542+0000 7f285effd640 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 0x558aecb2b120 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ca0 tx=0x7f2858003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789d1c19c2cd317b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.542+0000 7f2842ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:15.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.543+0000 7f2842ff5640 1 -- 172.21.15.79:0/2083636940 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858039050 con 0x558aecb24ea0 2024-04-22T06:47:15.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.543+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28580329d0 con 0x558aecb24ea0 2024-04-22T06:47:15.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.543+0000 7f2842ff5640 1 -- 172.21.15.79:0/2083636940 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f2858003f40 con 0x558aecb24ea0 2024-04-22T06:47:15.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.544+0000 7f2842ff5640 1 -- 172.21.15.79:0/2083636940 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f2858004140 con 0x558aecb24ea0 2024-04-22T06:47:15.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.546+0000 7f2868021740 10 client.16449 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:15.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.546+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558aec75b0f0 con 0x558aecb24ea0 2024-04-22T06:47:15.545 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.547+0000 7f2842ff5640 1 -- 172.21.15.79:0/2083636940 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28580329d0 con 0x558aecb24ea0 2024-04-22T06:47:15.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.547+0000 7f2842ff5640 1 -- 172.21.15.79:0/2083636940 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f285803c030 con 0x558aecb24ea0 2024-04-22T06:47:15.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.547+0000 7f2868021740 10 client.16449 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:15.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.547+0000 7f2868021740 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:15.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.547+0000 7f2868021740 20 client.16449 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:15.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.548+0000 7f2868021740 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecb3db20 0x558aecb3ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.548+0000 7f2868021740 4 client.16449 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:15.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.548+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558aeca96c00 con 0x558aecb3db20 2024-04-22T06:47:15.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.551+0000 7f285e7fc640 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecb3db20 0x558aecb3ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.561+0000 7f285e7fc640 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecb3db20 0x558aecb3ff40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ec6bf55995db8ad server_cookie=b03a4d19465117d4 in_seq=0 out_seq=0 2024-04-22T06:47:15.559 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.561+0000 7f2842ff5640 10 client.16449 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:15.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2842ff5640 1 -- 172.21.15.79:0/2083636940 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558aeca96c00 con 0x558aecb3db20 2024-04-22T06:47:15.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2842ff5640 10 client.16449 handle_command_reply: tid=0 2024-04-22T06:47:15.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 client.16449 shutdown 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 msgr2=0x558aecb2b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 0x558aecb2b120 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ca0 tx=0x7f2858003bd0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 shutdown_connections 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecb3db20 0x558aecb3ff40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb2da80 0x558aecb2b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb24ea0 0x558aecb2b120 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 --2- 172.21.15.79:0/2083636940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb2abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.570+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 >> 172.21.15.79:0/2083636940 conn(0x558aecb1ecb0 msgr2=0x558aecb20a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.571+0000 7f2842ff5640 0 client.16449 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:15.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.572+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 shutdown_connections 2024-04-22T06:47:15.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.575+0000 7f2868021740 1 -- 172.21.15.79:0/2083636940 wait complete. 2024-04-22T06:47:15.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.576+0000 7f2868021740 20 client.16449 trim_cache size 0 max 16384 2024-04-22T06:47:15.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.630+0000 7f2868021740 1 Processor -- start 2024-04-22T06:47:15.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 -- start start 2024-04-22T06:47:15.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb396d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.633 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb45850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f28601c7df0 con 0x558aecb2da80 2024-04-22T06:47:15.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f28601c56c0 con 0x558aecb24ea0 2024-04-22T06:47:15.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.635+0000 7f2868021740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28601c3510 con 0x558aecb24580 2024-04-22T06:47:15.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.636+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb45850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.637+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb45850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33766/0 (socket says 172.21.15.79:33766) 2024-04-22T06:47:15.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.637+0000 7f285e7fc640 1 -- 172.21.15.79:0/2259353128 learned_addr learned my addr 172.21.15.79:0/2259353128 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:15.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.638+0000 7f285e7fc640 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.638+0000 7f285effd640 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.638+0000 7f285e7fc640 1 -- 172.21.15.79:0/2259353128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 msgr2=0x558aecb396d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.638+0000 7f285e7fc640 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb396d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.638+0000 7f285e7fc640 1 -- 172.21.15.79:0/2259353128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 msgr2=0x558aecb2b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.638+0000 7f285e7fc640 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.639+0000 7f285e7fc640 1 -- 172.21.15.79:0/2259353128 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aecb23d20 con 0x558aecb2da80 2024-04-22T06:47:15.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.639+0000 7f285e7fc640 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb45850 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x558aecb12c40 tx=0x7f2850005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb06f9e521e779e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.641+0000 7f2844ff9640 1 -- 172.21.15.79:0/2259353128 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850005220 con 0x558aecb2da80 2024-04-22T06:47:15.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.641+0000 7f2844ff9640 1 -- 172.21.15.79:0/2259353128 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2850004c50 con 0x558aecb2da80 2024-04-22T06:47:15.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.642+0000 7f2844ff9640 1 -- 172.21.15.79:0/2259353128 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850012a40 con 0x558aecb2da80 2024-04-22T06:47:15.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.642+0000 7f2868021740 1 -- 172.21.15.79:0/2259353128 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 msgr2=0x558aecb45850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.642+0000 7f2868021740 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb45850 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x558aecb12c40 tx=0x7f2850005bd0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.645+0000 7f2868021740 1 -- 172.21.15.79:0/2259353128 shutdown_connections 2024-04-22T06:47:15.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.645+0000 7f2868021740 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24ea0 0x558aecb2b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.645+0000 7f2868021740 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb45850 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.645+0000 7f2868021740 1 --2- 172.21.15.79:0/2259353128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24580 0x558aecb396d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.645+0000 7f2868021740 1 -- 172.21.15.79:0/2259353128 >> 172.21.15.79:0/2259353128 conn(0x558aecb1ecb0 msgr2=0x558aecb1dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.646+0000 7f2868021740 1 -- 172.21.15.79:0/2259353128 shutdown_connections 2024-04-22T06:47:15.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.646+0000 7f2868021740 1 -- 172.21.15.79:0/2259353128 wait complete. 2024-04-22T06:47:15.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.647+0000 7f2868021740 1 -- start start 2024-04-22T06:47:15.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.647+0000 7f2868021740 1 Processor -- start 2024-04-22T06:47:15.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb36130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24ea0 0x558aecb314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb31be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f28601c3b80 con 0x558aecb2da80 2024-04-22T06:47:15.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f28601c4ff0 con 0x558aecb24580 2024-04-22T06:47:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28601c4250 con 0x558aecb24ea0 2024-04-22T06:47:15.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.648+0000 7f2868021740 10 client.0 _ll_register_callbacks cb 0x7f2858009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:15.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.652+0000 7f285effd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24ea0 0x558aecb314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.652+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb31be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.652+0000 7f285e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb31be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33772/0 (socket says 172.21.15.79:33772) 2024-04-22T06:47:15.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.652+0000 7f285e7fc640 1 -- 172.21.15.79:0/1505738700 learned_addr learned my addr 172.21.15.79:0/1505738700 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:15.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.652+0000 7f285e7fc640 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb36130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.657+0000 7f285effd640 1 -- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 msgr2=0x558aecb36130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.657+0000 7f285effd640 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb36130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.657+0000 7f285effd640 1 -- 172.21.15.79:0/1505738700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 msgr2=0x558aecb31be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:15.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.657+0000 7f285effd640 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb31be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.657+0000 7f285effd640 1 -- 172.21.15.79:0/1505738700 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aecba30c0 con 0x558aecb24ea0 2024-04-22T06:47:15.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.662+0000 7f285effd640 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24ea0 0x558aecb314a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f28580469a0 tx=0x7f2858047dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d43516fc021f0301 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.662+0000 7f2842ff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.663+0000 7f2842ff5640 1 -- 172.21.15.79:0/1505738700 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858039070 con 0x558aecb24ea0 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.663+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2850004c50 con 0x558aecb24ea0 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.664+0000 7f2842ff5640 1 -- 172.21.15.79:0/1505738700 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f2858003e50 con 0x558aecb24ea0 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.665+0000 7f2842ff5640 1 -- 172.21.15.79:0/1505738700 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f2858041070 con 0x558aecb24ea0 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.665+0000 7f2868021740 10 client.25606 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:15.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.665+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558aecba4270 con 0x558aecb24ea0 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2842ff5640 1 -- 172.21.15.79:0/1505738700 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2850004c50 con 0x558aecb24ea0 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2842ff5640 1 -- 172.21.15.79:0/1505738700 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f285803c040 con 0x558aecb24ea0 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2868021740 10 client.25606 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2868021740 10 client.25606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2868021740 20 client.25606 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2868021740 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecba4990 0x558aecba6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2868021740 4 client.25606 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:47:15.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.670+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558aeca90a70 con 0x558aecba4990 2024-04-22T06:47:15.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.676+0000 7f285e7fc640 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecba4990 0x558aecba6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.688+0000 7f2842ff5640 10 client.25606 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:15.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.688+0000 7f285e7fc640 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecba4990 0x558aecba6db0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d8c0429109b84fe server_cookie=27a4b2bcbd14144 in_seq=0 out_seq=0 2024-04-22T06:47:15.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.689+0000 7f2842ff5640 1 -- 172.21.15.79:0/1505738700 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558aeca90a70 con 0x558aecba4990 2024-04-22T06:47:15.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.689+0000 7f2842ff5640 10 client.25606 handle_command_reply: tid=0 2024-04-22T06:47:15.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.689+0000 7f2868021740 1 client.25606 shutdown 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24ea0 msgr2=0x558aecb314a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24ea0 0x558aecb314a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f28580469a0 tx=0x7f2858047dd0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 shutdown_connections 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558aecba4990 0x558aecba6db0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558aecb2da80 0x558aecb31be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558aecb24ea0 0x558aecb314a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 --2- 172.21.15.79:0/1505738700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558aecb24580 0x558aecb36130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.691+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 >> 172.21.15.79:0/1505738700 conn(0x558aecb1ecb0 msgr2=0x558aecb1fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.692+0000 7f2842ff5640 0 client.25606 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:15.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.692+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 shutdown_connections 2024-04-22T06:47:15.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.692+0000 7f2868021740 1 -- 172.21.15.79:0/1505738700 wait complete. 2024-04-22T06:47:15.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.694+0000 7f2868021740 20 client.25606 trim_cache size 0 max 16384 2024-04-22T06:47:15.695 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:15.695 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:47:15.695 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:47:15.695 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:15.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.707+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28480a3f20 msgr2=0x7f28480a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.707+0000 7f286543a640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28480a3f20 0x7f28480a63e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f285400bd80 tx=0x7f285401b000 comp rx=0 tx=0).stop 2024-04-22T06:47:15.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.707+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 msgr2=0x7f28601126e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.707+0000 7f286543a640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 0x7f28601126e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f285400b3c0 tx=0x7f285400b880 comp rx=0 tx=0).stop 2024-04-22T06:47:15.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.709+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 shutdown_connections 2024-04-22T06:47:15.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.709+0000 7f286543a640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f286011b730 0x7f28601126e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.709+0000 7f286543a640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28601106d0 0x7f2860119990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.709+0000 7f286543a640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f28480a3f20 0x7f28480a63e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.709+0000 7f286543a640 1 --2- 172.21.15.79:0/734508130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f286010e7b0 0x7f2860119450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.709+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 >> 172.21.15.79:0/734508130 conn(0x7f286006d070 msgr2=0x7f2860072a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.712+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 shutdown_connections 2024-04-22T06:47:15.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.712+0000 7f286543a640 1 -- 172.21.15.79:0/734508130 wait complete. 2024-04-22T06:47:15.742 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:47:15.742 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:47:15.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.984+0000 7f970d123640 1 -- 172.21.15.79:0/2538543476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f970810eb50 msgr2=0x7f970811d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.984+0000 7f970d123640 1 --2- 172.21.15.79:0/2538543476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f970810eb50 0x7f970811d740 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9700009900 tx=0x7f970002bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:15.984 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.985+0000 7f970d123640 1 -- 172.21.15.79:0/2538543476 shutdown_connections 2024-04-22T06:47:15.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.985+0000 7f970d123640 1 --2- 172.21.15.79:0/2538543476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810e030 0x7f970810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.985+0000 7f970d123640 1 --2- 172.21.15.79:0/2538543476 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f970810eb50 0x7f970811d740 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.985+0000 7f970d123640 1 --2- 172.21.15.79:0/2538543476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97081104d0 0x7f97081108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.985+0000 7f970d123640 1 -- 172.21.15.79:0/2538543476 >> 172.21.15.79:0/2538543476 conn(0x7f970806ce60 msgr2=0x7f970806d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:15.985 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.986+0000 7f970d123640 1 -- 172.21.15.79:0/2538543476 shutdown_connections 2024-04-22T06:47:15.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.989+0000 7f970d123640 1 -- 172.21.15.79:0/2538543476 wait complete. 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.990+0000 7f970d123640 1 Processor -- start 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.990+0000 7f970d123640 1 -- start start 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.991+0000 7f970d123640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f970810e030 0x7f97081a7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.991+0000 7f970d123640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810eb50 0x7f97081a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.991+0000 7f970d123640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 0x7f97081abc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.991+0000 7f970d123640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9708120e30 con 0x7f97081104d0 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.991+0000 7f970d123640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9708120cb0 con 0x7f970810eb50 2024-04-22T06:47:15.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.991+0000 7f970d123640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9708120fb0 con 0x7f970810e030 2024-04-22T06:47:15.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.993+0000 7f970c922640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 0x7f97081abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.993+0000 7f970c922640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 0x7f97081abc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33794/0 (socket says 172.21.15.79:33794) 2024-04-22T06:47:15.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.993+0000 7f970c922640 1 -- 172.21.15.79:0/3340298432 learned_addr learned my addr 172.21.15.79:0/3340298432 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:15.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.994+0000 7f97077fe640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810eb50 0x7f97081a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.994+0000 7f9707fff640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f970810e030 0x7f97081a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:15.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f970c922640 1 -- 172.21.15.79:0/3340298432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f970810e030 msgr2=0x7f97081a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f970c922640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f970810e030 0x7f97081a7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f970c922640 1 -- 172.21.15.79:0/3340298432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810eb50 msgr2=0x7f97081a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:15.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f970c922640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810eb50 0x7f97081a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:15.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f970c922640 1 -- 172.21.15.79:0/3340298432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97081ac360 con 0x7f97081104d0 2024-04-22T06:47:15.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f9707fff640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f970810e030 0x7f97081a7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:47:15.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.996+0000 7f97077fe640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810eb50 0x7f97081a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:15.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.997+0000 7f970c922640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 0x7f97081abc30 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9700006010 tx=0x7f97000048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:15.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.997+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700042020 con 0x7f97081104d0 2024-04-22T06:47:15.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.997+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9700004250 con 0x7f97081104d0 2024-04-22T06:47:15.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.997+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700005220 con 0x7f97081104d0 2024-04-22T06:47:15.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.997+0000 7f970d123640 1 -- 172.21.15.79:0/3340298432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97081ac5f0 con 0x7f97081104d0 2024-04-22T06:47:15.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:15.997+0000 7f970d123640 1 -- 172.21.15.79:0/3340298432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97081b3ee0 con 0x7f97081104d0 2024-04-22T06:47:16.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.001+0000 7f970d123640 1 -- 172.21.15.79:0/3340298432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f4005290 con 0x7f97081104d0 2024-04-22T06:47:16.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.003+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9700045020 con 0x7f97081104d0 2024-04-22T06:47:16.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.004+0000 7f97057fa640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f96f00a3e70 0x7f96f00a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.005+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f9700004400 con 0x7f97081104d0 2024-04-22T06:47:16.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.005+0000 7f9707fff640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f96f00a3e70 0x7f96f00a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.008+0000 7f9707fff640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f96f00a3e70 0x7f96f00a6330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009fa0 tx=0x7f96f8009570 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.015+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f970004c020 con 0x7f97081104d0 2024-04-22T06:47:16.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.452+0000 7f970d123640 1 -- 172.21.15.79:0/3340298432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96f4005850 con 0x7f97081104d0 2024-04-22T06:47:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.455+0000 7f97057fa640 1 -- 172.21.15.79:0/3340298432 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7f97000d6e80 con 0x7f97081104d0 2024-04-22T06:47:16.458 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:47:16.458 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:46:45.384403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25265,"mds_2":25261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":35,"state":"up:active","state_seq":22,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25261,24467,25265]},"id":1}]} 2024-04-22T06:47:16.459 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 40 2024-04-22T06:47:16.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 -- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f96f00a3e70 msgr2=0x7f96f00a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f96f00a3e70 0x7f96f00a6330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009fa0 tx=0x7f96f8009570 comp rx=0 tx=0).stop 2024-04-22T06:47:16.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 -- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 msgr2=0x7f97081abc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 0x7f97081abc30 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9700006010 tx=0x7f97000048f0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 -- 172.21.15.79:0/3340298432 shutdown_connections 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f96f00a3e70 0x7f96f00a6330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f970810eb50 0x7f97081a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f97081104d0 0x7f97081abc30 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 --2- 172.21.15.79:0/3340298432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f970810e030 0x7f97081a7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 -- 172.21.15.79:0/3340298432 >> 172.21.15.79:0/3340298432 conn(0x7f970806ce60 msgr2=0x7f9708071260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 -- 172.21.15.79:0/3340298432 shutdown_connections 2024-04-22T06:47:16.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.463+0000 7f96eeffd640 1 -- 172.21.15.79:0/3340298432 wait complete. 2024-04-22T06:47:16.498 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 -- 172.21.15.79:0/2177422287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea041094d0 msgr2=0x7fea041110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 --2- 172.21.15.79:0/2177422287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea041094d0 0x7fea041110a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009ca0 tx=0x7fe9fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 -- 172.21.15.79:0/2177422287 shutdown_connections 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 --2- 172.21.15.79:0/2177422287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea04104db0 0x7fea04108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 --2- 172.21.15.79:0/2177422287 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea041094d0 0x7fea041110a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 --2- 172.21.15.79:0/2177422287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea04104400 0x7fea041047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 -- 172.21.15.79:0/2177422287 >> 172.21.15.79:0/2177422287 conn(0x7fea040ffce0 msgr2=0x7fea04102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.678+0000 7fea0a715640 1 -- 172.21.15.79:0/2177422287 shutdown_connections 2024-04-22T06:47:16.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.679+0000 7fea0a715640 1 -- 172.21.15.79:0/2177422287 wait complete. 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.679+0000 7fea0a715640 1 Processor -- start 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.679+0000 7fea0a715640 1 -- start start 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea0a715640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 0x7fea041a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea0a715640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea041094d0 0x7fea041a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea0a715640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea0419a8b0 0x7fea0419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea0a715640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea04114510 con 0x7fea04104400 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea0a715640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea04114390 con 0x7fea0419a8b0 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea0a715640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea04114690 con 0x7fea041094d0 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 0x7fea041a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 0x7fea041a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33808/0 (socket says 172.21.15.79:33808) 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 -- 172.21.15.79:0/2602901518 learned_addr learned my addr 172.21.15.79:0/2602901518 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 -- 172.21.15.79:0/2602901518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea041094d0 msgr2=0x7fea041a0c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea041094d0 0x7fea041a0c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 -- 172.21.15.79:0/2602901518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea0419a8b0 msgr2=0x7fea0419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea0419a8b0 0x7fea0419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.680+0000 7fea03fff640 1 -- 172.21.15.79:0/2602901518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0419b3d0 con 0x7fea04104400 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.681+0000 7fea03fff640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 0x7fea041a0700 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400b4a0 tx=0x7fe9f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.681+0000 7fea017fa640 1 -- 172.21.15.79:0/2602901518 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4014070 con 0x7fea04104400 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.681+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea0419b6c0 con 0x7fea04104400 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.681+0000 7fea017fa640 1 -- 172.21.15.79:0/2602901518 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f4010ac0 con 0x7fea04104400 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.681+0000 7fea017fa640 1 -- 172.21.15.79:0/2602901518 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f401e060 con 0x7fea04104400 2024-04-22T06:47:16.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.681+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea041a7960 con 0x7fea04104400 2024-04-22T06:47:16.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.685+0000 7fea017fa640 1 -- 172.21.15.79:0/2602901518 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe9f4017070 con 0x7fea04104400 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.685+0000 7fea0d3e6740 1 Processor -- start 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.685+0000 7fea0d3e6740 1 -- start start 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183abe80 0x55ff183b4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183b4a00 0x55ff183b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ff183101f0 con 0x55ff183ab4d0 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ff182ad4b0 con 0x55ff183b4a00 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff1830c1b0 con 0x55ff183abe80 2024-04-22T06:47:16.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea017fa640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9ec0a3da0 0x7fe9ec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183ab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33812/0 (socket says 172.21.15.79:33812) 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 -- 172.21.15.79:0/3455226196 learned_addr learned my addr 172.21.15.79:0/3455226196 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9ec0a3da0 0x7fe9ec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 -- 172.21.15.79:0/3455226196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183abe80 msgr2=0x55ff183b4430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183abe80 0x55ff183b4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 -- 172.21.15.79:0/3455226196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183b4a00 msgr2=0x55ff183b6e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183b4a00 0x55ff183b6e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea017fa640 1 -- 172.21.15.79:0/2602901518 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe9f40cfb70 con 0x7fea04104400 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 -- 172.21.15.79:0/3455226196 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff1829f1b0 con 0x55ff183ab4d0 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.686+0000 7fea08cb2640 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183ab8b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009a60 tx=0x7fe9fc0324c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4daafaa5f1b0bc40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.687+0000 7fea08cb2640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9ec0a3da0 0x7fe9ec0a6260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc037040 tx=0x7fe9fc040000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.687+0000 7fe9ea7fc640 1 -- 172.21.15.79:0/3455226196 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc050070 con 0x55ff183ab4d0 2024-04-22T06:47:16.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.687+0000 7fe9ea7fc640 1 -- 172.21.15.79:0/3455226196 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9fc047030 con 0x55ff183ab4d0 2024-04-22T06:47:16.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.687+0000 7fe9ea7fc640 1 -- 172.21.15.79:0/3455226196 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc039040 con 0x55ff183ab4d0 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3455226196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 msgr2=0x55ff183ab8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183ab8b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009a60 tx=0x7fe9fc0324c0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3455226196 shutdown_connections 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183b4a00 0x55ff183b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183abe80 0x55ff183b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3455226196 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183ab8b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3455226196 >> 172.21.15.79:0/3455226196 conn(0x55ff183a5ba0 msgr2=0x55ff183a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3455226196 shutdown_connections 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.688+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3455226196 wait complete. 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 -- start start 2024-04-22T06:47:16.690 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 Processor -- start 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55ff183b7910 con 0x55ff183abe80 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ff182c9500 con 0x55ff183ab4d0 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff183b7a90 con 0x55ff183b4a00 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea0d3e6740 10 client.0 _ll_register_callbacks cb 0x55ff18297b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea037fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.689+0000 7fea037fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33814/0 (socket says 172.21.15.79:33814) 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 -- 172.21.15.79:0/3616086501 learned_addr learned my addr 172.21.15.79:0/3616086501 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 -- 172.21.15.79:0/3616086501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 msgr2=0x55ff183b1d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b1d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 -- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 msgr2=0x55ff183b4380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183b4380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 -- 172.21.15.79:0/3616086501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff180223f0 con 0x55ff183abe80 2024-04-22T06:47:16.691 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea037fe640 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183b1710 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009da0 tx=0x7fe9f800b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239fd9d098ab4f9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fe9e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fe9e37fe640 1 -- 172.21.15.79:0/3616086501 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8015020 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fe9e37fe640 1 -- 172.21.15.79:0/3616086501 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe9f8011420 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.690+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9fc047030 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fe9e37fe640 1 -- 172.21.15.79:0/3616086501 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe9f8020050 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 10 client.16473 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ff180ec7d0 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fe9e37fe640 1 -- 172.21.15.79:0/3616086501 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9fc047030 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fe9e37fe640 1 -- 172.21.15.79:0/3616086501 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fe9f8019860 con 0x55ff183abe80 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 10 client.16473 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 20 client.16473 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff183c4d30 0x55ff183c7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 4 client.16473 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:16.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.691+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff182af930 con 0x55ff183c4d30 2024-04-22T06:47:16.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.693+0000 7fea03fff640 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff183c4d30 0x55ff183c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.698+0000 7fea03fff640 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff183c4d30 0x55ff183c7190 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e7561a87ee82729 server_cookie=827e8ed87bb0657b in_seq=0 out_seq=0 2024-04-22T06:47:16.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.698+0000 7fe9e37fe640 10 client.16473 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:16.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.700+0000 7fe9e37fe640 1 -- 172.21.15.79:0/3616086501 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff182af930 con 0x55ff183c4d30 2024-04-22T06:47:16.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.700+0000 7fe9e37fe640 10 client.16473 handle_command_reply: tid=0 2024-04-22T06:47:16.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.700+0000 7fea0d3e6740 1 client.16473 shutdown 2024-04-22T06:47:16.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.700+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 msgr2=0x55ff183b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.700+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183b1710 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009da0 tx=0x7fe9f800b4b0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 shutdown_connections 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff183c4d30 0x55ff183c7190 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183b1710 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/3616086501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 >> 172.21.15.79:0/3616086501 conn(0x55ff183a5ba0 msgr2=0x55ff183a78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fe9e37fe640 0 client.16473 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:16.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 shutdown_connections 2024-04-22T06:47:16.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 1 -- 172.21.15.79:0/3616086501 wait complete. 2024-04-22T06:47:16.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.701+0000 7fea0d3e6740 20 client.16473 trim_cache size 0 max 16384 2024-04-22T06:47:16.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 Processor -- start 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 -- start start 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea041b29b0 con 0x55ff183abe80 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea041b0280 con 0x55ff183ab4d0 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea041ae0d0 con 0x55ff183b4a00 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea037fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea037fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33826/0 (socket says 172.21.15.79:33826) 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea037fe640 1 -- 172.21.15.79:0/2399087390 learned_addr learned my addr 172.21.15.79:0/2399087390 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea08cb2640 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.717+0000 7fea03fff640 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fea037fe640 1 -- 172.21.15.79:0/2399087390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 msgr2=0x55ff183b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fea037fe640 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fea037fe640 1 -- 172.21.15.79:0/2399087390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 msgr2=0x55ff183c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fea037fe640 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fea037fe640 1 -- 172.21.15.79:0/2399087390 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff183b2090 con 0x55ff183abe80 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fea037fe640 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183a4a80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009160 tx=0x7fe9f8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11472b3c69b4525 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fe9e9ffb640 1 -- 172.21.15.79:0/2399087390 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8015040 con 0x55ff183abe80 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fe9e9ffb640 1 -- 172.21.15.79:0/2399087390 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f801cc40 con 0x55ff183abe80 2024-04-22T06:47:16.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.718+0000 7fe9e9ffb640 1 -- 172.21.15.79:0/2399087390 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f80198e0 con 0x55ff183abe80 2024-04-22T06:47:16.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 -- 172.21.15.79:0/2399087390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 msgr2=0x55ff183a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183a4a80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009160 tx=0x7fe9f8009190 comp rx=0 tx=0).stop 2024-04-22T06:47:16.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 -- 172.21.15.79:0/2399087390 shutdown_connections 2024-04-22T06:47:16.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183b3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183abe80 0x55ff183a4a80 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/2399087390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183ab4d0 0x55ff183c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 -- 172.21.15.79:0/2399087390 >> 172.21.15.79:0/2399087390 conn(0x55ff183a5ba0 msgr2=0x55ff183a4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 -- 172.21.15.79:0/2399087390 shutdown_connections 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 -- 172.21.15.79:0/2399087390 wait complete. 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.719+0000 7fea0d3e6740 1 -- start start 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 Processor -- start 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183bcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183abe80 0x55ff183bd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183addd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fea041ae740 con 0x55ff183ab4d0 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea041afbb0 con 0x55ff183abe80 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea041aee10 con 0x55ff183b4a00 2024-04-22T06:47:16.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea0d3e6740 10 client.0 _ll_register_callbacks cb 0x7fe9f8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183bcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183bcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33838/0 (socket says 172.21.15.79:33838) 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 -- 172.21.15.79:0/1635006869 learned_addr learned my addr 172.21.15.79:0/1635006869 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea08cb2640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 -- 172.21.15.79:0/1635006869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 msgr2=0x55ff183addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 -- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183abe80 msgr2=0x55ff183bd220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.720+0000 7fea03fff640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183abe80 0x55ff183bd220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fea03fff640 1 -- 172.21.15.79:0/1635006869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff183ae600 con 0x55ff183ab4d0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fea08cb2640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fea03fff640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183bcce0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40cd360 tx=0x7fe9f4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f03cdaf695bab4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fe9e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fe9e37fe640 1 -- 172.21.15.79:0/1635006869 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f40b5060 con 0x55ff183ab4d0 2024-04-22T06:47:16.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fe9e37fe640 1 -- 172.21.15.79:0/1635006869 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe9f4099e00 con 0x55ff183ab4d0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f801cc40 con 0x55ff183ab4d0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fe9e37fe640 1 -- 172.21.15.79:0/1635006869 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe9f4010c70 con 0x55ff183ab4d0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fea0d3e6740 10 client.16479 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ff1842aef0 con 0x55ff183ab4d0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.721+0000 7fe9e37fe640 1 -- 172.21.15.79:0/1635006869 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f801cc40 con 0x55ff183ab4d0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fe9e37fe640 1 -- 172.21.15.79:0/1635006869 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fe9f4017b60 con 0x55ff183ab4d0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea0d3e6740 10 client.16479 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea0d3e6740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea0d3e6740 20 client.16479 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff1842b6a0 0x55ff1842dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea0d3e6740 4 client.16479 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff1831dc10 con 0x55ff1842b6a0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea037fe640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff1842b6a0 0x55ff1842dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fe9e37fe640 10 client.16479 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:16.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.722+0000 7fea037fe640 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff1842b6a0 0x55ff1842dac0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26e30d4aed6fe3de server_cookie=272114c83d33053f in_seq=0 out_seq=0 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fe9e37fe640 1 -- 172.21.15.79:0/1635006869 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff1831dc10 con 0x55ff1842b6a0 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fe9e37fe640 10 client.16479 handle_command_reply: tid=0 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 client.16479 shutdown 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 msgr2=0x55ff183bcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183bcce0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40cd360 tx=0x7fe9f4000bc0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 shutdown_connections 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55ff1842b6a0 0x55ff1842dac0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fe9e37fe640 0 client.16479 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff183abe80 0x55ff183bd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff183b4a00 0x55ff183addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 --2- 172.21.15.79:0/1635006869 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55ff183ab4d0 0x55ff183bcce0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 >> 172.21.15.79:0/1635006869 conn(0x55ff183a5ba0 msgr2=0x55ff183a4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 shutdown_connections 2024-04-22T06:47:16.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 1 -- 172.21.15.79:0/1635006869 wait complete. 2024-04-22T06:47:16.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.723+0000 7fea0d3e6740 20 client.16479 trim_cache size 0 max 16384 2024-04-22T06:47:16.727 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:47:16.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9ec0a3da0 msgr2=0x7fe9ec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9ec0a3da0 0x7fe9ec0a6260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc037040 tx=0x7fe9fc040000 comp rx=0 tx=0).stop 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 msgr2=0x7fea041a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 0x7fea041a0700 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400b4a0 tx=0x7fe9f400b960 comp rx=0 tx=0).stop 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 shutdown_connections 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe9ec0a3da0 0x7fe9ec0a6260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea0419a8b0 0x7fea0419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea041094d0 0x7fea041a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 --2- 172.21.15.79:0/2602901518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea04104400 0x7fea041a0700 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 >> 172.21.15.79:0/2602901518 conn(0x7fea040ffce0 msgr2=0x7fea041010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 shutdown_connections 2024-04-22T06:47:16.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.724+0000 7fea0a715640 1 -- 172.21.15.79:0/2602901518 wait complete. 2024-04-22T06:47:16.738 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:47:16.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:16 smithi079 ceph-mon[27946]: pgmap v92: 97 pgs: 97 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 551 MiB/s wr, 2.52k op/s 2024-04-22T06:47:16.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:16 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3340298432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:16.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 -- 172.21.15.79:0/2147066115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c102b40 msgr2=0x7f675c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 --2- 172.21.15.79:0/2147066115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c102b40 0x7f675c106b80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6758009900 tx=0x7f675802bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:16.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 -- 172.21.15.79:0/2147066115 shutdown_connections 2024-04-22T06:47:16.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 --2- 172.21.15.79:0/2147066115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675c107260 0x7f675c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 --2- 172.21.15.79:0/2147066115 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c102b40 0x7f675c106b80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 --2- 172.21.15.79:0/2147066115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c102190 0x7f675c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 -- 172.21.15.79:0/2147066115 >> 172.21.15.79:0/2147066115 conn(0x7f675c0793a0 msgr2=0x7f675c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 -- 172.21.15.79:0/2147066115 shutdown_connections 2024-04-22T06:47:16.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.858+0000 7f676458f640 1 -- 172.21.15.79:0/2147066115 wait complete. 2024-04-22T06:47:16.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 Processor -- start 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 -- start start 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675c102190 0x7f675c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c107260 0x7f675c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 0x7f675c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f675c112180 con 0x7f675c1985a0 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f675c112000 con 0x7f675c102190 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f676458f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f675c112300 con 0x7f675c107260 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 0x7f675c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 0x7f675c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33868/0 (socket says 172.21.15.79:33868) 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 -- 172.21.15.79:0/2335179353 learned_addr learned my addr 172.21.15.79:0/2335179353 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 -- 172.21.15.79:0/2335179353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c107260 msgr2=0x7f675c19e930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:16.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c107260 0x7f675c19e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 -- 172.21.15.79:0/2335179353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675c102190 msgr2=0x7f675c19e3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675c102190 0x7f675c19e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.859+0000 7f6762b2c640 1 -- 172.21.15.79:0/2335179353 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c1990c0 con 0x7f675c1985a0 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.860+0000 7f6762b2c640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 0x7f675c198a50 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6754009fd0 tx=0x7f675400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.860+0000 7f67537fe640 1 -- 172.21.15.79:0/2335179353 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754017070 con 0x7f675c1985a0 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.860+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c1993b0 con 0x7f675c1985a0 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.860+0000 7f67537fe640 1 -- 172.21.15.79:0/2335179353 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6754003e20 con 0x7f675c1985a0 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.860+0000 7f67537fe640 1 -- 172.21.15.79:0/2335179353 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754010460 con 0x7f675c1985a0 2024-04-22T06:47:16.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.860+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c1a55b0 con 0x7f675c1985a0 2024-04-22T06:47:16.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.863+0000 7f67537fe640 1 -- 172.21.15.79:0/2335179353 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6754003970 con 0x7f675c1985a0 2024-04-22T06:47:16.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.863+0000 7f67537fe640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f67480a3d20 0x7f67480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676232b640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f67480a3d20 0x7f67480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f67537fe640 1 -- 172.21.15.79:0/2335179353 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f67540d0710 con 0x7f675c1985a0 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676717c740 1 Processor -- start 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676232b640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f67480a3d20 0x7f67480a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f674c00b9b0 tx=0x7f674c00b370 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676717c740 1 -- start start 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676717c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676717c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc62430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676717c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc64e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.864+0000 7f676717c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5613dbbbe1f0 con 0x5613dbc594d0 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676717c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5613dbb5b4b0 con 0x5613dbc62a00 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676717c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613dbbba1b0 con 0x5613dbc59e80 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33882/0 (socket says 172.21.15.79:33882) 2024-04-22T06:47:16.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 -- 172.21.15.79:0/2416633639 learned_addr learned my addr 172.21.15.79:0/2416633639 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f6761b2a640 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc62430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 -- 172.21.15.79:0/2416633639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 msgr2=0x5613dbc62430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc62430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 -- 172.21.15.79:0/2416633639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 msgr2=0x5613dbc64e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc64e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 -- 172.21.15.79:0/2416633639 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613dbb4d1b0 con 0x5613dbc594d0 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f676232b640 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc598b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f674c00dd70 tx=0x7f674c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2779b8bab2f1342 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f6750bfe640 1 -- 172.21.15.79:0/2416633639 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c00de60 con 0x5613dbc594d0 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f6750bfe640 1 -- 172.21.15.79:0/2416633639 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c016030 con 0x5613dbc594d0 2024-04-22T06:47:16.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.865+0000 7f6750bfe640 1 -- 172.21.15.79:0/2416633639 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c01d440 con 0x5613dbc594d0 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 -- 172.21.15.79:0/2416633639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 msgr2=0x5613dbc598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc598b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f674c00dd70 tx=0x7f674c010000 comp rx=0 tx=0).stop 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 -- 172.21.15.79:0/2416633639 shutdown_connections 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc64e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc62430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 --2- 172.21.15.79:0/2416633639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc598b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 -- 172.21.15.79:0/2416633639 >> 172.21.15.79:0/2416633639 conn(0x5613dbc53ba0 msgr2=0x5613dbc53f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 -- 172.21.15.79:0/2416633639 shutdown_connections 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.866+0000 7f676717c740 1 -- 172.21.15.79:0/2416633639 wait complete. 2024-04-22T06:47:16.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 -- start start 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 Processor -- start 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc62300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5613dbc65910 con 0x5613dbc594d0 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5613dbb77500 con 0x5613dbc62a00 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613dbc65a90 con 0x5613dbc59e80 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676717c740 10 client.0 _ll_register_callbacks cb 0x5613dbb45b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc62300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc62300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33886/0 (socket says 172.21.15.79:33886) 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f676232b640 1 -- 172.21.15.79:0/4059428470 learned_addr learned my addr 172.21.15.79:0/4059428470 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.867+0000 7f6762b2c640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676232b640 1 -- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 msgr2=0x5613dbc5f690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676232b640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5f690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676232b640 1 -- 172.21.15.79:0/4059428470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 msgr2=0x5613dbc5fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676232b640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676232b640 1 -- 172.21.15.79:0/4059428470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613db8d03f0 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f6762b2c640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676232b640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc62300 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f674c003c10 tx=0x7f674c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae74dff63c70a13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f6745ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f6745ffb640 1 -- 172.21.15.79:0/4059428470 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c02c070 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674c016030 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.868+0000 7f6745ffb640 1 -- 172.21.15.79:0/4059428470 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f674c003d70 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f6745ffb640 1 -- 172.21.15.79:0/4059428470 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f674c038050 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 10 client.16491 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5613db99a7d0 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f6745ffb640 1 -- 172.21.15.79:0/4059428470 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c016030 con 0x5613dbc594d0 2024-04-22T06:47:16.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f6745ffb640 1 -- 172.21.15.79:0/4059428470 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f674c0284d0 con 0x5613dbc594d0 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 10 client.16491 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 10 client.16491 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 20 client.16491 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbc72e00 0x5613dbc751f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 4 client.16491 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.869+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613dbb5d930 con 0x5613dbc72e00 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.870+0000 7f6761b2a640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbc72e00 0x5613dbc751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.873+0000 7f6745ffb640 10 client.16491 ms_handle_connect on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:16.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.873+0000 7f6761b2a640 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbc72e00 0x5613dbc751f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c83088f7b4219be0 server_cookie=dcf37f2f787b855c in_seq=0 out_seq=0 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f6745ffb640 1 -- 172.21.15.79:0/4059428470 <== mds.2 v2:172.21.15.193:6838/1588799940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613dbb5d930 con 0x5613dbc72e00 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f6745ffb640 10 client.16491 handle_command_reply: tid=0 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 client.16491 shutdown 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 msgr2=0x5613dbc62300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc62300 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f674c003c10 tx=0x7f674c00da90 comp rx=0 tx=0).stop 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 shutdown_connections 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbc72e00 0x5613dbc751f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 --2- 172.21.15.79:0/4059428470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc62300 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 >> 172.21.15.79:0/4059428470 conn(0x5613dbc53ba0 msgr2=0x5613dbc558f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f6745ffb640 0 client.16491 ms_handle_reset on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:16.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 shutdown_connections 2024-04-22T06:47:16.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.875+0000 7f676717c740 1 -- 172.21.15.79:0/4059428470 wait complete. 2024-04-22T06:47:16.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.876+0000 7f676717c740 20 client.16491 trim_cache size 0 max 16384 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 Processor -- start 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 -- start start 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc78130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc55700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b0780 con 0x5613dbc594d0 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1ae050 con 0x5613dbc62a00 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.907+0000 7f676717c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1abea0 con 0x5613dbc59e80 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc78130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc78130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33890/0 (socket says 172.21.15.79:33890) 2024-04-22T06:47:16.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 -- 172.21.15.79:0/4154901776 learned_addr learned my addr 172.21.15.79:0/4154901776 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f6761b2a640 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 -- 172.21.15.79:0/4154901776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 msgr2=0x5613dbc5e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 -- 172.21.15.79:0/4154901776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 msgr2=0x5613dbc55700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc55700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 -- 172.21.15.79:0/4154901776 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613dbc58760 con 0x5613dbc594d0 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f676232b640 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc78130 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f674c00d6b0 tx=0x7f674c00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0a1df2684383da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f6747fff640 1 -- 172.21.15.79:0/4154901776 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c02c070 con 0x5613dbc594d0 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f6747fff640 1 -- 172.21.15.79:0/4154901776 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c016070 con 0x5613dbc594d0 2024-04-22T06:47:16.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.908+0000 7f6747fff640 1 -- 172.21.15.79:0/4154901776 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c003a70 con 0x5613dbc594d0 2024-04-22T06:47:16.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 -- 172.21.15.79:0/4154901776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 msgr2=0x5613dbc78130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc78130 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f674c00d6b0 tx=0x7f674c00da20 comp rx=0 tx=0).stop 2024-04-22T06:47:16.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 -- 172.21.15.79:0/4154901776 shutdown_connections 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc55700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 --2- 172.21.15.79:0/4154901776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc78130 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 -- 172.21.15.79:0/4154901776 >> 172.21.15.79:0/4154901776 conn(0x5613dbc53ba0 msgr2=0x5613dbc54ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 -- 172.21.15.79:0/4154901776 shutdown_connections 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.909+0000 7f676717c740 1 -- 172.21.15.79:0/4154901776 wait complete. 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 -- start start 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 Processor -- start 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc5b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1ac510 con 0x5613dbc594d0 2024-04-22T06:47:16.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1ad980 con 0x5613dbc62a00 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1acbe0 con 0x5613dbc59e80 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676717c740 10 client.0 _ll_register_callbacks cb 0x7f674c005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc5b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.910+0000 7f676232b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc5b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33902/0 (socket says 172.21.15.79:33902) 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 -- 172.21.15.79:0/2406844886 learned_addr learned my addr 172.21.15.79:0/2406844886 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 -- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 msgr2=0x5613dbc5be40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5be40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 -- 172.21.15.79:0/2406844886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 msgr2=0x5613dbc5c380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 -- 172.21.15.79:0/2406844886 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613dbcdb0d0 con 0x5613dbc594d0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676232b640 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc5b900 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f674c028990 tx=0x7f674c028670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cceb7433b8895412 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f6745ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f6745ffb640 1 -- 172.21.15.79:0/2406844886 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c02c070 con 0x5613dbc594d0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f6745ffb640 1 -- 172.21.15.79:0/2406844886 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f674c028c70 con 0x5613dbc594d0 2024-04-22T06:47:16.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674c016070 con 0x5613dbc594d0 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f6745ffb640 1 -- 172.21.15.79:0/2406844886 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f674c005040 con 0x5613dbc594d0 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676717c740 10 client.16497 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.911+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5613dbcdc300 con 0x5613dbc594d0 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f6745ffb640 1 -- 172.21.15.79:0/2406844886 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c016070 con 0x5613dbc594d0 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f6745ffb640 1 -- 172.21.15.79:0/2406844886 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f674c004b80 con 0x5613dbc594d0 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f676717c740 10 client.16497 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f676717c740 10 client.16497 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f676717c740 20 client.16497 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f676717c740 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbcdca40 0x5613dbcdee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f676717c740 4 client.16497 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:47:16.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.912+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613dbbcbc10 con 0x5613dbcdca40 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.914+0000 7f6761b2a640 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbcdca40 0x5613dbcdee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.916+0000 7f6745ffb640 10 client.16497 ms_handle_connect on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.916+0000 7f6761b2a640 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbcdca40 0x5613dbcdee60 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a5260aa0348b4db server_cookie=e959475fa3584199 in_seq=0 out_seq=0 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f6745ffb640 1 -- 172.21.15.79:0/2406844886 <== mds.2 v2:172.21.15.193:6838/1588799940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613dbbcbc10 con 0x5613dbcdca40 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f6745ffb640 10 client.16497 handle_command_reply: tid=0 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 client.16497 shutdown 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 msgr2=0x5613dbc5b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc5b900 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f674c028990 tx=0x7f674c028670 comp rx=0 tx=0).stop 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 shutdown_connections 2024-04-22T06:47:16.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x5613dbcdca40 0x5613dbcdee60 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613dbc62a00 0x5613dbc5c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613dbc59e80 0x5613dbc5be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f6745ffb640 0 client.16497 ms_handle_reset on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 --2- 172.21.15.79:0/2406844886 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613dbc594d0 0x5613dbc5b900 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 >> 172.21.15.79:0/2406844886 conn(0x5613dbc53ba0 msgr2=0x5613dbc55880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 shutdown_connections 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 1 -- 172.21.15.79:0/2406844886 wait complete. 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.917+0000 7f676717c740 20 client.16497 trim_cache size 0 max 16384 2024-04-22T06:47:16.917 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:47:16.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f67480a3d20 msgr2=0x7f67480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f67480a3d20 0x7f67480a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f674c00b9b0 tx=0x7f674c00b370 comp rx=0 tx=0).stop 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 msgr2=0x7f675c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 0x7f675c198a50 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6754009fd0 tx=0x7f675400b260 comp rx=0 tx=0).stop 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 shutdown_connections 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f675c1985a0 0x7f675c198a50 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c107260 0x7f675c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f67480a3d20 0x7f67480a61e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 --2- 172.21.15.79:0/2335179353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f675c102190 0x7f675c19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 >> 172.21.15.79:0/2335179353 conn(0x7f675c0793a0 msgr2=0x7f675c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 shutdown_connections 2024-04-22T06:47:16.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:16.918+0000 7f676458f640 1 -- 172.21.15.79:0/2335179353 wait complete. 2024-04-22T06:47:16.931 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:47:16.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:16 smithi193 ceph-mon[31815]: pgmap v92: 97 pgs: 97 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 551 MiB/s wr, 2.52k op/s 2024-04-22T06:47:16.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:16 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3340298432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:16.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:16 smithi190 ceph-mon[31421]: pgmap v92: 97 pgs: 97 active+clean; 12 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 551 MiB/s wr, 2.52k op/s 2024-04-22T06:47:16.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:16 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3340298432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:17.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.202+0000 7fc50543f640 1 -- 172.21.15.79:0/3149541712 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500109a80 msgr2=0x7fc500114b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.202+0000 7fc50543f640 1 --2- 172.21.15.79:0/3149541712 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500109a80 0x7fc500114b60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009900 tx=0x7fc4f402bc80 comp rx=0 tx=0).stop 2024-04-22T06:47:17.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.203+0000 7fc50543f640 1 -- 172.21.15.79:0/3149541712 shutdown_connections 2024-04-22T06:47:17.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.203+0000 7fc50543f640 1 --2- 172.21.15.79:0/3149541712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5001150a0 0x7fc500117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.203+0000 7fc50543f640 1 --2- 172.21.15.79:0/3149541712 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500109a80 0x7fc500114b60 secure :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009900 tx=0x7fc4f402bc80 comp rx=0 tx=0).stop 2024-04-22T06:47:17.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.203+0000 7fc50543f640 1 --2- 172.21.15.79:0/3149541712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc500109160 0x7fc500109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.203+0000 7fc50543f640 1 -- 172.21.15.79:0/3149541712 >> 172.21.15.79:0/3149541712 conn(0x7fc50006d060 msgr2=0x7fc50006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:17.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.209+0000 7fc50543f640 1 -- 172.21.15.79:0/3149541712 shutdown_connections 2024-04-22T06:47:17.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.209+0000 7fc50543f640 1 -- 172.21.15.79:0/3149541712 wait complete. 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.210+0000 7fc50543f640 1 Processor -- start 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.210+0000 7fc50543f640 1 -- start start 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc50543f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc500109160 0x7fc5001a7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc50543f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5001150a0 0x7fc500114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc50543f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500114540 0x7fc5001149f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc50543f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc50011aa70 con 0x7fc500114540 2024-04-22T06:47:17.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc50543f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc50011a8f0 con 0x7fc5001150a0 2024-04-22T06:47:17.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc50543f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc50011abf0 con 0x7fc500109160 2024-04-22T06:47:17.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc4ff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500114540 0x7fc5001149f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc4ff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500114540 0x7fc5001149f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33922/0 (socket says 172.21.15.79:33922) 2024-04-22T06:47:17.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.211+0000 7fc4ff7fe640 1 -- 172.21.15.79:0/1708625149 learned_addr learned my addr 172.21.15.79:0/1708625149 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:17.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.213+0000 7fc4feffd640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc500109160 0x7fc5001a7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.219+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5001150a0 0x7fc500114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.219+0000 7fc4feffd640 1 -- 172.21.15.79:0/1708625149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5001150a0 msgr2=0x7fc500114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.219+0000 7fc4feffd640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5001150a0 0x7fc500114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.219+0000 7fc4feffd640 1 -- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500114540 msgr2=0x7fc5001149f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.219+0000 7fc4feffd640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500114540 0x7fc5001149f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.219+0000 7fc4feffd640 1 -- 172.21.15.79:0/1708625149 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5001a39a0 con 0x7fc500109160 2024-04-22T06:47:17.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.221+0000 7fc4feffd640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc500109160 0x7fc5001a7e90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800cce0 tx=0x7fc4f80075f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:17.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.221+0000 7fc4e7fff640 1 -- 172.21.15.79:0/1708625149 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8004b20 con 0x7fc500109160 2024-04-22T06:47:17.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.221+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5001a3c90 con 0x7fc500109160 2024-04-22T06:47:17.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.222+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5001a4200 con 0x7fc500109160 2024-04-22T06:47:17.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.222+0000 7fc4e7fff640 1 -- 172.21.15.79:0/1708625149 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f8004cc0 con 0x7fc500109160 2024-04-22T06:47:17.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.222+0000 7fc4e7fff640 1 -- 172.21.15.79:0/1708625149 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800f6b0 con 0x7fc500109160 2024-04-22T06:47:17.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.226+0000 7fc50802b740 1 Processor -- start 2024-04-22T06:47:17.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.226+0000 7fc50802b740 1 -- start start 2024-04-22T06:47:17.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.228+0000 7fc50802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b094d0 0x559f11b098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.228+0000 7fc50802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b09e80 0x559f11b12430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.228+0000 7fc50802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b14e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.228+0000 7fc50802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x559f11a6e1f0 con 0x559f11b12a00 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.228+0000 7fc50802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559f11a0b4b0 con 0x559f11b09e80 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.228+0000 7fc50802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f11a6a1b0 con 0x559f11b094d0 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b14e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b14e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33926/0 (socket says 172.21.15.79:33926) 2024-04-22T06:47:17.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2458495249 learned_addr learned my addr 172.21.15.79:0/2458495249 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:17.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2458495249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b094d0 msgr2=0x559f11b098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:17.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b094d0 0x559f11b098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2458495249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b09e80 msgr2=0x559f11b12430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:17.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b09e80 0x559f11b12430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.229+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2458495249 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f119fd1b0 con 0x559f11b12a00 2024-04-22T06:47:17.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.230+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b14e50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4006010 tx=0x7fc4f4004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4f332e2c196adc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:17.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.231+0000 7fc4e7fff640 1 -- 172.21.15.79:0/1708625149 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4f8005020 con 0x7fc500109160 2024-04-22T06:47:17.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.231+0000 7fc4e57fa640 1 -- 172.21.15.79:0/2458495249 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4042020 con 0x559f11b12a00 2024-04-22T06:47:17.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.231+0000 7fc4e57fa640 1 -- 172.21.15.79:0/2458495249 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f402c620 con 0x559f11b12a00 2024-04-22T06:47:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.232+0000 7fc4e57fa640 1 -- 172.21.15.79:0/2458495249 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4003830 con 0x559f11b12a00 2024-04-22T06:47:17.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.232+0000 7fc4e7fff640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc4e80a3de0 0x7fc4e80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc4e7fff640 1 -- 172.21.15.79:0/1708625149 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc4f80d0840 con 0x7fc500109160 2024-04-22T06:47:17.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.232+0000 7fc50802b740 1 -- 172.21.15.79:0/2458495249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 msgr2=0x559f11b14e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc50802b740 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b14e50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4006010 tx=0x7fc4f4004800 comp rx=0 tx=0).stop 2024-04-22T06:47:17.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc4ff7fe640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc4e80a3de0 0x7fc4e80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc50802b740 1 -- 172.21.15.79:0/2458495249 shutdown_connections 2024-04-22T06:47:17.232 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc50802b740 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b09e80 0x559f11b12430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc50802b740 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b14e50 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc50802b740 1 --2- 172.21.15.79:0/2458495249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b094d0 0x559f11b098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.233+0000 7fc50802b740 1 -- 172.21.15.79:0/2458495249 >> 172.21.15.79:0/2458495249 conn(0x559f11b03ba0 msgr2=0x559f11b03f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:17.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.234+0000 7fc50802b740 1 -- 172.21.15.79:0/2458495249 shutdown_connections 2024-04-22T06:47:17.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.234+0000 7fc4ff7fe640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc4e80a3de0 0x7fc4e80a62a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc4f002cad0 tx=0x7fc4f002c2e0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:17.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.234+0000 7fc50802b740 1 -- 172.21.15.79:0/2458495249 wait complete. 2024-04-22T06:47:17.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 -- start start 2024-04-22T06:47:17.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 Processor -- start 2024-04-22T06:47:17.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b12200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 0x559f11b0fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x559f11b15960 con 0x559f11b09e80 2024-04-22T06:47:17.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x559f11a27500 con 0x559f11b094d0 2024-04-22T06:47:17.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f11b15ae0 con 0x559f11b12a00 2024-04-22T06:47:17.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.235+0000 7fc50802b740 10 client.0 _ll_register_callbacks cb 0x559f119f5b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:17.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33936/0 (socket says 172.21.15.79:33936) 2024-04-22T06:47:17.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/3009749038 learned_addr learned my addr 172.21.15.79:0/3009749038 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/3009749038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 msgr2=0x559f11b0fad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 0x559f11b0fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 msgr2=0x559f11b12200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b12200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.236+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/3009749038 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f117803f0 con 0x559f11b09e80 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.237+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0f590 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009850 tx=0x7fc4f402ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d4de28113872b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.237+0000 7fc4e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:17.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.238+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/3009749038 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4042030 con 0x559f11b09e80 2024-04-22T06:47:17.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.238+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4f402c620 con 0x559f11b09e80 2024-04-22T06:47:17.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.238+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/3009749038 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc4f40035f0 con 0x559f11b09e80 2024-04-22T06:47:17.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.238+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/3009749038 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc4f4055050 con 0x559f11b09e80 2024-04-22T06:47:17.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.238+0000 7fc50802b740 10 client.16509 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:17.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.238+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x559f1184a7d0 con 0x559f11b09e80 2024-04-22T06:47:17.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.239+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/3009749038 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f402c620 con 0x559f11b09e80 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/3009749038 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fc4f4003810 con 0x559f11b09e80 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc50802b740 10 client.16509 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc50802b740 10 client.16509 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc50802b740 20 client.16509 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc50802b740 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b22a20 0x559f11b24e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc50802b740 4 client.16509 mds_command: new command op to 25265 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:17.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.241+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 --> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f11a0d930 con 0x559f11b22a20 2024-04-22T06:47:17.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.243+0000 7fc4feffd640 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b22a20 0x559f11b24e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.248+0000 7fc4feffd640 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b22a20 0x559f11b24e10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=391d45c17d28ec51 server_cookie=69073c98101affea in_seq=0 out_seq=0 2024-04-22T06:47:17.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.248+0000 7fc4e2ff5640 10 client.16509 ms_handle_connect on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:17.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/3009749038 <== mds.1 v2:172.21.15.190:6836/1635214680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f11a0d930 con 0x559f11b22a20 2024-04-22T06:47:17.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc4e2ff5640 10 client.16509 handle_command_reply: tid=0 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 client.16509 shutdown 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 msgr2=0x559f11b0f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0f590 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009850 tx=0x7fc4f402ca00 comp rx=0 tx=0).stop 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 shutdown_connections 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b22a20 0x559f11b24e10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 0x559f11b0fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0f590 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 --2- 172.21.15.79:0/3009749038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b12200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.251+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 >> 172.21.15.79:0/3009749038 conn(0x559f11b03ba0 msgr2=0x559f11b058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:17.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.252+0000 7fc4e2ff5640 0 client.16509 ms_handle_reset on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:17.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.252+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 shutdown_connections 2024-04-22T06:47:17.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.252+0000 7fc50802b740 1 -- 172.21.15.79:0/3009749038 wait complete. 2024-04-22T06:47:17.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.252+0000 7fc50802b740 20 client.16509 trim_cache size 0 max 16384 2024-04-22T06:47:17.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.283+0000 7fc50802b740 1 Processor -- start 2024-04-22T06:47:17.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 -- start start 2024-04-22T06:47:17.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b0c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 0x559f11b1ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec011190 con 0x559f11b09e80 2024-04-22T06:47:17.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00ea60 con 0x559f11b094d0 2024-04-22T06:47:17.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc50802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00c960 con 0x559f11b12a00 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc4fe7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc4fe7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33944/0 (socket says 172.21.15.79:33944) 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.284+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2043376973 learned_addr learned my addr 172.21.15.79:0/2043376973 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2043376973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 msgr2=0x559f11b1ddf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 0x559f11b1ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2043376973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 msgr2=0x559f11b0c890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b0c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4fe7fc640 1 -- 172.21.15.79:0/2043376973 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f11b09c20 con 0x559f11b09e80 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4fe7fc640 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0cdd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4053040 tx=0x7fc4f4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3561de1f02f4cc04 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.285+0000 7fc4e4ff9640 1 -- 172.21.15.79:0/2043376973 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4042070 con 0x559f11b09e80 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.286+0000 7fc4e4ff9640 1 -- 172.21.15.79:0/2043376973 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f4005850 con 0x559f11b09e80 2024-04-22T06:47:17.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.286+0000 7fc4e4ff9640 1 -- 172.21.15.79:0/2043376973 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f403e6d0 con 0x559f11b09e80 2024-04-22T06:47:17.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 -- 172.21.15.79:0/2043376973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 msgr2=0x559f11b0cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0cdd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4053040 tx=0x7fc4f4000bc0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 -- 172.21.15.79:0/2043376973 shutdown_connections 2024-04-22T06:47:17.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b12a00 0x559f11b1ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b09e80 0x559f11b0cdd0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 --2- 172.21.15.79:0/2043376973 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b0c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 -- 172.21.15.79:0/2043376973 >> 172.21.15.79:0/2043376973 conn(0x559f11b03ba0 msgr2=0x559f11b0dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:17.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 -- 172.21.15.79:0/2043376973 shutdown_connections 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.287+0000 7fc50802b740 1 -- 172.21.15.79:0/2043376973 wait complete. 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.288+0000 7fc50802b740 1 -- start start 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.288+0000 7fc50802b740 1 Processor -- start 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b27f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b09e80 0x559f11b25360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b25aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00cf20 con 0x559f11b12a00 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00e390 con 0x559f11b094d0 2024-04-22T06:47:17.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec00d5f0 con 0x559f11b09e80 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc50802b740 10 client.0 _ll_register_callbacks cb 0x7fc4f4004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc4ff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b25aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc4ff7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b25aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33948/0 (socket says 172.21.15.79:33948) 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.289+0000 7fc4ff7fe640 1 -- 172.21.15.79:0/4206518034 learned_addr learned my addr 172.21.15.79:0/4206518034 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4ff7fe640 1 -- 172.21.15.79:0/4206518034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b09e80 msgr2=0x559f11b25360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4ff7fe640 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b09e80 0x559f11b25360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4ff7fe640 1 -- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 msgr2=0x559f11b27f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:17.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4ff7fe640 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b27f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4ff7fe640 1 -- 172.21.15.79:0/4206518034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f11b87c00 con 0x559f11b12a00 2024-04-22T06:47:17.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4ff7fe640 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b25aa0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc4f002c5d0 tx=0x7fc4f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9acd6837dbe07bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:17.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.290+0000 7fc4e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:17.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.291+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/4206518034 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0038070 con 0x559f11b12a00 2024-04-22T06:47:17.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.291+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/4206518034 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc4f003d960 con 0x559f11b12a00 2024-04-22T06:47:17.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.291+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4f4005850 con 0x559f11b12a00 2024-04-22T06:47:17.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.291+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/4206518034 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc4f003db80 con 0x559f11b12a00 2024-04-22T06:47:17.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.291+0000 7fc50802b740 10 client.16515 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:17.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.291+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x559f11b88dc0 con 0x559f11b12a00 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/4206518034 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f4005850 con 0x559f11b12a00 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/4206518034 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fc4f003e420 con 0x559f11b12a00 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc50802b740 10 client.16515 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc50802b740 10 client.16515 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc50802b740 20 client.16515 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc50802b740 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b89570 0x559f11b8b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc50802b740 4 client.16515 mds_command: new command op to 25265 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:47:17.291 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.292+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 --> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f11a7bc10 con 0x559f11b89570 2024-04-22T06:47:17.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.295+0000 7fc4feffd640 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b89570 0x559f11b8b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.299+0000 7fc4e2ff5640 10 client.16515 ms_handle_connect on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:17.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.299+0000 7fc4feffd640 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b89570 0x559f11b8b990 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cde0392440115412 server_cookie=b5a7d091e23fc7ba in_seq=0 out_seq=0 2024-04-22T06:47:17.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc4e2ff5640 1 -- 172.21.15.79:0/4206518034 <== mds.1 v2:172.21.15.190:6836/1635214680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f11a7bc10 con 0x559f11b89570 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc4e2ff5640 10 client.16515 handle_command_reply: tid=0 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 client.16515 shutdown 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 msgr2=0x559f11b25aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b25aa0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc4f002c5d0 tx=0x7fc4f0004730 comp rx=0 tx=0).stop 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 shutdown_connections 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x559f11b89570 0x559f11b8b990 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x559f11b12a00 0x559f11b25aa0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f11b09e80 0x559f11b25360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 --2- 172.21.15.79:0/4206518034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f11b094d0 0x559f11b27f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 >> 172.21.15.79:0/4206518034 conn(0x559f11b03ba0 msgr2=0x559f11b02370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:17.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.300+0000 7fc4e2ff5640 0 client.16515 ms_handle_reset on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:17.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.301+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 shutdown_connections 2024-04-22T06:47:17.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.301+0000 7fc50802b740 1 -- 172.21.15.79:0/4206518034 wait complete. 2024-04-22T06:47:17.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.301+0000 7fc50802b740 20 client.16515 trim_cache size 0 max 16384 2024-04-22T06:47:17.301 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.302+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc4e80a3de0 msgr2=0x7fc4e80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.302+0000 7fc50543f640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc4e80a3de0 0x7fc4e80a62a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc4f002cad0 tx=0x7fc4f002c2e0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.302+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc500109160 msgr2=0x7fc5001a7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.302+0000 7fc50543f640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc500109160 0x7fc5001a7e90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800cce0 tx=0x7fc4f80075f0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 shutdown_connections 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc4e80a3de0 0x7fc4e80a62a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5001150a0 0x7fc500114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc500114540 0x7fc5001149f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 --2- 172.21.15.79:0/1708625149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc500109160 0x7fc5001a7e90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:17.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 >> 172.21.15.79:0/1708625149 conn(0x7fc50006d060 msgr2=0x7fc500070e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:17.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 shutdown_connections 2024-04-22T06:47:17.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:17.303+0000 7fc50543f640 1 -- 172.21.15.79:0/1708625149 wait complete. 2024-04-22T06:47:18.338 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:47:18.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:47:18.338 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:47:18.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.694+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3558881536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4106d20 msgr2=0x7fe7a4107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.694+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3558881536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4106d20 0x7fe7a4107100 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009ca0 tx=0x7fe79c02c100 comp rx=0 tx=0).stop 2024-04-22T06:47:18.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.698+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3558881536 shutdown_connections 2024-04-22T06:47:18.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.698+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3558881536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a4100cf0 0x7fe7a4104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.698+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3558881536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7a4100350 0x7fe7a41007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.698+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3558881536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4106d20 0x7fe7a4107100 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.698+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3558881536 >> 172.21.15.79:0/3558881536 conn(0x7fe7a40772e0 msgr2=0x7fe7a40fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:18.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.700+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3558881536 shutdown_connections 2024-04-22T06:47:18.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.700+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3558881536 wait complete. 2024-04-22T06:47:18.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.709+0000 7fe7aa3cf640 1 Processor -- start 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 -- start start 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 0x7fe7a4115910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7a4100cf0 0x7fe7a410f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a4106d20 0x7fe7a410fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a4108340 con 0x7fe7a4100350 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a41081c0 con 0x7fe7a4106d20 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.710+0000 7fe7aa3cf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a41084c0 con 0x7fe7a4100cf0 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.712+0000 7fe7a93cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 0x7fe7a4115910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.712+0000 7fe7a93cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 0x7fe7a4115910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33980/0 (socket says 172.21.15.79:33980) 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.712+0000 7fe7a93cd640 1 -- 172.21.15.79:0/3427321371 learned_addr learned my addr 172.21.15.79:0/3427321371 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:18.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.712+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7a4100cf0 0x7fe7a410f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.714+0000 7fe7a9bce640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a4106d20 0x7fe7a410fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.714+0000 7fe7a93cd640 1 -- 172.21.15.79:0/3427321371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7a4100cf0 msgr2=0x7fe7a410f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.714+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7a4100cf0 0x7fe7a410f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.714+0000 7fe7a93cd640 1 -- 172.21.15.79:0/3427321371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a4106d20 msgr2=0x7fe7a410fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.714+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a4106d20 0x7fe7a410fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.714+0000 7fe7a93cd640 1 -- 172.21.15.79:0/3427321371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a4110500 con 0x7fe7a4100350 2024-04-22T06:47:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.717+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 0x7fe7a4115910 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe79c02c690 tx=0x7fe79c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.717+0000 7fe79a7fc640 1 -- 172.21.15.79:0/3427321371 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c039040 con 0x7fe7a4100350 2024-04-22T06:47:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.718+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a4110790 con 0x7fe7a4100350 2024-04-22T06:47:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.718+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a406bc60 con 0x7fe7a4100350 2024-04-22T06:47:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.720+0000 7fe79a7fc640 1 -- 172.21.15.79:0/3427321371 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe79c044070 con 0x7fe7a4100350 2024-04-22T06:47:18.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.720+0000 7fe79a7fc640 1 -- 172.21.15.79:0/3427321371 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c0042a0 con 0x7fe7a4100350 2024-04-22T06:47:18.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.723+0000 7fe7ae223740 1 Processor -- start 2024-04-22T06:47:18.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.726+0000 7fe79a7fc640 1 -- 172.21.15.79:0/3427321371 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe79c004440 con 0x7fe7a4100350 2024-04-22T06:47:18.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.727+0000 7fe79a7fc640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe7900a3f20 0x7fe7900a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.727+0000 7fe79a7fc640 1 -- 172.21.15.79:0/3427321371 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe79c0f2490 con 0x7fe7a4100350 2024-04-22T06:47:18.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.728+0000 7fe7ae223740 1 -- start start 2024-04-22T06:47:18.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 0x5601c0cb0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 0x5601c0cb99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 0x5601c0cbc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7ae223740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5601c0c14f00 con 0x5601c0cb1430 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7ae223740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5601c0bb2590 con 0x5601c0cb0a80 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7ae223740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5601c0c10ec0 con 0x5601c0cb9fb0 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.729+0000 7fe7a9bce640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe7900a3f20 0x7fe7900a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.730+0000 7fe7a8bcc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 0x5601c0cb99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.730+0000 7fe7a8bcc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 0x5601c0cb99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33988/0 (socket says 172.21.15.79:33988) 2024-04-22T06:47:18.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.730+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/3402102081 learned_addr learned my addr 172.21.15.79:0/3402102081 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:18.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a9bce640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe7900a3f20 0x7fe7900a63e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0009b30 tx=0x7fe7a0005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:18.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/3402102081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 msgr2=0x5601c0cbc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:18.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 0x5601c0cbc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/3402102081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 msgr2=0x5601c0cb0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:18.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 0x5601c0cb0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/3402102081 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601c09ef7e0 con 0x5601c0cb1430 2024-04-22T06:47:18.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.731+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 0x5601c0cb99e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe794009c60 tx=0x7fe794011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1355042018a4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:18.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.734+0000 7fe78f7fe640 1 -- 172.21.15.79:0/3402102081 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794004230 con 0x5601c0cb1430 2024-04-22T06:47:18.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.734+0000 7fe78f7fe640 1 -- 172.21.15.79:0/3402102081 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7940043d0 con 0x5601c0cb1430 2024-04-22T06:47:18.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.734+0000 7fe78f7fe640 1 -- 172.21.15.79:0/3402102081 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794004cf0 con 0x5601c0cb1430 2024-04-22T06:47:18.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.735+0000 7fe7ae223740 1 -- 172.21.15.79:0/3402102081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 msgr2=0x5601c0cb99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.735+0000 7fe7ae223740 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 0x5601c0cb99e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe794009c60 tx=0x7fe794011810 comp rx=0 tx=0).stop 2024-04-22T06:47:18.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 -- 172.21.15.79:0/3402102081 shutdown_connections 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 0x5601c0cbc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb1430 0x5601c0cb99e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 --2- 172.21.15.79:0/3402102081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 0x5601c0cb0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 -- 172.21.15.79:0/3402102081 >> 172.21.15.79:0/3402102081 conn(0x5601c0cab150 msgr2=0x5601c0cab530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 -- 172.21.15.79:0/3402102081 shutdown_connections 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 -- 172.21.15.79:0/3402102081 wait complete. 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 -- start start 2024-04-22T06:47:18.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.736+0000 7fe7ae223740 1 Processor -- start 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 0x5601c0cb7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 0x5601c0cb75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cc6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5601c0cbcf20 con 0x5601c0cb7b00 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5601c0bce580 con 0x5601c0cb9fb0 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5601c0cbd0a0 con 0x5601c0cb0a80 2024-04-22T06:47:18.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.737+0000 7fe7ae223740 10 client.0 _ll_register_callbacks cb 0x5601c0bd4650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:18.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.738+0000 7fe7a9bce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cc6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.738+0000 7fe7a9bce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cc6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33994/0 (socket says 172.21.15.79:33994) 2024-04-22T06:47:18.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.738+0000 7fe7a9bce640 1 -- 172.21.15.79:0/1595020910 learned_addr learned my addr 172.21.15.79:0/1595020910 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:18.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe7a9bce640 1 -- 172.21.15.79:0/1595020910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 msgr2=0x5601c0cb7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:18.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe7a9bce640 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 0x5601c0cb7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe7a9bce640 1 -- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 msgr2=0x5601c0cb75c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:18.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe7a9bce640 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 0x5601c0cb75c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe7a9bce640 1 -- 172.21.15.79:0/1595020910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601c0a4ff70 con 0x5601c0cb7b00 2024-04-22T06:47:18.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe7a9bce640 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cc6420 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0012820 tx=0x7fe7a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a117444578abbd5f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:18.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.739+0000 7fe78cff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:18.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.740+0000 7fe78cff9640 1 -- 172.21.15.79:0/1595020910 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000fad0 con 0x5601c0cb7b00 2024-04-22T06:47:18.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.740+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7940043d0 con 0x5601c0cb7b00 2024-04-22T06:47:18.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.740+0000 7fe78cff9640 1 -- 172.21.15.79:0/1595020910 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe7a000fc70 con 0x5601c0cb7b00 2024-04-22T06:47:18.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.740+0000 7fe78cff9640 1 -- 172.21.15.79:0/1595020910 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe7a002e480 con 0x5601c0cb7b00 2024-04-22T06:47:18.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.740+0000 7fe7ae223740 10 client.16527 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.740+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5601c08e7170 con 0x5601c0cb7b00 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe78cff9640 1 -- 172.21.15.79:0/1595020910 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7940043d0 con 0x5601c0cb7b00 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe78cff9640 1 -- 172.21.15.79:0/1595020910 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fe7a001f5e0 con 0x5601c0cb7b00 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe7ae223740 10 client.16527 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe7ae223740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe7ae223740 20 client.16527 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe7ae223740 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0ccafe0 0x5601c0ccd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe7ae223740 4 client.16527 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:18.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.741+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601c0c22920 con 0x5601c0ccafe0 2024-04-22T06:47:18.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:18 smithi079 ceph-mon[27946]: pgmap v93: 97 pgs: 97 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 850 MiB/s wr, 5.02k op/s 2024-04-22T06:47:18.742 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.744+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0ccafe0 0x5601c0ccd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.749+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0ccafe0 0x5601c0ccd4a0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd1e498fcaa34b1 server_cookie=3e41a01893e5d563 in_seq=0 out_seq=0 2024-04-22T06:47:18.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.749+0000 7fe78cff9640 10 client.16527 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:18.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe78cff9640 1 -- 172.21.15.79:0/1595020910 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601c0c22920 con 0x5601c0ccafe0 2024-04-22T06:47:18.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe78cff9640 10 client.16527 handle_command_reply: tid=0 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 client.16527 shutdown 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 msgr2=0x5601c0cc6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cc6420 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0012820 tx=0x7fe7a00046c0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 shutdown_connections 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0ccafe0 0x5601c0ccd4a0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 0x5601c0cb75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cc6420 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe78cff9640 0 client.16527 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 --2- 172.21.15.79:0/1595020910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 0x5601c0cb7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 >> 172.21.15.79:0/1595020910 conn(0x5601c0cab150 msgr2=0x5601c0cacea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 shutdown_connections 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 1 -- 172.21.15.79:0/1595020910 wait complete. 2024-04-22T06:47:18.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.754+0000 7fe7ae223740 20 client.16527 trim_cache size 0 max 16384 2024-04-22T06:47:18.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 Processor -- start 2024-04-22T06:47:18.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 -- start start 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 0x5601c0ca9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cd2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 0x5601c0cd32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe7880112d0 con 0x5601c0cb7b00 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe78800eba0 con 0x5601c0cb9fb0 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.777+0000 7fe7ae223740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe78800c9f0 con 0x5601c0cb0a80 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cd2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cd2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34010/0 (socket says 172.21.15.79:34010) 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/112338907 learned_addr learned my addr 172.21.15.79:0/112338907 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/112338907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 msgr2=0x5601c0ca9fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 0x5601c0ca9fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/112338907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 msgr2=0x5601c0cd32f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 0x5601c0cd32f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/112338907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601c0cb6050 con 0x5601c0cb7b00 2024-04-22T06:47:18.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cd2db0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe794009b80 tx=0x7fe794009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f749627c0be51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:18.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe78effd640 1 -- 172.21.15.79:0/112338907 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79401e040 con 0x5601c0cb7b00 2024-04-22T06:47:18.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe78effd640 1 -- 172.21.15.79:0/112338907 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7940035f0 con 0x5601c0cb7b00 2024-04-22T06:47:18.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.778+0000 7fe78effd640 1 -- 172.21.15.79:0/112338907 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794022690 con 0x5601c0cb7b00 2024-04-22T06:47:18.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 -- 172.21.15.79:0/112338907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 msgr2=0x5601c0cd2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cd2db0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe794009b80 tx=0x7fe794009f30 comp rx=0 tx=0).stop 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 -- 172.21.15.79:0/112338907 shutdown_connections 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb9fb0 0x5601c0cd32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cd2db0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 --2- 172.21.15.79:0/112338907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb0a80 0x5601c0ca9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 -- 172.21.15.79:0/112338907 >> 172.21.15.79:0/112338907 conn(0x5601c0cab150 msgr2=0x5601c0ca98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 -- 172.21.15.79:0/112338907 shutdown_connections 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.779+0000 7fe7ae223740 1 -- 172.21.15.79:0/112338907 wait complete. 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 -- start start 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 Processor -- start 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 0x5601c0cc25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cbf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 0x5601c0cbf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe78800d060 con 0x5601c0cb7b00 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe78800e4d0 con 0x5601c0cb0a80 2024-04-22T06:47:18.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe78800d730 con 0x5601c0cb9fb0 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7ae223740 10 client.0 _ll_register_callbacks cb 0x7fe7a0012520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cbf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cbf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34020/0 (socket says 172.21.15.79:34020) 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/2377576270 learned_addr learned my addr 172.21.15.79:0/2377576270 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/2377576270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 msgr2=0x5601c0cbf890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 0x5601c0cbf890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 msgr2=0x5601c0cc25e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 0x5601c0cc25e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.780+0000 7fe7a8bcc640 1 -- 172.21.15.79:0/2377576270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601c0cbfed0 con 0x5601c0cb7b00 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe7a8bcc640 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cbf350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe794005e00 tx=0x7fe794005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac558fc99953b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe78cff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe78cff9640 1 -- 172.21.15.79:0/2377576270 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7940039b0 con 0x5601c0cb7b00 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe78cff9640 1 -- 172.21.15.79:0/2377576270 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe794003b50 con 0x5601c0cb7b00 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7940035f0 con 0x5601c0cb7b00 2024-04-22T06:47:18.782 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe78cff9640 1 -- 172.21.15.79:0/2377576270 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe794003d50 con 0x5601c0cb7b00 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe7ae223740 10 client.16533 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5601c0d324c0 con 0x5601c0cb7b00 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.781+0000 7fe78cff9640 1 -- 172.21.15.79:0/2377576270 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7940035f0 con 0x5601c0cb7b00 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe78cff9640 1 -- 172.21.15.79:0/2377576270 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fe794021070 con 0x5601c0cb7b00 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe7ae223740 10 client.16533 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe7ae223740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe7ae223740 20 client.16533 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe7ae223740 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0d32a10 0x5601c0d34e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe7ae223740 4 client.16533 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:47:18.783 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.782+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5601c0c1c790 con 0x5601c0d32a10 2024-04-22T06:47:18.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.783+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0d32a10 0x5601c0d34e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:18.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.785+0000 7fe7a93cd640 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0d32a10 0x5601c0d34e30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500b9a3e1349a8a0 server_cookie=ed9d49742c115d66 in_seq=0 out_seq=0 2024-04-22T06:47:18.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.785+0000 7fe78cff9640 10 client.16533 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:18.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.938+0000 7fe78cff9640 1 -- 172.21.15.79:0/2377576270 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5601c0c1c790 con 0x5601c0d32a10 2024-04-22T06:47:18.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.938+0000 7fe78cff9640 10 client.16533 handle_command_reply: tid=0 2024-04-22T06:47:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:18 smithi193 ceph-mon[31815]: pgmap v93: 97 pgs: 97 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 850 MiB/s wr, 5.02k op/s 2024-04-22T06:47:18.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.945+0000 7fe7ae223740 1 client.16533 shutdown 2024-04-22T06:47:18.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.950+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 msgr2=0x5601c0cbf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.950+0000 7fe7ae223740 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cbf350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe794005e00 tx=0x7fe794005590 comp rx=0 tx=0).stop 2024-04-22T06:47:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:18 smithi190 ceph-mon[31421]: pgmap v93: 97 pgs: 97 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 850 MiB/s wr, 5.02k op/s 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 shutdown_connections 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe7ae223740 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5601c0d32a10 0x5601c0d34e30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe7ae223740 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601c0cb9fb0 0x5601c0cbf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe7ae223740 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5601c0cb7b00 0x5601c0cbf350 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe7ae223740 1 --2- 172.21.15.79:0/2377576270 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5601c0cb0a80 0x5601c0cc25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 >> 172.21.15.79:0/2377576270 conn(0x5601c0cab150 msgr2=0x5601c0caa140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.958+0000 7fe78cff9640 0 client.16533 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:18.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.959+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 shutdown_connections 2024-04-22T06:47:18.960 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.961+0000 7fe7ae223740 1 -- 172.21.15.79:0/2377576270 wait complete. 2024-04-22T06:47:18.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.962+0000 7fe7ae223740 20 client.16533 trim_cache size 0 max 16384 2024-04-22T06:47:18.961 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:18.961 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:47:18.961 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "a133a457-5e1c-46a9-ab62-e61000b8f3e3", 2024-04-22T06:47:18.961 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:47:18.961 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:18.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.972+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe7900a3f20 msgr2=0x7fe7900a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.972+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe7900a3f20 0x7fe7900a63e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0009b30 tx=0x7fe7a0005f90 comp rx=0 tx=0).stop 2024-04-22T06:47:18.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.972+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 msgr2=0x7fe7a4115910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:18.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.972+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 0x7fe7a4115910 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe79c02c690 tx=0x7fe79c035ce0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.976+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 shutdown_connections 2024-04-22T06:47:18.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.976+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe7900a3f20 0x7fe7900a63e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.976+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7a4106d20 0x7fe7a410fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.976+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7a4100cf0 0x7fe7a410f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.976+0000 7fe7aa3cf640 1 --2- 172.21.15.79:0/3427321371 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe7a4100350 0x7fe7a4115910 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:18.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.976+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 >> 172.21.15.79:0/3427321371 conn(0x7fe7a40772e0 msgr2=0x7fe7a41024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:18.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.983+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 shutdown_connections 2024-04-22T06:47:18.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:18.985+0000 7fe7aa3cf640 1 -- 172.21.15.79:0/3427321371 wait complete. 2024-04-22T06:47:19.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:47:19.037 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 -- 172.21.15.79:0/2377370552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf841010a0 msgr2=0x7fdf84101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 --2- 172.21.15.79:0/2377370552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf841010a0 0x7fdf84101500 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009900 tx=0x7fdf6c02bc50 comp rx=0 tx=0).stop 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 -- 172.21.15.79:0/2377370552 shutdown_connections 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 --2- 172.21.15.79:0/2377370552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf84101a40 0x7fdf8410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 --2- 172.21.15.79:0/2377370552 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf841010a0 0x7fdf84101500 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 --2- 172.21.15.79:0/2377370552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf84106ed0 0x7fdf841072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 -- 172.21.15.79:0/2377370552 >> 172.21.15.79:0/2377370552 conn(0x7fdf84077c40 msgr2=0x7fdf84078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 -- 172.21.15.79:0/2377370552 shutdown_connections 2024-04-22T06:47:19.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.238+0000 7fdf89499640 1 -- 172.21.15.79:0/2377370552 wait complete. 2024-04-22T06:47:19.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 Processor -- start 2024-04-22T06:47:19.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 -- start start 2024-04-22T06:47:19.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 0x7fdf8419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf84101a40 0x7fdf8419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf84106ed0 0x7fdf84198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84111170 con 0x7fdf84106ed0 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84110ff0 con 0x7fdf84101a40 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.239+0000 7fdf89499640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841112f0 con 0x7fdf841010a0 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 0x7fdf8419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf84106ed0 0x7fdf84198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 0x7fdf8419e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:35888/0 (socket says 172.21.15.79:35888) 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 -- 172.21.15.79:0/1404729259 learned_addr learned my addr 172.21.15.79:0/1404729259 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 -- 172.21.15.79:0/1404729259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf84101a40 msgr2=0x7fdf8419eae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf84101a40 0x7fdf8419eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 -- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf84106ed0 msgr2=0x7fdf84198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf84106ed0 0x7fdf84198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 -- 172.21.15.79:0/1404729259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf84198f20 con 0x7fdf841010a0 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf837fe640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf84106ed0 0x7fdf84198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:47:19.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.240+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 0x7fdf8419e5a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400eab0 tx=0x7fdf7400ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.241+0000 7fdf80ff9640 1 -- 172.21.15.79:0/1404729259 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400ce40 con 0x7fdf841010a0 2024-04-22T06:47:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.241+0000 7fdf80ff9640 1 -- 172.21.15.79:0/1404729259 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf74011920 con 0x7fdf841010a0 2024-04-22T06:47:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.241+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf84199210 con 0x7fdf841010a0 2024-04-22T06:47:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.241+0000 7fdf80ff9640 1 -- 172.21.15.79:0/1404729259 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74010a30 con 0x7fdf841010a0 2024-04-22T06:47:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.241+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf841a5520 con 0x7fdf841010a0 2024-04-22T06:47:19.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.243+0000 7fdf80ff9640 1 -- 172.21.15.79:0/1404729259 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdf74010bd0 con 0x7fdf841010a0 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.244+0000 7fdf80ff9640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdf700a3d20 0x7fdf700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.244+0000 7fdf8c085740 1 Processor -- start 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 -- start start 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf7a7fc640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdf700a3d20 0x7fdf700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf80ff9640 1 -- 172.21.15.79:0/1404729259 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fdf740d0f30 con 0x7fdf841010a0 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 0x55baf7885960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7890e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55baf77ea1e0 con 0x55baf788ea80 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55baf7787390 con 0x55baf7885580 2024-04-22T06:47:19.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf8c085740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55baf77e61a0 con 0x55baf7885ea0 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7890e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7890e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34048/0 (socket says 172.21.15.79:34048) 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf837fe640 1 -- 172.21.15.79:0/3852753781 learned_addr learned my addr 172.21.15.79:0/3852753781 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.245+0000 7fdf7a7fc640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdf700a3d20 0x7fdf700a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c036040 tx=0x7fdf6c005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf837fe640 1 -- 172.21.15.79:0/3852753781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 msgr2=0x55baf788e540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf837fe640 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf837fe640 1 -- 172.21.15.79:0/3852753781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 msgr2=0x55baf7885960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf837fe640 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 0x55baf7885960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf837fe640 1 -- 172.21.15.79:0/3852753781 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baf75c67d0 con 0x55baf788ea80 2024-04-22T06:47:19.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf837fe640 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7890e70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009d70 tx=0x7fdf7c00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2f71986adaa078 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:19.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf797fa640 1 -- 172.21.15.79:0/3852753781 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c017070 con 0x55baf788ea80 2024-04-22T06:47:19.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf797fa640 1 -- 172.21.15.79:0/3852753781 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7c013920 con 0x55baf788ea80 2024-04-22T06:47:19.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.246+0000 7fdf797fa640 1 -- 172.21.15.79:0/3852753781 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c0127c0 con 0x55baf788ea80 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 -- 172.21.15.79:0/3852753781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 msgr2=0x55baf7890e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7890e70 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009d70 tx=0x7fdf7c00e5a0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 -- 172.21.15.79:0/3852753781 shutdown_connections 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7890e70 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3852753781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 0x55baf7885960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 -- 172.21.15.79:0/3852753781 >> 172.21.15.79:0/3852753781 conn(0x55baf787fcb0 msgr2=0x55baf7880090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 -- 172.21.15.79:0/3852753781 shutdown_connections 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.247+0000 7fdf8c085740 1 -- 172.21.15.79:0/3852753781 wait complete. 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.248+0000 7fdf8c085740 1 -- start start 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.248+0000 7fdf8c085740 1 Processor -- start 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.248+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 0x55baf788ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.248+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf788c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.249 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf8c085740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55baf7891910 con 0x55baf788ea80 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf8c085740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55baf77a32f0 con 0x55baf7885580 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf8c085740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55baf7891a90 con 0x55baf7885ea0 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf788c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf7a7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf788c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34052/0 (socket says 172.21.15.79:34052) 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 -- 172.21.15.79:0/3739628533 learned_addr learned my addr 172.21.15.79:0/3739628533 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf8c085740 10 client.0 _ll_register_callbacks cb 0x55baf77b0580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 -- 172.21.15.79:0/3739628533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 msgr2=0x55baf788bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 -- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 msgr2=0x55baf788ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 0x55baf788ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.249+0000 7fdf837fe640 1 -- 172.21.15.79:0/3739628533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baf74fc3f0 con 0x55baf788ea80 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf837fe640 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf788c4b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c014d90 tx=0x7fdf7c014dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d6c9c2d156d862 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:19.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/3739628533 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c029060 con 0x55baf788ea80 2024-04-22T06:47:19.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf7c013920 con 0x55baf788ea80 2024-04-22T06:47:19.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/3739628533 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fdf7c012710 con 0x55baf788ea80 2024-04-22T06:47:19.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/3739628533 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdf7c030050 con 0x55baf788ea80 2024-04-22T06:47:19.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf8c085740 10 client.16545 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:19.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55baf74bc0f0 con 0x55baf788ea80 2024-04-22T06:47:19.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.250+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/3739628533 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7c013920 con 0x55baf788ea80 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/3739628533 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fdf7c01ca60 con 0x55baf788ea80 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf8c085740 10 client.16545 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf8c085740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf8c085740 20 client.16545 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf789e940 0x55baf78a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf8c085740 4 client.16545 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.251+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baf77f7c00 con 0x55baf789e940 2024-04-22T06:47:19.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.253+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf789e940 0x55baf78a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.254 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.256+0000 7fdf6a7fc640 10 client.16545 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:19.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.256+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf789e940 0x55baf78a0d30 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfdae74db9be52 server_cookie=2e3e84195babee9b in_seq=0 out_seq=0 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.260+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/3739628533 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baf77f7c00 con 0x55baf789e940 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.260+0000 7fdf6a7fc640 10 client.16545 handle_command_reply: tid=0 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.260+0000 7fdf8c085740 1 client.16545 shutdown 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.260+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 msgr2=0x55baf788c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.260+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf788c4b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c014d90 tx=0x7fdf7c014dc0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.260+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 shutdown_connections 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf789e940 0x55baf78a0d30 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf788c4b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885ea0 0x55baf788bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 --2- 172.21.15.79:0/3739628533 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885580 0x55baf788ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 >> 172.21.15.79:0/3739628533 conn(0x55baf787fcb0 msgr2=0x55baf7881a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf6a7fc640 0 client.16545 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 shutdown_connections 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 1 -- 172.21.15.79:0/3739628533 wait complete. 2024-04-22T06:47:19.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.261+0000 7fdf8c085740 20 client.16545 trim_cache size 0 max 16384 2024-04-22T06:47:19.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.281+0000 7fdf8c085740 1 Processor -- start 2024-04-22T06:47:19.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.281+0000 7fdf8c085740 1 -- start start 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 0x55baf788e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 0x55baf789a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf788ea80 0x55baf788c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf8c085740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841b0760 con 0x55baf7885ea0 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf8c085740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841ae030 con 0x55baf788ea80 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf8c085740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841abe80 con 0x55baf7885580 2024-04-22T06:47:19.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 0x55baf789a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 0x55baf789a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34060/0 (socket says 172.21.15.79:34060) 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 -- 172.21.15.79:0/1025464873 learned_addr learned my addr 172.21.15.79:0/1025464873 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf837fe640 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf788ea80 0x55baf788c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 -- 172.21.15.79:0/1025464873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 msgr2=0x55baf788e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 0x55baf788e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 -- 172.21.15.79:0/1025464873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf788ea80 msgr2=0x55baf788c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf788ea80 0x55baf788c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 -- 172.21.15.79:0/1025464873 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baf7885cd0 con 0x55baf7885ea0 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.282+0000 7fdf7a7fc640 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 0x55baf789a830 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdf84199eb0 tx=0x7fdf6c03b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a760110d8a818f47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.283+0000 7fdf78ff9640 1 -- 172.21.15.79:0/1025464873 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c02ca00 con 0x55baf7885ea0 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.283+0000 7fdf78ff9640 1 -- 172.21.15.79:0/1025464873 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf6c04f070 con 0x55baf7885ea0 2024-04-22T06:47:19.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.283+0000 7fdf78ff9640 1 -- 172.21.15.79:0/1025464873 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c03d6a0 con 0x55baf7885ea0 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 -- 172.21.15.79:0/1025464873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 msgr2=0x55baf789a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 0x55baf789a830 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdf84199eb0 tx=0x7fdf6c03b000 comp rx=0 tx=0).stop 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 -- 172.21.15.79:0/1025464873 shutdown_connections 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf788ea80 0x55baf788c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf7885ea0 0x55baf789a830 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 --2- 172.21.15.79:0/1025464873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 0x55baf788e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 -- 172.21.15.79:0/1025464873 >> 172.21.15.79:0/1025464873 conn(0x55baf787fcb0 msgr2=0x55baf788b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 -- 172.21.15.79:0/1025464873 shutdown_connections 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.284+0000 7fdf8c085740 1 -- 172.21.15.79:0/1025464873 wait complete. 2024-04-22T06:47:19.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 -- start start 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 Processor -- start 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 0x55baf78888a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885ea0 0x55baf7888de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7889320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841ac4f0 con 0x55baf788ea80 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841ad960 con 0x55baf7885ea0 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf841acbc0 con 0x55baf7885580 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf8c085740 10 client.0 _ll_register_callbacks cb 0x7fdf7c009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7889320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7889320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34076/0 (socket says 172.21.15.79:34076) 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.285+0000 7fdf837fe640 1 -- 172.21.15.79:0/540399631 learned_addr learned my addr 172.21.15.79:0/540399631 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf837fe640 1 -- 172.21.15.79:0/540399631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 msgr2=0x55baf78888a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf837fe640 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 0x55baf78888a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.286 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf837fe640 1 -- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885ea0 msgr2=0x55baf7888de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf837fe640 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885ea0 0x55baf7888de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf837fe640 1 -- 172.21.15.79:0/540399631 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baf7886850 con 0x55baf788ea80 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf837fe640 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7889320 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c006fd0 tx=0x7fdf7c01a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70f28880c2207dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/540399631 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c029040 con 0x55baf788ea80 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/540399631 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fdf7c013c40 con 0x55baf788ea80 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.286+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf6c04f070 con 0x55baf788ea80 2024-04-22T06:47:19.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.287+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/540399631 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdf7c017040 con 0x55baf788ea80 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.288+0000 7fdf8c085740 10 client.16551 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.288+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55baf7904f40 con 0x55baf788ea80 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.288+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/540399631 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf6c04f070 con 0x55baf788ea80 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.288+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/540399631 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fdf7c012710 con 0x55baf788ea80 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.288+0000 7fdf8c085740 10 client.16551 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.288+0000 7fdf8c085740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.289+0000 7fdf8c085740 20 client.16551 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:19.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.289+0000 7fdf8c085740 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf7905660 0x55baf7907a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:19.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.289+0000 7fdf8c085740 4 client.16551 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:47:19.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.289+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55baf77f1a70 con 0x55baf7905660 2024-04-22T06:47:19.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.289+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf7905660 0x55baf7907a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:19.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.291+0000 7fdf82ffd640 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf7905660 0x55baf7907a80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bef69479616b24f server_cookie=75a2875a2c403092 in_seq=0 out_seq=0 2024-04-22T06:47:19.289 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.291+0000 7fdf6a7fc640 10 client.16551 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf6a7fc640 1 -- 172.21.15.79:0/540399631 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55baf77f1a70 con 0x55baf7905660 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf6a7fc640 10 client.16551 handle_command_reply: tid=0 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 client.16551 shutdown 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 msgr2=0x55baf7889320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7889320 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c006fd0 tx=0x7fdf7c01a4b0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 shutdown_connections 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55baf7905660 0x55baf7907a80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baf7885ea0 0x55baf7888de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55baf788ea80 0x55baf7889320 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf6a7fc640 0 client.16551 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.293+0000 7fdf8c085740 1 --2- 172.21.15.79:0/540399631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55baf7885580 0x55baf78888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.294+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 >> 172.21.15.79:0/540399631 conn(0x55baf787fcb0 msgr2=0x55baf787ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.294+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 shutdown_connections 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.294+0000 7fdf8c085740 1 -- 172.21.15.79:0/540399631 wait complete. 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.294+0000 7fdf8c085740 20 client.16551 trim_cache size 0 max 16384 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-22T06:47:19.293 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout: "a133a457-5e1c-46a9-ab62-e61000b8f3e3": { 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "a133a457-5e1c-46a9-ab62-e61000b8f3e3", 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:19.294 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdf700a3d20 msgr2=0x7fdf700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdf700a3d20 0x7fdf700a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c036040 tx=0x7fdf6c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 msgr2=0x7fdf8419e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 0x7fdf8419e5a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400eab0 tx=0x7fdf7400ef70 comp rx=0 tx=0).stop 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 shutdown_connections 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fdf700a3d20 0x7fdf700a61e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf84101a40 0x7fdf8419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf84106ed0 0x7fdf84198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 --2- 172.21.15.79:0/1404729259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf841010a0 0x7fdf8419e5a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 >> 172.21.15.79:0/1404729259 conn(0x7fdf84077c40 msgr2=0x7fdf84105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 shutdown_connections 2024-04-22T06:47:19.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:19.295+0000 7fdf89499640 1 -- 172.21.15.79:0/1404729259 wait complete. 2024-04-22T06:47:19.309 INFO:tasks.cephfs.filesystem:scrub status for tag:a133a457-5e1c-46a9-ab62-e61000b8f3e3 - {'path': '/', 'tag': 'a133a457-5e1c-46a9-ab62-e61000b8f3e3', 'options': 'recursive,force'} 2024-04-22T06:47:19.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:19 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:47:19.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:19 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:19.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:19 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:47:19.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:19 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:47:19.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:19 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:19.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:19 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:47:19.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:19 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:47:19.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:19 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:19.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:19 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:47:20.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:20 smithi193 ceph-mon[31815]: pgmap v94: 97 pgs: 97 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 562 MiB/s wr, 4.64k op/s 2024-04-22T06:47:20.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:20 smithi190 ceph-mon[31421]: pgmap v94: 97 pgs: 97 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 562 MiB/s wr, 4.64k op/s 2024-04-22T06:47:20.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:20 smithi079 ceph-mon[27946]: pgmap v94: 97 pgs: 97 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 562 MiB/s wr, 4.64k op/s 2024-04-22T06:47:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 -- 172.21.15.79:0/3712036365 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8858106680 msgr2=0x7f8858106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 --2- 172.21.15.79:0/3712036365 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8858106680 0x7f8858106a60 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8850009900 tx=0x7f885002bc50 comp rx=0 tx=0).stop 2024-04-22T06:47:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 -- 172.21.15.79:0/3712036365 shutdown_connections 2024-04-22T06:47:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 --2- 172.21.15.79:0/3712036365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8858100f80 0x7f885810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 --2- 172.21.15.79:0/3712036365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88581005c0 0x7f8858100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 --2- 172.21.15.79:0/3712036365 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8858106680 0x7f8858106a60 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 -- 172.21.15.79:0/3712036365 >> 172.21.15.79:0/3712036365 conn(0x7f88580773e0 msgr2=0x7f88580fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 -- 172.21.15.79:0/3712036365 shutdown_connections 2024-04-22T06:47:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.692+0000 7f885d53d640 1 -- 172.21.15.79:0/3712036365 wait complete. 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 Processor -- start 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 -- start start 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 0x7f88581986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8858100f80 0x7f8858198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8858106680 0x7f885819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8858112270 con 0x7f88581005c0 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88581120f0 con 0x7f8858100f80 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f885d53d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88581123f0 con 0x7f8858106680 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f8856ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 0x7f88581986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f8856ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 0x7f88581986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34096/0 (socket says 172.21.15.79:34096) 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.693+0000 7f8856ffd640 1 -- 172.21.15.79:0/662904584 learned_addr learned my addr 172.21.15.79:0/662904584 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f8856ffd640 1 -- 172.21.15.79:0/662904584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8858106680 msgr2=0x7f885819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f8856ffd640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8858106680 0x7f885819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f8856ffd640 1 -- 172.21.15.79:0/662904584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8858100f80 msgr2=0x7f8858198bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f8856ffd640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8858100f80 0x7f8858198bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f8856ffd640 1 -- 172.21.15.79:0/662904584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885819d650 con 0x7f88581005c0 2024-04-22T06:47:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f8856ffd640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 0x7f88581986b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8850009230 tx=0x7f885002c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.694+0000 7f883ffff640 1 -- 172.21.15.79:0/662904584 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850043020 con 0x7f88581005c0 2024-04-22T06:47:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.695+0000 7f883ffff640 1 -- 172.21.15.79:0/662904584 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88500052b0 con 0x7f88581005c0 2024-04-22T06:47:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.695+0000 7f883ffff640 1 -- 172.21.15.79:0/662904584 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850003d10 con 0x7f88581005c0 2024-04-22T06:47:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.695+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885819d8e0 con 0x7f88581005c0 2024-04-22T06:47:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.695+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885810ec10 con 0x7f88581005c0 2024-04-22T06:47:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 Processor -- start 2024-04-22T06:47:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 -- start start 2024-04-22T06:47:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 0x55eac8b34e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 0x55eac8b357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b40ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55eac8a918d0 con 0x55eac8b3e8f0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55eac8a2ee80 con 0x55eac8b35370 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f8860207740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eac8a8d890 con 0x55eac8b360e0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f883ffff640 1 -- 172.21.15.79:0/662904584 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8850046020 con 0x7f88581005c0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f88567fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b40ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f88567fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b40ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34110/0 (socket says 172.21.15.79:34110) 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.699+0000 7f88567fc640 1 -- 172.21.15.79:0/3818219543 learned_addr learned my addr 172.21.15.79:0/3818219543 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88567fc640 1 -- 172.21.15.79:0/3818219543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 msgr2=0x55eac8b34e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88567fc640 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 0x55eac8b34e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88567fc640 1 -- 172.21.15.79:0/3818219543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 msgr2=0x55eac8b357f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88567fc640 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 0x55eac8b357f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88567fc640 1 -- 172.21.15.79:0/3818219543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eac8a50320 con 0x55eac8b3e8f0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88567fc640 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b40ce0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f884c00ba50 tx=0x7f884c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1948bbf7873657b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f883d7fa640 1 -- 172.21.15.79:0/3818219543 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c016070 con 0x55eac8b3e8f0 2024-04-22T06:47:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f883ffff640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88400a3f20 0x7f88400a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f883d7fa640 1 -- 172.21.15.79:0/3818219543 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f884c0054d0 con 0x55eac8b3e8f0 2024-04-22T06:47:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f883d7fa640 1 -- 172.21.15.79:0/3818219543 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c010460 con 0x55eac8b3e8f0 2024-04-22T06:47:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f883ffff640 1 -- 172.21.15.79:0/662904584 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f88500f2aa0 con 0x7f88581005c0 2024-04-22T06:47:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.700+0000 7f88577fe640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88400a3f20 0x7f88400a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 -- 172.21.15.79:0/3818219543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 msgr2=0x55eac8b40ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f88577fe640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88400a3f20 0x7f88400a63e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f884800aab0 tx=0x7f8848009250 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b40ce0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f884c00ba50 tx=0x7f884c00bf10 comp rx=0 tx=0).stop 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 -- 172.21.15.79:0/3818219543 shutdown_connections 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 0x55eac8b357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b40ce0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 --2- 172.21.15.79:0/3818219543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 0x55eac8b34e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 -- 172.21.15.79:0/3818219543 >> 172.21.15.79:0/3818219543 conn(0x55eac8af3c90 msgr2=0x55eac8b30060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 -- 172.21.15.79:0/3818219543 shutdown_connections 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.701+0000 7f8860207740 1 -- 172.21.15.79:0/3818219543 wait complete. 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 -- start start 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 Processor -- start 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 0x55eac8b3b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 0x55eac8b3bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f88577fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f88577fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34120/0 (socket says 172.21.15.79:34120) 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f88577fe640 1 -- 172.21.15.79:0/1049425620 learned_addr learned my addr 172.21.15.79:0/1049425620 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55eac8b41620 con 0x55eac8b3e8f0 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.702+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55eac8a19eb0 con 0x55eac8b35370 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eac8b417a0 con 0x55eac8b360e0 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f88577fe640 1 -- 172.21.15.79:0/1049425620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 msgr2=0x55eac8b3bc90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f88577fe640 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 0x55eac8b3bc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f88577fe640 1 -- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 msgr2=0x55eac8b3b750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f88577fe640 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 0x55eac8b3b750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f88577fe640 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eac87a4210 con 0x55eac8b3e8f0 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f88577fe640 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3c1d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f884800dce0 tx=0x7f8848004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886b64c863a012ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f883aff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f8860207740 10 client.0 _ll_register_callbacks cb 0x55eac8a46b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.703+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f884c0054d0 con 0x55eac8b3e8f0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f883aff5640 1 -- 172.21.15.79:0/1049425620 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848023480 con 0x55eac8b3e8f0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f883aff5640 1 -- 172.21.15.79:0/1049425620 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f8848023620 con 0x55eac8b3e8f0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f883aff5640 1 -- 172.21.15.79:0/1049425620 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f8848023840 con 0x55eac8b3e8f0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 10 client.16563 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({fsmap=40}) -- 0x55eac8870620 con 0x55eac8b3e8f0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f883aff5640 1 -- 172.21.15.79:0/1049425620 <== mon.0 v2:172.21.15.79:3300/0 4 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f88480239e0 con 0x55eac8b3e8f0 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 10 client.16563 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 20 client.16563 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8b4e9c0 0x55eac8b50e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 4 client.16563 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.704+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eac8aa5880 con 0x55eac8b4e9c0 2024-04-22T06:47:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.706+0000 7f8856ffd640 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8b4e9c0 0x55eac8b50e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.711+0000 7f883aff5640 10 client.16563 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.711+0000 7f8856ffd640 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8b4e9c0 0x55eac8b50e80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d16ad745b50e31e3 server_cookie=6614ec8fb71b80ef in_seq=0 out_seq=0 2024-04-22T06:47:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.715+0000 7f883aff5640 1 -- 172.21.15.79:0/1049425620 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eac8aa5880 con 0x55eac8b4e9c0 2024-04-22T06:47:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.715+0000 7f883aff5640 10 client.16563 handle_command_reply: tid=0 2024-04-22T06:47:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.715+0000 7f8860207740 1 client.16563 shutdown 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 msgr2=0x55eac8b3c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3c1d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f884800dce0 tx=0x7f8848004b40 comp rx=0 tx=0).stop 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 shutdown_connections 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8b4e9c0 0x55eac8b50e80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3c1d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f883aff5640 0 client.16563 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b360e0 0x55eac8b3bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 --2- 172.21.15.79:0/1049425620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b35370 0x55eac8b3b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 >> 172.21.15.79:0/1049425620 conn(0x55eac8af3c90 msgr2=0x55eac8b31db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 shutdown_connections 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 1 -- 172.21.15.79:0/1049425620 wait complete. 2024-04-22T06:47:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.716+0000 7f8860207740 20 client.16563 trim_cache size 0 max 16384 2024-04-22T06:47:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.737+0000 7f8860207740 1 Processor -- start 2024-04-22T06:47:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.737+0000 7f8860207740 1 -- start start 2024-04-22T06:47:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.737+0000 7f8860207740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b4a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.738+0000 7f8860207740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b3a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.738+0000 7f8860207740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.738+0000 7f8860207740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f88581ae350 con 0x55eac8b360e0 2024-04-22T06:47:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.738+0000 7f8860207740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88581abc20 con 0x55eac8b3e8f0 2024-04-22T06:47:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.738+0000 7f8860207740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88581a9a70 con 0x55eac8b35370 2024-04-22T06:47:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.738+0000 7f88567fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b3a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b3a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34136/0 (socket says 172.21.15.79:34136) 2024-04-22T06:47:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 -- 172.21.15.79:0/3526270679 learned_addr learned my addr 172.21.15.79:0/3526270679 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f8856ffd640 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b4a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 -- 172.21.15.79:0/3526270679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 msgr2=0x55eac8b4a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b4a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 -- 172.21.15.79:0/3526270679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 msgr2=0x55eac8b3bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.739+0000 7f88567fc640 1 -- 172.21.15.79:0/3526270679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eac8b3c530 con 0x55eac8b360e0 2024-04-22T06:47:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.740+0000 7f88567fc640 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b3a8b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f884c00ba50 tx=0x7f884c00b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e3d4d295c6020f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.740+0000 7f883cff9640 1 -- 172.21.15.79:0/3526270679 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c016070 con 0x55eac8b360e0 2024-04-22T06:47:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.741+0000 7f883cff9640 1 -- 172.21.15.79:0/3526270679 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f884c0052e0 con 0x55eac8b360e0 2024-04-22T06:47:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.741+0000 7f883cff9640 1 -- 172.21.15.79:0/3526270679 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c010460 con 0x55eac8b360e0 2024-04-22T06:47:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.743+0000 7f8860207740 1 -- 172.21.15.79:0/3526270679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 msgr2=0x55eac8b3a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.743+0000 7f8860207740 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b3a8b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f884c00ba50 tx=0x7f884c00b320 comp rx=0 tx=0).stop 2024-04-22T06:47:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.744+0000 7f8860207740 1 -- 172.21.15.79:0/3526270679 shutdown_connections 2024-04-22T06:47:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.744+0000 7f8860207740 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b3bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.744+0000 7f8860207740 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b3a8b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.744+0000 7f8860207740 1 --2- 172.21.15.79:0/3526270679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b4a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.743 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.744+0000 7f8860207740 1 -- 172.21.15.79:0/3526270679 >> 172.21.15.79:0/3526270679 conn(0x55eac8af3c90 msgr2=0x55eac8b314b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.746+0000 7f8860207740 1 -- 172.21.15.79:0/3526270679 shutdown_connections 2024-04-22T06:47:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.747+0000 7f8860207740 1 -- 172.21.15.79:0/3526270679 wait complete. 2024-04-22T06:47:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.747+0000 7f8860207740 1 -- start start 2024-04-22T06:47:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.747+0000 7f8860207740 1 Processor -- start 2024-04-22T06:47:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b38710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b45a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f88581aa0e0 con 0x55eac8b360e0 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88581ab550 con 0x55eac8b3e8f0 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88581aa7b0 con 0x55eac8b35370 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.748+0000 7f8860207740 10 client.0 _ll_register_callbacks cb 0x7f884800d960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.749+0000 7f88567fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b38710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.749+0000 7f88567fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b38710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34144/0 (socket says 172.21.15.79:34144) 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.749+0000 7f88567fc640 1 -- 172.21.15.79:0/2761401172 learned_addr learned my addr 172.21.15.79:0/2761401172 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:21.747 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.749+0000 7f8856ffd640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.750+0000 7f88577fe640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b45a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.751+0000 7f88567fc640 1 -- 172.21.15.79:0/2761401172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 msgr2=0x55eac8b381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.751+0000 7f88567fc640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.751+0000 7f88567fc640 1 -- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 msgr2=0x55eac8b45a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.751+0000 7f88567fc640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b45a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.751+0000 7f88567fc640 1 -- 172.21.15.79:0/2761401172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eac8b45fd0 con 0x55eac8b360e0 2024-04-22T06:47:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.758+0000 7f88567fc640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b38710 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f884c00bf50 tx=0x7f884c004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a7694721457617 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.764+0000 7f883aff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.764+0000 7f883aff5640 1 -- 172.21.15.79:0/2761401172 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c016070 con 0x55eac8b360e0 2024-04-22T06:47:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.765+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f884c0052e0 con 0x55eac8b360e0 2024-04-22T06:47:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.765+0000 7f883aff5640 1 -- 172.21.15.79:0/2761401172 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f884c003a00 con 0x55eac8b360e0 2024-04-22T06:47:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.766+0000 7f883aff5640 1 -- 172.21.15.79:0/2761401172 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f884c034050 con 0x55eac8b360e0 2024-04-22T06:47:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.767+0000 7f8860207740 10 client.16569 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.767+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55eac8b472e0 con 0x55eac8b360e0 2024-04-22T06:47:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.771+0000 7f883aff5640 1 -- 172.21.15.79:0/2761401172 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f884c0052e0 con 0x55eac8b360e0 2024-04-22T06:47:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.771+0000 7f883aff5640 1 -- 172.21.15.79:0/2761401172 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f884c010bd0 con 0x55eac8b360e0 2024-04-22T06:47:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.773+0000 7f8860207740 10 client.16569 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.773+0000 7f8860207740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.773+0000 7f8860207740 20 client.16569 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.774+0000 7f8860207740 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8bb50a0 0x55eac8bb7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.774+0000 7f8860207740 4 client.16569 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:47:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.774+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eac8a313d0 con 0x55eac8bb50a0 2024-04-22T06:47:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.777+0000 7f8856ffd640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8bb50a0 0x55eac8bb7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.783+0000 7f8856ffd640 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8bb50a0 0x55eac8bb7490 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f66739388c2242b7 server_cookie=2eccf30067d915 in_seq=0 out_seq=0 2024-04-22T06:47:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:21.784+0000 7f883aff5640 10 client.16569 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:22.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:22 smithi193 ceph-mon[31815]: pgmap v95: 97 pgs: 97 active+clean; 14 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 721 MiB/s wr, 5.11k op/s 2024-04-22T06:47:22.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:22 smithi190 ceph-mon[31421]: pgmap v95: 97 pgs: 97 active+clean; 14 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 721 MiB/s wr, 5.11k op/s 2024-04-22T06:47:22.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:22 smithi079 ceph-mon[27946]: pgmap v95: 97 pgs: 97 active+clean; 14 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 721 MiB/s wr, 5.11k op/s 2024-04-22T06:47:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.591+0000 7f883aff5640 1 -- 172.21.15.79:0/2761401172 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eac8a313d0 con 0x55eac8bb50a0 2024-04-22T06:47:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.591+0000 7f883aff5640 10 client.16569 handle_command_reply: tid=0 2024-04-22T06:47:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.591+0000 7f8860207740 1 client.16569 shutdown 2024-04-22T06:47:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 msgr2=0x55eac8b38710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b38710 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f884c00bf50 tx=0x7f884c004f70 comp rx=0 tx=0).stop 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 shutdown_connections 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55eac8bb50a0 0x55eac8bb7490 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eac8b3e8f0 0x55eac8b45a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55eac8b360e0 0x55eac8b38710 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 --2- 172.21.15.79:0/2761401172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eac8b35370 0x55eac8b381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 >> 172.21.15.79:0/2761401172 conn(0x55eac8af3c90 msgr2=0x55eac8b26040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f883aff5640 0 client.16569 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.593+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 shutdown_connections 2024-04-22T06:47:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.594+0000 7f8860207740 1 -- 172.21.15.79:0/2761401172 wait complete. 2024-04-22T06:47:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.594+0000 7f8860207740 20 client.16569 trim_cache size 0 max 16384 2024-04-22T06:47:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:47:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:47:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:47:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:47:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.602+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88400a3f20 msgr2=0x7f88400a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.602+0000 7f885d53d640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88400a3f20 0x7f88400a63e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f884800aab0 tx=0x7f8848009250 comp rx=0 tx=0).stop 2024-04-22T06:47:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.602+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 msgr2=0x7f88581986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.602+0000 7f885d53d640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 0x7f88581986b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f8850009230 tx=0x7f885002c820 comp rx=0 tx=0).stop 2024-04-22T06:47:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.606+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 shutdown_connections 2024-04-22T06:47:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.606+0000 7f885d53d640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88400a3f20 0x7f88400a63e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.606+0000 7f885d53d640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8858100f80 0x7f8858198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.606+0000 7f885d53d640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8858106680 0x7f885819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.606+0000 7f885d53d640 1 --2- 172.21.15.79:0/662904584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88581005c0 0x7f88581986b0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.606+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 >> 172.21.15.79:0/662904584 conn(0x7f88580773e0 msgr2=0x7f88580fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.607+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 shutdown_connections 2024-04-22T06:47:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:24.607+0000 7f885d53d640 1 -- 172.21.15.79:0/662904584 wait complete. 2024-04-22T06:47:24.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:24 smithi193 ceph-mon[31815]: pgmap v96: 97 pgs: 97 active+clean; 14 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 678 MiB/s wr, 4.14k op/s 2024-04-22T06:47:24.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:24 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:24.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:24 smithi190 ceph-mon[31421]: pgmap v96: 97 pgs: 97 active+clean; 14 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 678 MiB/s wr, 4.14k op/s 2024-04-22T06:47:24.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:24 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:24 smithi079 ceph-mon[27946]: pgmap v96: 97 pgs: 97 active+clean; 14 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 678 MiB/s wr, 4.14k op/s 2024-04-22T06:47:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:24 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:26.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:26 smithi193 ceph-mon[31815]: pgmap v97: 97 pgs: 97 active+clean; 14 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 570 MiB/s wr, 3.20k op/s 2024-04-22T06:47:26.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:26 smithi190 ceph-mon[31421]: pgmap v97: 97 pgs: 97 active+clean; 14 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 570 MiB/s wr, 3.20k op/s 2024-04-22T06:47:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:26 smithi079 ceph-mon[27946]: pgmap v97: 97 pgs: 97 active+clean; 14 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 570 MiB/s wr, 3.20k op/s 2024-04-22T06:47:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:28 smithi079 ceph-mon[27946]: pgmap v98: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 858 MiB/s wr, 3.98k op/s 2024-04-22T06:47:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:28 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:47:29.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:28 smithi193 ceph-mon[31815]: pgmap v98: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 858 MiB/s wr, 3.98k op/s 2024-04-22T06:47:29.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:28 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:47:29.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:28 smithi190 ceph-mon[31421]: pgmap v98: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 858 MiB/s wr, 3.98k op/s 2024-04-22T06:47:29.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:28 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:47:30.990 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:30 smithi193 ceph-mon[31815]: pgmap v99: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 559 MiB/s wr, 1.49k op/s 2024-04-22T06:47:31.001 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:30 smithi079 ceph-mon[27946]: pgmap v99: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 559 MiB/s wr, 1.49k op/s 2024-04-22T06:47:31.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:30 smithi190 ceph-mon[31421]: pgmap v99: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 559 MiB/s wr, 1.49k op/s 2024-04-22T06:47:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:32 smithi079 ceph-mon[27946]: pgmap v100: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 773 MiB/s wr, 2.28k op/s 2024-04-22T06:47:33.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:32 smithi193 ceph-mon[31815]: pgmap v100: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 773 MiB/s wr, 2.28k op/s 2024-04-22T06:47:33.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:32 smithi190 ceph-mon[31421]: pgmap v100: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 773 MiB/s wr, 2.28k op/s 2024-04-22T06:47:34.994 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:34 smithi079 ceph-mon[27946]: pgmap v101: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 739 MiB/s wr, 2.26k op/s 2024-04-22T06:47:34.995 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:34 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:34 smithi193 ceph-mon[31815]: pgmap v101: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 739 MiB/s wr, 2.26k op/s 2024-04-22T06:47:35.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:34 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:35.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:34 smithi190 ceph-mon[31421]: pgmap v101: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 739 MiB/s wr, 2.26k op/s 2024-04-22T06:47:35.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:34 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:37.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:36 smithi193 ceph-mon[31815]: pgmap v102: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 626 MiB/s wr, 2.02k op/s 2024-04-22T06:47:37.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:36 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:37.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:36 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:37.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:36 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:47:37.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:36 smithi190 ceph-mon[31421]: pgmap v102: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 626 MiB/s wr, 2.02k op/s 2024-04-22T06:47:37.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:36 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:37.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:36 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:37.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:36 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:47:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:36 smithi079 ceph-mon[27946]: pgmap v102: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 626 MiB/s wr, 2.02k op/s 2024-04-22T06:47:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:36 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:36 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:36 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:47:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.912+0000 7fc33c10e640 1 -- 172.21.15.79:0/1305544441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334110b20 msgr2=0x7fc334109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.912+0000 7fc33c10e640 1 --2- 172.21.15.79:0/1305544441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334110b20 0x7fc334109490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc324009890 tx=0x7fc32402bbf0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.912+0000 7fc33c10e640 1 -- 172.21.15.79:0/1305544441 shutdown_connections 2024-04-22T06:47:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.913+0000 7fc33c10e640 1 --2- 172.21.15.79:0/1305544441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc334110b20 0x7fc334109490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.913+0000 7fc33c10e640 1 --2- 172.21.15.79:0/1305544441 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc33411b730 0x7fc33411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.913+0000 7fc33c10e640 1 --2- 172.21.15.79:0/1305544441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc334110740 0x7fc334108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.913+0000 7fc33c10e640 1 -- 172.21.15.79:0/1305544441 >> 172.21.15.79:0/1305544441 conn(0x7fc33406d020 msgr2=0x7fc33406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.918+0000 7fc33c10e640 1 -- 172.21.15.79:0/1305544441 shutdown_connections 2024-04-22T06:47:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.925+0000 7fc33c10e640 1 -- 172.21.15.79:0/1305544441 wait complete. 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.927+0000 7fc33c10e640 1 Processor -- start 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.928+0000 7fc33c10e640 1 -- start start 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.928+0000 7fc33c10e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 0x7fc33411a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.928+0000 7fc33c10e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc33411b730 0x7fc33411aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.929+0000 7fc33c10e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc334114ac0 0x7fc334114f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.929+0000 7fc33c10e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc3341113e0 con 0x7fc334110740 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.929+0000 7fc33c10e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc334111260 con 0x7fc33411b730 2024-04-22T06:47:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.929+0000 7fc33c10e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc334111560 con 0x7fc334114ac0 2024-04-22T06:47:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.934+0000 7fc33a6ab640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc334114ac0 0x7fc334114f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.937+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc33411b730 0x7fc33411aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.937+0000 7fc339eaa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 0x7fc33411a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.937+0000 7fc339eaa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 0x7fc33411a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39534/0 (socket says 172.21.15.79:39534) 2024-04-22T06:47:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.937+0000 7fc339eaa640 1 -- 172.21.15.79:0/914154433 learned_addr learned my addr 172.21.15.79:0/914154433 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.946+0000 7fc339eaa640 1 -- 172.21.15.79:0/914154433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc334114ac0 msgr2=0x7fc334114f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.946+0000 7fc339eaa640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc334114ac0 0x7fc334114f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.946+0000 7fc339eaa640 1 -- 172.21.15.79:0/914154433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc33411b730 msgr2=0x7fc33411aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.946+0000 7fc339eaa640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc33411b730 0x7fc33411aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.946+0000 7fc339eaa640 1 -- 172.21.15.79:0/914154433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3341156a0 con 0x7fc334110740 2024-04-22T06:47:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.953+0000 7fc339eaa640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 0x7fc33411a9b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00eab0 tx=0x7fc32c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.959+0000 7fc32affd640 1 -- 172.21.15.79:0/914154433 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c00ce40 con 0x7fc334110740 2024-04-22T06:47:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.959+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3341bc8c0 con 0x7fc334110740 2024-04-22T06:47:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.959+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3341bccc0 con 0x7fc334110740 2024-04-22T06:47:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.959+0000 7fc32affd640 1 -- 172.21.15.79:0/914154433 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c011920 con 0x7fc334110740 2024-04-22T06:47:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.959+0000 7fc32affd640 1 -- 172.21.15.79:0/914154433 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c00ce40 con 0x7fc334110740 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.969+0000 7fc33eddb740 1 Processor -- start 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.970+0000 7fc33eddb740 1 -- start start 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.970+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14d6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.971+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d6b60 0x5611d14d6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.971+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 0x5611d14e24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.971+0000 7fc33eddb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5611d143b8d0 con 0x5611d14e00e0 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.971+0000 7fc33eddb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5611d13d8e80 con 0x5611d14d78d0 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.971+0000 7fc33eddb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5611d1437890 con 0x5611d14d6b60 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.973+0000 7fc33a6ab640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.973+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 0x5611d14e24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.973+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 0x5611d14e24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39536/0 (socket says 172.21.15.79:39536) 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.973+0000 7fc3396a9640 1 -- 172.21.15.79:0/219840836 learned_addr learned my addr 172.21.15.79:0/219840836 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.975+0000 7fc3396a9640 1 -- 172.21.15.79:0/219840836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d6b60 msgr2=0x5611d14d6fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.975+0000 7fc3396a9640 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d6b60 0x5611d14d6fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.975+0000 7fc3396a9640 1 -- 172.21.15.79:0/219840836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 msgr2=0x5611d14d6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.975+0000 7fc3396a9640 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14d6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.975+0000 7fc3396a9640 1 -- 172.21.15.79:0/219840836 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611d13fa320 con 0x5611d14e00e0 2024-04-22T06:47:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.978+0000 7fc3396a9640 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 0x5611d14e24d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc324009890 tx=0x7fc324009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe6b6f7e054999e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.981+0000 7fc32affd640 1 -- 172.21.15.79:0/914154433 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc32c011470 con 0x7fc334110740 2024-04-22T06:47:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.981+0000 7fc31ffff640 1 -- 172.21.15.79:0/219840836 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324042020 con 0x5611d14e00e0 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.982+0000 7fc31ffff640 1 -- 172.21.15.79:0/219840836 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc324034a20 con 0x5611d14e00e0 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.983+0000 7fc33eddb740 1 -- 172.21.15.79:0/219840836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 msgr2=0x5611d14e24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.983+0000 7fc33eddb740 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 0x5611d14e24d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc324009890 tx=0x7fc324009500 comp rx=0 tx=0).stop 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.986+0000 7fc33eddb740 1 -- 172.21.15.79:0/219840836 shutdown_connections 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.986+0000 7fc33eddb740 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14e00e0 0x5611d14e24d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.986+0000 7fc33eddb740 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d6b60 0x5611d14d6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.986+0000 7fc33eddb740 1 --2- 172.21.15.79:0/219840836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14d6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.986+0000 7fc33eddb740 1 -- 172.21.15.79:0/219840836 >> 172.21.15.79:0/219840836 conn(0x5611d149dc90 msgr2=0x5611d14d1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.986+0000 7fc32affd640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc3200a42a0 0x7fc3200a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.987+0000 7fc32affd640 1 -- 172.21.15.79:0/914154433 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc32c00cc40 con 0x7fc334110740 2024-04-22T06:47:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.990+0000 7fc33eddb740 1 -- 172.21.15.79:0/219840836 shutdown_connections 2024-04-22T06:47:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.994+0000 7fc3396a9640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc3200a42a0 0x7fc3200a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.994+0000 7fc33eddb740 1 -- 172.21.15.79:0/219840836 wait complete. 2024-04-22T06:47:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.995+0000 7fc33eddb740 1 -- start start 2024-04-22T06:47:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.995+0000 7fc33eddb740 1 Processor -- start 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.998+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14dfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.998+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d78d0 0x5611d14dcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.998+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14e00e0 0x5611d14dd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.998+0000 7fc33eddb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5611d14e2e00 con 0x5611d14d6b60 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.998+0000 7fc33eddb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5611d13c3eb0 con 0x5611d14e00e0 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.998+0000 7fc33eddb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5611d14e2f80 con 0x5611d14d78d0 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:37.999+0000 7fc33eddb740 10 client.0 _ll_register_callbacks cb 0x5611d13f0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.000+0000 7fc339eaa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14dfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.000+0000 7fc339eaa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14dfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39550/0 (socket says 172.21.15.79:39550) 2024-04-22T06:47:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.000+0000 7fc339eaa640 1 -- 172.21.15.79:0/4068485672 learned_addr learned my addr 172.21.15.79:0/4068485672 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.001+0000 7fc339eaa640 1 -- 172.21.15.79:0/4068485672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d78d0 msgr2=0x5611d14dcda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.001+0000 7fc339eaa640 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d78d0 0x5611d14dcda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.001+0000 7fc339eaa640 1 -- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14e00e0 msgr2=0x5611d14dd2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.001+0000 7fc339eaa640 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14e00e0 0x5611d14dd2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.001+0000 7fc339eaa640 1 -- 172.21.15.79:0/4068485672 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611d114e210 con 0x5611d14d6b60 2024-04-22T06:47:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.004+0000 7fc339eaa640 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14dfa10 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0cdbc0 tx=0x7fc32c09bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102c91837b58acc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.013+0000 7fc31d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.013+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4068485672 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c0cdff0 con 0x5611d14d6b60 2024-04-22T06:47:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.013+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4068485672 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc32c09ba00 con 0x5611d14d6b60 2024-04-22T06:47:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.014+0000 7fc3396a9640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc3200a42a0 0x7fc3200a6760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc324009860 tx=0x7fc32402bbf0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.016+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc324034a20 con 0x5611d14d6b60 2024-04-22T06:47:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.021+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4068485672 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc32c00ce40 con 0x5611d14d6b60 2024-04-22T06:47:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.023+0000 7fc33eddb740 10 client.16581 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.023+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5611d121a620 con 0x5611d14d6b60 2024-04-22T06:47:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.025+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4068485672 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc324034a20 con 0x5611d14d6b60 2024-04-22T06:47:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4068485672 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fc32c021020 con 0x5611d14d6b60 2024-04-22T06:47:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc33eddb740 10 client.16581 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc33eddb740 10 client.16581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:38.025 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc33eddb740 20 client.16581 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:38.025 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d14f06a0 0x5611d14f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.025 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc33eddb740 4 client.16581 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:38.025 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.026+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611d144f880 con 0x5611d14f06a0 2024-04-22T06:47:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.034+0000 7fc3396a9640 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d14f06a0 0x5611d14f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.043+0000 7fc3396a9640 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d14f06a0 0x5611d14f2b20 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24e9cdbc501b647 server_cookie=6b933142eac33cf5 in_seq=0 out_seq=0 2024-04-22T06:47:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.043+0000 7fc31d7fa640 10 client.16581 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.048+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4068485672 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611d144f880 con 0x5611d14f06a0 2024-04-22T06:47:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.049+0000 7fc31d7fa640 10 client.16581 handle_command_reply: tid=0 2024-04-22T06:47:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.049+0000 7fc33eddb740 1 client.16581 shutdown 2024-04-22T06:47:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.051+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 msgr2=0x5611d14dfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.051+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14dfa10 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0cdbc0 tx=0x7fc32c09bfe0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.056+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 shutdown_connections 2024-04-22T06:47:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.056+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d14f06a0 0x5611d14f2b20 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.056+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14e00e0 0x5611d14dd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.056+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14d78d0 0x5611d14dcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.056+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4068485672 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14dfa10 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.056+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 >> 172.21.15.79:0/4068485672 conn(0x5611d149dc90 msgr2=0x5611d14d35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.057+0000 7fc31d7fa640 0 client.16581 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.057+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 shutdown_connections 2024-04-22T06:47:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.061+0000 7fc33eddb740 1 -- 172.21.15.79:0/4068485672 wait complete. 2024-04-22T06:47:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.061+0000 7fc33eddb740 20 client.16581 trim_cache size 0 max 16384 2024-04-22T06:47:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.113+0000 7fc33eddb740 1 Processor -- start 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.113+0000 7fc33eddb740 1 -- start start 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.113+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14ebde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.113+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14df4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.113+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14dfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.113+0000 7fc33eddb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc318011380 con 0x5611d14d6b60 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.114+0000 7fc33eddb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc31800ec50 con 0x5611d14d78d0 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.114+0000 7fc33eddb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc31800caa0 con 0x5611d14e00e0 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.114+0000 7fc33a6ab640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14dfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.115+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14df4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.115+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14df4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:42428/0 (socket says 172.21.15.79:42428) 2024-04-22T06:47:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.115+0000 7fc3396a9640 1 -- 172.21.15.79:0/1558595813 learned_addr learned my addr 172.21.15.79:0/1558595813 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.117+0000 7fc3396a9640 1 -- 172.21.15.79:0/1558595813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 msgr2=0x5611d14dfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.117+0000 7fc3396a9640 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14dfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.117+0000 7fc3396a9640 1 -- 172.21.15.79:0/1558595813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 msgr2=0x5611d14ebde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.117+0000 7fc3396a9640 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14ebde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.117+0000 7fc3396a9640 1 -- 172.21.15.79:0/1558595813 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611d14ccf30 con 0x5611d14d78d0 2024-04-22T06:47:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.123+0000 7fc3396a9640 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14df4d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc324034d90 tx=0x7fc32404a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb59c34f36e742e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.127+0000 7fc31f7fe640 1 -- 172.21.15.79:0/1558595813 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324042050 con 0x5611d14d78d0 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.127+0000 7fc31f7fe640 1 -- 172.21.15.79:0/1558595813 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc324058020 con 0x5611d14d78d0 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.128+0000 7fc33eddb740 1 -- 172.21.15.79:0/1558595813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 msgr2=0x5611d14df4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.128+0000 7fc33eddb740 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14df4d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc324034d90 tx=0x7fc32404a000 comp rx=0 tx=0).stop 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.129+0000 7fc33eddb740 1 -- 172.21.15.79:0/1558595813 shutdown_connections 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.129+0000 7fc33eddb740 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d78d0 0x5611d14df4d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.129+0000 7fc33eddb740 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14dfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.129+0000 7fc33eddb740 1 --2- 172.21.15.79:0/1558595813 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d6b60 0x5611d14ebde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.129+0000 7fc33eddb740 1 -- 172.21.15.79:0/1558595813 >> 172.21.15.79:0/1558595813 conn(0x5611d149dc90 msgr2=0x5611d14dcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.130+0000 7fc33eddb740 1 -- 172.21.15.79:0/1558595813 shutdown_connections 2024-04-22T06:47:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.130+0000 7fc33eddb740 1 -- 172.21.15.79:0/1558595813 wait complete. 2024-04-22T06:47:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.131+0000 7fc33eddb740 1 -- start start 2024-04-22T06:47:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.131+0000 7fc33eddb740 1 Processor -- start 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d6b60 0x5611d14e3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 0x5611d14e4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14e47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc31800d110 con 0x5611d14d78d0 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc31800e580 con 0x5611d14d6b60 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc31800d7e0 con 0x5611d14e00e0 2024-04-22T06:47:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.132+0000 7fc33eddb740 10 client.0 _ll_register_callbacks cb 0x7fc32c09c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.133+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 0x5611d14e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.133+0000 7fc33a6ab640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14e47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.133+0000 7fc3396a9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 0x5611d14e4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39566/0 (socket says 172.21.15.79:39566) 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.133+0000 7fc3396a9640 1 -- 172.21.15.79:0/4123303524 learned_addr learned my addr 172.21.15.79:0/4123303524 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.133+0000 7fc339eaa640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d6b60 0x5611d14e3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc3396a9640 1 -- 172.21.15.79:0/4123303524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 msgr2=0x5611d14e47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc3396a9640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14e47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc3396a9640 1 -- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d6b60 msgr2=0x5611d14e3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc3396a9640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d6b60 0x5611d14e3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc3396a9640 1 -- 172.21.15.79:0/4123303524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611d1555cd0 con 0x5611d14d78d0 2024-04-22T06:47:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc33a6ab640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14e47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:47:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.134+0000 7fc3396a9640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 0x5611d14e4280 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc324004530 tx=0x7fc3240346c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef4a81ee4c2406e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.135+0000 7fc31d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.135+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4123303524 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324042070 con 0x5611d14d78d0 2024-04-22T06:47:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.135+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc324058020 con 0x5611d14d78d0 2024-04-22T06:47:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.136+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4123303524 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc324034440 con 0x5611d14d78d0 2024-04-22T06:47:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.136+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4123303524 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc324005270 con 0x5611d14d78d0 2024-04-22T06:47:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.136+0000 7fc33eddb740 10 client.16584 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.136+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5611d1556fc0 con 0x5611d14d78d0 2024-04-22T06:47:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4123303524 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc324058020 con 0x5611d14d78d0 2024-04-22T06:47:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4123303524 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fc32402c880 con 0x5611d14d78d0 2024-04-22T06:47:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc33eddb740 10 client.16584 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc33eddb740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc33eddb740 20 client.16584 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d15575e0 0x5611d1559a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc33eddb740 4 client.16584 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:47:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.138+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5611d13db3d0 con 0x5611d15575e0 2024-04-22T06:47:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.141+0000 7fc339eaa640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d15575e0 0x5611d1559a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.142+0000 7fc339eaa640 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d15575e0 0x5611d1559a00 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b64e7b92dc05ce1 server_cookie=76952e879df08b8c in_seq=0 out_seq=0 2024-04-22T06:47:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:38.143+0000 7fc31d7fa640 10 client.16584 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:39.043 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:38 smithi079 ceph-mon[27946]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 982 MiB/s wr, 2.44k op/s 2024-04-22T06:47:39.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:38 smithi193 ceph-mon[31815]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 982 MiB/s wr, 2.44k op/s 2024-04-22T06:47:39.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:38 smithi190 ceph-mon[31421]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 982 MiB/s wr, 2.44k op/s 2024-04-22T06:47:41.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:40 smithi193 ceph-mon[31815]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 694 MiB/s wr, 1.67k op/s 2024-04-22T06:47:41.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:40 smithi190 ceph-mon[31421]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 694 MiB/s wr, 1.67k op/s 2024-04-22T06:47:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:40 smithi079 ceph-mon[27946]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 694 MiB/s wr, 1.67k op/s 2024-04-22T06:47:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:42 smithi193 ceph-mon[31815]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 930 MiB/s wr, 2.21k op/s 2024-04-22T06:47:43.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:42 smithi190 ceph-mon[31421]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 930 MiB/s wr, 2.21k op/s 2024-04-22T06:47:43.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:42 smithi079 ceph-mon[27946]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 930 MiB/s wr, 2.21k op/s 2024-04-22T06:47:45.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:44 smithi193 ceph-mon[31815]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 856 MiB/s wr, 1.71k op/s 2024-04-22T06:47:45.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:44 smithi190 ceph-mon[31421]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 856 MiB/s wr, 1.71k op/s 2024-04-22T06:47:45.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:44 smithi079 ceph-mon[27946]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 856 MiB/s wr, 1.71k op/s 2024-04-22T06:47:47.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:46 smithi193 ceph-mon[31815]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 731 MiB/s wr, 1.26k op/s 2024-04-22T06:47:47.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:46 smithi190 ceph-mon[31421]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 731 MiB/s wr, 1.26k op/s 2024-04-22T06:47:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:46 smithi079 ceph-mon[27946]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 731 MiB/s wr, 1.26k op/s 2024-04-22T06:47:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc31d7fa640 1 -- 172.21.15.79:0/4123303524 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5611d13db3d0 con 0x5611d15575e0 2024-04-22T06:47:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc31d7fa640 10 client.16584 handle_command_reply: tid=0 2024-04-22T06:47:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 client.16584 shutdown 2024-04-22T06:47:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 msgr2=0x5611d14e4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 0x5611d14e4280 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc324004530 tx=0x7fc3240346c0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 shutdown_connections 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5611d15575e0 0x5611d1559a00 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611d14e00e0 0x5611d14e47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5611d14d78d0 0x5611d14e4280 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 --2- 172.21.15.79:0/4123303524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5611d14d6b60 0x5611d14e3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 >> 172.21.15.79:0/4123303524 conn(0x5611d149dc90 msgr2=0x5611d14d7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc31d7fa640 0 client.16584 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.517+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 shutdown_connections 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.518+0000 7fc33eddb740 1 -- 172.21.15.79:0/4123303524 wait complete. 2024-04-22T06:47:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.518+0000 7fc33eddb740 20 client.16584 trim_cache size 0 max 16384 2024-04-22T06:47:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:47:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:47:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:47:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:47:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.520+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc3200a42a0 msgr2=0x7fc3200a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.520+0000 7fc33c10e640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc3200a42a0 0x7fc3200a6760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc324009860 tx=0x7fc32402bbf0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.520+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 msgr2=0x7fc33411a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.520+0000 7fc33c10e640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 0x7fc33411a9b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00eab0 tx=0x7fc32c00ef70 comp rx=0 tx=0).stop 2024-04-22T06:47:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.523+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 shutdown_connections 2024-04-22T06:47:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.523+0000 7fc33c10e640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc3200a42a0 0x7fc3200a6760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.523+0000 7fc33c10e640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc33411b730 0x7fc33411aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.523+0000 7fc33c10e640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc334114ac0 0x7fc334114f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.523+0000 7fc33c10e640 1 --2- 172.21.15.79:0/914154433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc334110740 0x7fc33411a9b0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.523+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 >> 172.21.15.79:0/914154433 conn(0x7fc33406d020 msgr2=0x7fc33411bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.525+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 shutdown_connections 2024-04-22T06:47:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:47:47.526+0000 7fc33c10e640 1 -- 172.21.15.79:0/914154433 wait complete. 2024-04-22T06:47:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:48 smithi193 ceph-mon[31815]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 GiB/s wr, 1.91k op/s 2024-04-22T06:47:49.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:48 smithi190 ceph-mon[31421]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 GiB/s wr, 1.91k op/s 2024-04-22T06:47:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:48 smithi079 ceph-mon[27946]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 GiB/s wr, 1.91k op/s 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:47:49.311 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:47:49.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.763+0000 7fd174cb2640 1 -- 172.21.15.79:0/3724718296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1701106d0 msgr2=0x7fd17010e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.763+0000 7fd174cb2640 1 --2- 172.21.15.79:0/3724718296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1701106d0 0x7fd17010e270 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd164009930 tx=0x7fd16402bc60 comp rx=0 tx=0).stop 2024-04-22T06:47:49.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.766+0000 7fd174cb2640 1 -- 172.21.15.79:0/3724718296 shutdown_connections 2024-04-22T06:47:49.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.766+0000 7fd174cb2640 1 --2- 172.21.15.79:0/3724718296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd17010e7b0 0x7fd17010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.766+0000 7fd174cb2640 1 --2- 172.21.15.79:0/3724718296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd17011b730 0x7fd17011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.766+0000 7fd174cb2640 1 --2- 172.21.15.79:0/3724718296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd1701106d0 0x7fd17010e270 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.766+0000 7fd174cb2640 1 -- 172.21.15.79:0/3724718296 >> 172.21.15.79:0/3724718296 conn(0x7fd17006d070 msgr2=0x7fd17006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.774+0000 7fd174cb2640 1 -- 172.21.15.79:0/3724718296 shutdown_connections 2024-04-22T06:47:49.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.775+0000 7fd174cb2640 1 -- 172.21.15.79:0/3724718296 wait complete. 2024-04-22T06:47:49.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.777+0000 7fd174cb2640 1 Processor -- start 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.780+0000 7fd174cb2640 1 -- start start 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.780+0000 7fd174cb2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd17010e7b0 0x7fd170118c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.780+0000 7fd174cb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1701106d0 0x7fd1701191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.781+0000 7fd174cb2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd17011b730 0x7fd170111f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.781+0000 7fd174cb2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd170121100 con 0x7fd17011b730 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.781+0000 7fd174cb2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd170120f80 con 0x7fd1701106d0 2024-04-22T06:47:49.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.781+0000 7fd174cb2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd170121280 con 0x7fd17010e7b0 2024-04-22T06:47:49.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.783+0000 7fd16dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1701106d0 0x7fd1701191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.788+0000 7fd16ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd17011b730 0x7fd170111f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.788+0000 7fd16ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd17011b730 0x7fd170111f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47050/0 (socket says 172.21.15.79:47050) 2024-04-22T06:47:49.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.788+0000 7fd16ed9d640 1 -- 172.21.15.79:0/1518214748 learned_addr learned my addr 172.21.15.79:0/1518214748 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:49.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.788+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd17010e7b0 0x7fd170118c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.795+0000 7fd16dd9b640 1 -- 172.21.15.79:0/1518214748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd17010e7b0 msgr2=0x7fd170118c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.795+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd17010e7b0 0x7fd170118c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.795+0000 7fd16dd9b640 1 -- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd17011b730 msgr2=0x7fd170111f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.795+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd17011b730 0x7fd170111f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.795+0000 7fd16dd9b640 1 -- 172.21.15.79:0/1518214748 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd170112820 con 0x7fd1701106d0 2024-04-22T06:47:49.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.799+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1701106d0 0x7fd1701191c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd160009fd0 tx=0x7fd16000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:49.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.802+0000 7fd15f7fe640 1 -- 172.21.15.79:0/1518214748 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160015020 con 0x7fd1701106d0 2024-04-22T06:47:49.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.802+0000 7fd15f7fe640 1 -- 172.21.15.79:0/1518214748 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd160011ce0 con 0x7fd1701106d0 2024-04-22T06:47:49.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.802+0000 7fd15f7fe640 1 -- 172.21.15.79:0/1518214748 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160010e50 con 0x7fd1701106d0 2024-04-22T06:47:49.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.802+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd170112b10 con 0x7fd1701106d0 2024-04-22T06:47:49.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.803+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd170116a40 con 0x7fd1701106d0 2024-04-22T06:47:49.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.811+0000 7fd15f7fe640 1 -- 172.21.15.79:0/1518214748 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd160018020 con 0x7fd1701106d0 2024-04-22T06:47:49.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.813+0000 7fd15f7fe640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd1580a40d0 0x7fd1580a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.814+0000 7fd15f7fe640 1 -- 172.21.15.79:0/1518214748 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fd1600d0ed0 con 0x7fd1701106d0 2024-04-22T06:47:49.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.814+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd1580a40d0 0x7fd1580a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.816+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd1580a40d0 0x7fd1580a6590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd164006010 tx=0x7fd164000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:49.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.817+0000 7fd177976740 1 Processor -- start 2024-04-22T06:47:49.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.818+0000 7fd177976740 1 -- start start 2024-04-22T06:47:49.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.821+0000 7fd177976740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611580 0x55b115611960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.821+0000 7fd177976740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b11561a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.821+0000 7fd177976740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b11561ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.822+0000 7fd177976740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b1155761e0 con 0x55b11561aa80 2024-04-22T06:47:49.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.822+0000 7fd177976740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b115513390 con 0x55b115611ea0 2024-04-22T06:47:49.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.822+0000 7fd177976740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b1155721a0 con 0x55b115611580 2024-04-22T06:47:49.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.823+0000 7fd16dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b11561a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.823+0000 7fd16dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b11561a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:56612/0 (socket says 172.21.15.79:56612) 2024-04-22T06:47:49.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.823+0000 7fd16dd9b640 1 -- 172.21.15.79:0/1425055251 learned_addr learned my addr 172.21.15.79:0/1425055251 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:49.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.823+0000 7fd16ed9d640 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b11561ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.824+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611580 0x55b115611960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.827+0000 7fd16e59c640 1 -- 172.21.15.79:0/1425055251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 msgr2=0x55b11561a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.827+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b11561a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.827+0000 7fd16e59c640 1 -- 172.21.15.79:0/1425055251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 msgr2=0x55b11561ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.827+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b11561ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.827+0000 7fd16e59c640 1 -- 172.21.15.79:0/1425055251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1153527d0 con 0x55b115611580 2024-04-22T06:47:49.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.829+0000 7fd16e59c640 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611580 0x55b115611960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd164009850 tx=0x7fd16403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cf42fe23318d974 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:49.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.833+0000 7fd15cff9640 1 -- 172.21.15.79:0/1425055251 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16404e070 con 0x55b115611580 2024-04-22T06:47:49.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.833+0000 7fd15cff9640 1 -- 172.21.15.79:0/1425055251 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1640046c0 con 0x55b115611580 2024-04-22T06:47:49.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.833+0000 7fd15cff9640 1 -- 172.21.15.79:0/1425055251 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164003850 con 0x55b115611580 2024-04-22T06:47:49.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.835+0000 7fd177976740 1 -- 172.21.15.79:0/1425055251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611580 msgr2=0x55b115611960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.835+0000 7fd177976740 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611580 0x55b115611960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd164009850 tx=0x7fd16403f000 comp rx=0 tx=0).stop 2024-04-22T06:47:49.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.835+0000 7fd177976740 1 -- 172.21.15.79:0/1425055251 shutdown_connections 2024-04-22T06:47:49.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.835+0000 7fd177976740 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b11561a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.835+0000 7fd177976740 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b11561ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.835+0000 7fd177976740 1 --2- 172.21.15.79:0/1425055251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611580 0x55b115611960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.836+0000 7fd177976740 1 -- 172.21.15.79:0/1425055251 >> 172.21.15.79:0/1425055251 conn(0x55b11560bcb0 msgr2=0x55b11560c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.837+0000 7fd177976740 1 -- 172.21.15.79:0/1425055251 shutdown_connections 2024-04-22T06:47:49.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.838+0000 7fd177976740 1 -- 172.21.15.79:0/1425055251 wait complete. 2024-04-22T06:47:49.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.839+0000 7fd177976740 1 -- start start 2024-04-22T06:47:49.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.839+0000 7fd177976740 1 Processor -- start 2024-04-22T06:47:49.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.840+0000 7fd177976740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115617a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.840+0000 7fd177976740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115617fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.840+0000 7fd177976740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b11561aa80 0x55b1156184f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd177976740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b11561d810 con 0x55b115611580 2024-04-22T06:47:49.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd177976740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b11552f2f0 con 0x55b11561aa80 2024-04-22T06:47:49.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd177976740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b11561d990 con 0x55b115611ea0 2024-04-22T06:47:49.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd177976740 10 client.0 _ll_register_callbacks cb 0x55b11553c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:49.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd16dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115617fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd16dd9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115617fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:36994/0 (socket says 172.21.15.79:36994) 2024-04-22T06:47:49.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd16dd9b640 1 -- 172.21.15.79:0/3744135837 learned_addr learned my addr 172.21.15.79:0/3744135837 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:49.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.841+0000 7fd16e59c640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115617a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.842+0000 7fd16ed9d640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b11561aa80 0x55b1156184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.842+0000 7fd16e59c640 1 -- 172.21.15.79:0/3744135837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 msgr2=0x55b115617fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.842+0000 7fd16e59c640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115617fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.842+0000 7fd16e59c640 1 -- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b11561aa80 msgr2=0x55b1156184f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.842+0000 7fd16e59c640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b11561aa80 0x55b1156184f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.843+0000 7fd16e59c640 1 -- 172.21.15.79:0/3744135837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1152883f0 con 0x55b115611580 2024-04-22T06:47:49.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.843+0000 7fd16e59c640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115617a70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd164036040 tx=0x7fd164004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d777a022ff89a09f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:49.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.844+0000 7fd155ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:49.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.844+0000 7fd155ffb640 1 -- 172.21.15.79:0/3744135837 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16405a020 con 0x55b115611580 2024-04-22T06:47:49.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.844+0000 7fd155ffb640 1 -- 172.21.15.79:0/3744135837 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fd1640037f0 con 0x55b115611580 2024-04-22T06:47:49.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.844+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1640046c0 con 0x55b115611580 2024-04-22T06:47:49.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.845+0000 7fd155ffb640 1 -- 172.21.15.79:0/3744135837 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd16404e040 con 0x55b115611580 2024-04-22T06:47:49.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.845+0000 7fd177976740 10 client.16596 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:49.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.845+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b1152480f0 con 0x55b115611580 2024-04-22T06:47:49.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.846+0000 7fd155ffb640 1 -- 172.21.15.79:0/3744135837 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd164005500 con 0x55b115611580 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.847+0000 7fd155ffb640 1 -- 172.21.15.79:0/3744135837 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fd164053990 con 0x55b115611580 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.847+0000 7fd177976740 10 client.16596 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.847+0000 7fd177976740 10 client.16596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.847+0000 7fd177976740 20 client.16596 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.847+0000 7fd177976740 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b11562a890 0x55b11562cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.848+0000 7fd177976740 4 client.16596 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:49.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.848+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd1640046c0 con 0x55b11562a890 2024-04-22T06:47:49.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.850+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b11562a890 0x55b11562cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.855+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b11562a890 0x55b11562cc80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8e8344025e811f server_cookie=1df9374b3155feb9 in_seq=0 out_seq=0 2024-04-22T06:47:49.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.855+0000 7fd155ffb640 10 client.16596 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:49.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.860+0000 7fd155ffb640 1 -- 172.21.15.79:0/3744135837 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fd1640046c0 con 0x55b11562a890 2024-04-22T06:47:49.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.860+0000 7fd155ffb640 10 client.16596 handle_command_reply: tid=0 2024-04-22T06:47:49.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.860+0000 7fd177976740 1 client.16596 shutdown 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.862+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 msgr2=0x55b115617a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.862+0000 7fd177976740 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115617a70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd164036040 tx=0x7fd164004aa0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.862+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 shutdown_connections 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.862+0000 7fd177976740 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b11562a890 0x55b11562cc80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.862+0000 7fd177976740 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b11561aa80 0x55b1156184f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.863+0000 7fd177976740 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115617fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.863+0000 7fd177976740 1 --2- 172.21.15.79:0/3744135837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115617a70 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.863+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 >> 172.21.15.79:0/3744135837 conn(0x55b11560bcb0 msgr2=0x55b11560da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.863+0000 7fd155ffb640 0 client.16596 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:49.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.869+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 shutdown_connections 2024-04-22T06:47:49.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.869+0000 7fd177976740 1 -- 172.21.15.79:0/3744135837 wait complete. 2024-04-22T06:47:49.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.870+0000 7fd177976740 20 client.16596 trim_cache size 0 max 16384 2024-04-22T06:47:49.915 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.916+0000 7fd177976740 1 Processor -- start 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.916+0000 7fd177976740 1 -- start start 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.916+0000 7fd177976740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115626730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.916+0000 7fd177976740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b115618310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd177976740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b11561aa80 0x55b11560abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd177976740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd150011320 con 0x55b115611580 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd177976740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd15000ebf0 con 0x55b115611ea0 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd16e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115626730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd16e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115626730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47084/0 (socket says 172.21.15.79:47084) 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd16e59c640 1 -- 172.21.15.79:0/2810745676 learned_addr learned my addr 172.21.15.79:0/2810745676 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:49.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.917+0000 7fd177976740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd15000ca40 con 0x55b11561aa80 2024-04-22T06:47:49.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.919+0000 7fd16e59c640 1 -- 172.21.15.79:0/2810745676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b11561aa80 msgr2=0x55b11560abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.919+0000 7fd16e59c640 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b11561aa80 0x55b11560abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.919+0000 7fd16e59c640 1 -- 172.21.15.79:0/2810745676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 msgr2=0x55b115618310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.919+0000 7fd16e59c640 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b115618310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.919+0000 7fd16e59c640 1 -- 172.21.15.79:0/2810745676 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b115627260 con 0x55b115611580 2024-04-22T06:47:49.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.919+0000 7fd16e59c640 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115626730 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd164009850 tx=0x7fd164004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f4e426758d001f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:49.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.920+0000 7fd157fff640 1 -- 172.21.15.79:0/2810745676 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16406c070 con 0x55b115611580 2024-04-22T06:47:49.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.920+0000 7fd157fff640 1 -- 172.21.15.79:0/2810745676 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16404ad60 con 0x55b115611580 2024-04-22T06:47:49.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.920+0000 7fd157fff640 1 -- 172.21.15.79:0/2810745676 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164042070 con 0x55b115611580 2024-04-22T06:47:49.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 -- 172.21.15.79:0/2810745676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 msgr2=0x55b115626730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115626730 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd164009850 tx=0x7fd164004f10 comp rx=0 tx=0).stop 2024-04-22T06:47:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 -- 172.21.15.79:0/2810745676 shutdown_connections 2024-04-22T06:47:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115611ea0 0x55b115618310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b11561aa80 0x55b11560abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 --2- 172.21.15.79:0/2810745676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b115611580 0x55b115626730 secure :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd164009850 tx=0x7fd164004f10 comp rx=0 tx=0).stop 2024-04-22T06:47:49.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.922+0000 7fd177976740 1 -- 172.21.15.79:0/2810745676 >> 172.21.15.79:0/2810745676 conn(0x55b11560bcb0 msgr2=0x55b11560a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.924+0000 7fd177976740 1 -- 172.21.15.79:0/2810745676 shutdown_connections 2024-04-22T06:47:49.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.925+0000 7fd177976740 1 -- 172.21.15.79:0/2810745676 wait complete. 2024-04-22T06:47:49.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.925+0000 7fd177976740 1 -- start start 2024-04-22T06:47:49.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 Processor -- start 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115623100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b115614440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 0x55b115614dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd15000d0b0 con 0x55b11561aa80 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd15000e520 con 0x55b115614980 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd15000d780 con 0x55b115611ea0 2024-04-22T06:47:49.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.926+0000 7fd177976740 10 client.0 _ll_register_callbacks cb 0x7fd168009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.928+0000 7fd16ed9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 0x55b115614dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.928+0000 7fd16ed9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 0x55b115614dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:56630/0 (socket says 172.21.15.79:56630) 2024-04-22T06:47:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.928+0000 7fd16ed9d640 1 -- 172.21.15.79:0/3315289472 learned_addr learned my addr 172.21.15.79:0/3315289472 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:49.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.928+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b115614440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.930+0000 7fd16dd9b640 1 -- 172.21.15.79:0/3315289472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 msgr2=0x55b115623100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:49.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.930+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115623100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.930+0000 7fd16dd9b640 1 -- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 msgr2=0x55b115614dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:49.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.930+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 0x55b115614dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.930+0000 7fd16dd9b640 1 -- 172.21.15.79:0/3315289472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11568fc40 con 0x55b11561aa80 2024-04-22T06:47:49.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.931+0000 7fd16ed9d640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 0x55b115614dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:47:49.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.931+0000 7fd16dd9b640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b115614440 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd1600997d0 tx=0x7fd1600ce620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19345b8113f5d1cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:49.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.932+0000 7fd155ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:49.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.933+0000 7fd155ffb640 1 -- 172.21.15.79:0/3315289472 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600ba070 con 0x55b11561aa80 2024-04-22T06:47:49.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.933+0000 7fd155ffb640 1 -- 172.21.15.79:0/3315289472 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fd160099470 con 0x55b11561aa80 2024-04-22T06:47:49.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.934+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd16404ad60 con 0x55b11561aa80 2024-04-22T06:47:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.935+0000 7fd155ffb640 1 -- 172.21.15.79:0/3315289472 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd1600ce720 con 0x55b11561aa80 2024-04-22T06:47:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.935+0000 7fd177976740 10 client.16602 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:49.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.935+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b115690f50 con 0x55b11561aa80 2024-04-22T06:47:49.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.936+0000 7fd155ffb640 1 -- 172.21.15.79:0/3315289472 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd16404ad60 con 0x55b11561aa80 2024-04-22T06:47:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd155ffb640 1 -- 172.21.15.79:0/3315289472 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fd160098b50 con 0x55b11561aa80 2024-04-22T06:47:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd177976740 10 client.16602 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd177976740 10 client.16602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd177976740 20 client.16602 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd177976740 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1156917a0 0x55b115693bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:49.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd177976740 4 client.16602 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:47:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.938+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b115583c00 con 0x55b1156917a0 2024-04-22T06:47:49.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.944+0000 7fd16e59c640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1156917a0 0x55b115693bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:49.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.947+0000 7fd16e59c640 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1156917a0 0x55b115693bc0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=230bfebbc4d23bf2 server_cookie=f760dba492687aef in_seq=0 out_seq=0 2024-04-22T06:47:49.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.947+0000 7fd155ffb640 10 client.16602 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:49.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.951+0000 7fd155ffb640 1 -- 172.21.15.79:0/3315289472 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b115583c00 con 0x55b1156917a0 2024-04-22T06:47:49.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.952+0000 7fd155ffb640 10 client.16602 handle_command_reply: tid=0 2024-04-22T06:47:49.950 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.952+0000 7fd177976740 1 client.16602 shutdown 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.953+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 msgr2=0x55b115614440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.953+0000 7fd177976740 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b115614440 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd1600997d0 tx=0x7fd1600ce620 comp rx=0 tx=0).stop 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 shutdown_connections 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b1156917a0 0x55b115693bc0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b115614980 0x55b115614dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b11561aa80 0x55b115614440 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 --2- 172.21.15.79:0/3315289472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b115611ea0 0x55b115623100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 >> 172.21.15.79:0/3315289472 conn(0x55b11560bcb0 msgr2=0x55b115616860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd155ffb640 0 client.16602 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 shutdown_connections 2024-04-22T06:47:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.955+0000 7fd177976740 1 -- 172.21.15.79:0/3315289472 wait complete. 2024-04-22T06:47:49.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.960+0000 7fd177976740 20 client.16602 trim_cache size 0 max 16384 2024-04-22T06:47:49.960 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:47:49.960 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:47:49.960 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.964+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd1580a40d0 msgr2=0x7fd1580a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.964+0000 7fd174cb2640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd1580a40d0 0x7fd1580a6590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd164006010 tx=0x7fd164000bc0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.964+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1701106d0 msgr2=0x7fd1701191c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1701106d0 0x7fd1701191c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd160009fd0 tx=0x7fd16000b870 comp rx=0 tx=0).stop 2024-04-22T06:47:49.963 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 shutdown_connections 2024-04-22T06:47:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd1580a40d0 0x7fd1580a6590 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1701106d0 0x7fd1701191c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd17011b730 0x7fd170111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 --2- 172.21.15.79:0/1518214748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd17010e7b0 0x7fd170118c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.964 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.965+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 >> 172.21.15.79:0/1518214748 conn(0x7fd17006d070 msgr2=0x7fd17011a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.968+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 shutdown_connections 2024-04-22T06:47:49.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:49.968+0000 7fd174cb2640 1 -- 172.21.15.79:0/1518214748 wait complete. 2024-04-22T06:47:50.043 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:47:50.043 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:47:50.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.476+0000 7f3b887db640 1 -- 172.21.15.79:0/3051718800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b80109b70 msgr2=0x7f3b8011d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:50.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.476+0000 7f3b887db640 1 --2- 172.21.15.79:0/3051718800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b80109b70 0x7f3b8011d7a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009ca0 tx=0x7f3b7802c140 comp rx=0 tx=0).stop 2024-04-22T06:47:50.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.484+0000 7f3b887db640 1 -- 172.21.15.79:0/3051718800 shutdown_connections 2024-04-22T06:47:50.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.484+0000 7f3b887db640 1 --2- 172.21.15.79:0/3051718800 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b80109b70 0x7f3b8011d7a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:50.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.484+0000 7f3b887db640 1 --2- 172.21.15.79:0/3051718800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b80110b30 0x7f3b80109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:50.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.484+0000 7f3b887db640 1 --2- 172.21.15.79:0/3051718800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b801106c0 0x7f3b80108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:50.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.484+0000 7f3b887db640 1 -- 172.21.15.79:0/3051718800 >> 172.21.15.79:0/3051718800 conn(0x7f3b8006e410 msgr2=0x7f3b800719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:50.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.491+0000 7f3b887db640 1 -- 172.21.15.79:0/3051718800 shutdown_connections 2024-04-22T06:47:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.492+0000 7f3b887db640 1 -- 172.21.15.79:0/3051718800 wait complete. 2024-04-22T06:47:50.491 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.493+0000 7f3b887db640 1 Processor -- start 2024-04-22T06:47:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.493+0000 7f3b887db640 1 -- start start 2024-04-22T06:47:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.494+0000 7f3b887db640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b801106c0 0x7f3b8011d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.494+0000 7f3b887db640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 0x7f3b80117500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.494+0000 7f3b887db640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b80117a40 0x7f3b80117ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.494+0000 7f3b887db640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3b80111810 con 0x7f3b801106c0 2024-04-22T06:47:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.494+0000 7f3b887db640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b80111690 con 0x7f3b80110b30 2024-04-22T06:47:50.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.494+0000 7f3b887db640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3b80111990 con 0x7f3b80117a40 2024-04-22T06:47:50.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.499+0000 7f3b85d76640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 0x7f3b80117500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:50.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.499+0000 7f3b85d76640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 0x7f3b80117500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:56642/0 (socket says 172.21.15.79:56642) 2024-04-22T06:47:50.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.499+0000 7f3b85d76640 1 -- 172.21.15.79:0/370746445 learned_addr learned my addr 172.21.15.79:0/370746445 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:50.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.501+0000 7f3b86d78640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b80117a40 0x7f3b80117ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.502+0000 7f3b85d76640 1 -- 172.21.15.79:0/370746445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b80117a40 msgr2=0x7f3b80117ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.502+0000 7f3b85d76640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b80117a40 0x7f3b80117ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.502+0000 7f3b85d76640 1 -- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b801106c0 msgr2=0x7f3b8011d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.502+0000 7f3b85d76640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b801106c0 0x7f3b8011d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:50.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.502+0000 7f3b85d76640 1 -- 172.21.15.79:0/370746445 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b801bc5b0 con 0x7f3b80110b30 2024-04-22T06:47:50.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.505+0000 7f3b85d76640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 0x7f3b80117500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000b4a0 tx=0x7f3b7000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:50.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.506+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70014070 con 0x7f3b80110b30 2024-04-22T06:47:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.506+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b801bc8a0 con 0x7f3b80110b30 2024-04-22T06:47:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.506+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b801bcdf0 con 0x7f3b80110b30 2024-04-22T06:47:50.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.509+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b80073e40 con 0x7f3b80110b30 2024-04-22T06:47:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.509+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b70010ac0 con 0x7f3b80110b30 2024-04-22T06:47:50.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.509+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70017be0 con 0x7f3b80110b30 2024-04-22T06:47:50.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.521+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b70017070 con 0x7f3b80110b30 2024-04-22T06:47:50.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.522+0000 7f3b777fe640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3b6c0a3ea0 0x7f3b6c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:50.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.523+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f3b700d0ee0 con 0x7f3b80110b30 2024-04-22T06:47:50.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.523+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b700d1400 con 0x7f3b80110b30 2024-04-22T06:47:50.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.525+0000 7f3b86577640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3b6c0a3ea0 0x7f3b6c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:50.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:50.558+0000 7f3b86577640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3b6c0a3ea0 0x7f3b6c0a6360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0096f0 tx=0x7f3b7c006c40 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:50 smithi193 ceph-mon[31815]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 729 MiB/s wr, 1.48k op/s 2024-04-22T06:47:51.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:50 smithi190 ceph-mon[31421]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 729 MiB/s wr, 1.48k op/s 2024-04-22T06:47:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:50 smithi079 ceph-mon[27946]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 729 MiB/s wr, 1.48k op/s 2024-04-22T06:47:51.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.428+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b80068de0 con 0x7f3b80110b30 2024-04-22T06:47:51.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.430+0000 7f3b777fe640 1 -- 172.21.15.79:0/370746445 <== mon.2 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7f3b7009d5f0 con 0x7f3b80110b30 2024-04-22T06:47:51.435 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:47:51.435 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:46:45.384403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25265,"mds_2":25261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":35,"state":"up:active","state_seq":22,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25261,24467,25265]},"id":1}]} 2024-04-22T06:47:51.435 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 40 2024-04-22T06:47:51.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3b6c0a3ea0 msgr2=0x7f3b6c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3b6c0a3ea0 0x7f3b6c0a6360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0096f0 tx=0x7f3b7c006c40 comp rx=0 tx=0).stop 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 msgr2=0x7f3b80117500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 0x7f3b80117500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000b4a0 tx=0x7f3b7000b960 comp rx=0 tx=0).stop 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 shutdown_connections 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3b6c0a3ea0 0x7f3b6c0a6360 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0096f0 tx=0x7f3b7c006c40 comp rx=0 tx=0).stop 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b80110b30 0x7f3b80117500 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b80117a40 0x7f3b80117ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 --2- 172.21.15.79:0/370746445 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3b801106c0 0x7f3b8011d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.454+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 >> 172.21.15.79:0/370746445 conn(0x7f3b8006e410 msgr2=0x7f3b80070080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:51.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.457+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 shutdown_connections 2024-04-22T06:47:51.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:51.457+0000 7f3b887db640 1 -- 172.21.15.79:0/370746445 wait complete. 2024-04-22T06:47:51.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:47:51.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.000+0000 7f4b0eb10640 1 -- 172.21.15.79:0/119276404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0811b730 msgr2=0x7f4b0811db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:51.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.000+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/119276404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0811b730 0x7f4b0811db20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4afc01a940 tx=0x7f4afc03cca0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.001+0000 7f4b0eb10640 1 -- 172.21.15.79:0/119276404 shutdown_connections 2024-04-22T06:47:52.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.001+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/119276404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b0810e7d0 0x7f4b0810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.001+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/119276404 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0811b730 0x7f4b0811db20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.001+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/119276404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b081107a0 0x7f4b0810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.001+0000 7f4b0eb10640 1 -- 172.21.15.79:0/119276404 >> 172.21.15.79:0/119276404 conn(0x7f4b0806d060 msgr2=0x7f4b0806d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.002+0000 7f4b0eb10640 1 -- 172.21.15.79:0/119276404 shutdown_connections 2024-04-22T06:47:52.002 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.003+0000 7f4b0eb10640 1 -- 172.21.15.79:0/119276404 wait complete. 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.004+0000 7f4b0eb10640 1 Processor -- start 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 -- start start 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b0810e7d0 0x7f4b0811aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b081107a0 0x7f4b08116f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b0811b730 0x7f4b08117470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4b08113830 con 0x7f4b081107a0 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081136b0 con 0x7f4b0810e7d0 2024-04-22T06:47:52.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.005+0000 7f4b0eb10640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081139b0 con 0x7f4b0811b730 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.006+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b081107a0 0x7f4b08116f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.006+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b081107a0 0x7f4b08116f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47144/0 (socket says 172.21.15.79:47144) 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.006+0000 7f4b07fff640 1 -- 172.21.15.79:0/2402243350 learned_addr learned my addr 172.21.15.79:0/2402243350 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.006+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b0811b730 0x7f4b08117470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.007+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/2402243350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b0810e7d0 msgr2=0x7f4b0811aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.007+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b0810e7d0 0x7f4b0811aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.007+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b081107a0 msgr2=0x7f4b08116f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.007+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b081107a0 0x7f4b08116f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.007+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/2402243350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b08117b70 con 0x7f4b0811b730 2024-04-22T06:47:52.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.009+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b0811b730 0x7f4b08117470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4afc01a860 tx=0x7f4afc004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.009+0000 7f4b05ffb640 1 -- 172.21.15.79:0/2402243350 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc052020 con 0x7f4b0811b730 2024-04-22T06:47:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.009+0000 7f4b05ffb640 1 -- 172.21.15.79:0/2402243350 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4afc03dd00 con 0x7f4b0811b730 2024-04-22T06:47:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.010+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b081ab4c0 con 0x7f4b0811b730 2024-04-22T06:47:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.011+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b081ab9a0 con 0x7f4b0811b730 2024-04-22T06:47:52.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.011+0000 7f4b05ffb640 1 -- 172.21.15.79:0/2402243350 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc003c90 con 0x7f4b0811b730 2024-04-22T06:47:52.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.015+0000 7f4b117d8740 1 Processor -- start 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 -- start start 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c0a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56445c00c1f0 con 0x56445c0b0a00 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56445bfa94b0 con 0x56445c0a74d0 2024-04-22T06:47:52.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.016+0000 7f4b117d8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56445c0081b0 con 0x56445c0a7e80 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.018+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c0a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.018+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.019+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47158/0 (socket says 172.21.15.79:47158) 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.019+0000 7f4b07fff640 1 -- 172.21.15.79:0/1299397722 learned_addr learned my addr 172.21.15.79:0/1299397722 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.019+0000 7f4b05ffb640 1 -- 172.21.15.79:0/2402243350 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4afc055020 con 0x7f4b0811b730 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.020+0000 7f4b07fff640 1 -- 172.21.15.79:0/1299397722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 msgr2=0x56445c0b0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.020+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0b0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.020+0000 7f4b07fff640 1 -- 172.21.15.79:0/1299397722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 msgr2=0x56445c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.020+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.020+0000 7f4b07fff640 1 -- 172.21.15.79:0/1299397722 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56445bf9b1b0 con 0x56445c0a74d0 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.020+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c0a78b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4b00008a10 tx=0x7f4b000099d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64dfb2ac2119596f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.020 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.021+0000 7f4aeeffd640 1 -- 172.21.15.79:0/1299397722 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0003f070 con 0x56445c0a74d0 2024-04-22T06:47:52.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.021+0000 7f4aeeffd640 1 -- 172.21.15.79:0/1299397722 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0003a3f0 con 0x56445c0a74d0 2024-04-22T06:47:52.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.022+0000 7f4b117d8740 1 -- 172.21.15.79:0/1299397722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 msgr2=0x56445c0a78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.022 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.022+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c0a78b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4b00008a10 tx=0x7f4b000099d0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.023+0000 7f4b117d8740 1 -- 172.21.15.79:0/1299397722 shutdown_connections 2024-04-22T06:47:52.023 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.023+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.023+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.023+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1299397722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c0a78b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.024 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.023+0000 7f4b117d8740 1 -- 172.21.15.79:0/1299397722 >> 172.21.15.79:0/1299397722 conn(0x56445c0a1ba0 msgr2=0x56445c0a1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.024+0000 7f4b117d8740 1 -- 172.21.15.79:0/1299397722 shutdown_connections 2024-04-22T06:47:52.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.024+0000 7f4b117d8740 1 -- 172.21.15.79:0/1299397722 wait complete. 2024-04-22T06:47:52.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.024+0000 7f4b117d8740 1 -- start start 2024-04-22T06:47:52.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.024+0000 7f4b117d8740 1 Processor -- start 2024-04-22T06:47:52.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 0x56445c0b0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a7e80 0x56445c0ad620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 0x56445c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56445c0b3a10 con 0x56445c0a74d0 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56445bfc5500 con 0x56445c0a7e80 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56445c0b3b90 con 0x56445c0b0a00 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.025+0000 7f4b117d8740 10 client.0 _ll_register_callbacks cb 0x56445bf93b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:52.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.026+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 0x56445c0b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.026+0000 7f4b07fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 0x56445c0b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47168/0 (socket says 172.21.15.79:47168) 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.026+0000 7f4b07fff640 1 -- 172.21.15.79:0/1268849986 learned_addr learned my addr 172.21.15.79:0/1268849986 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.027+0000 7f4b07fff640 1 -- 172.21.15.79:0/1268849986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 msgr2=0x56445c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.027+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 0x56445c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.027+0000 7f4b07fff640 1 -- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a7e80 msgr2=0x56445c0ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.027+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a7e80 0x56445c0ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.027+0000 7f4b07fff640 1 -- 172.21.15.79:0/1268849986 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56445bd1e3f0 con 0x56445c0a74d0 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.027+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 0x56445c0b0290 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009a40 tx=0x7f4b0003aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cceb7c53bcb9f34 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.028+0000 7f4ae7fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.029+0000 7f4b05ffb640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4af00a3d20 0x7f4af00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.029+0000 7f4b05ffb640 1 -- 172.21.15.79:0/2402243350 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f4afc103790 con 0x7f4b0811b730 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.029+0000 7f4b0c8ac640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4af00a3d20 0x7f4af00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.030 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.029+0000 7f4ae7fff640 1 -- 172.21.15.79:0/1268849986 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0003f030 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.029+0000 7f4ae7fff640 1 -- 172.21.15.79:0/1268849986 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f4b00040070 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.030+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b0003a3f0 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.030+0000 7f4ae7fff640 1 -- 172.21.15.79:0/1268849986 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4b00003a30 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.030+0000 7f4b117d8740 10 client.16617 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.030+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56445bde87d0 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.030+0000 7f4ae7fff640 1 -- 172.21.15.79:0/1268849986 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0003a3f0 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.030+0000 7f4ae7fff640 1 -- 172.21.15.79:0/1268849986 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f4b00039070 con 0x56445c0a74d0 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.031+0000 7f4b117d8740 10 client.16617 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.031+0000 7f4b117d8740 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.031+0000 7f4b117d8740 20 client.16617 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.031+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c0c1050 0x56445c0c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.031+0000 7f4b117d8740 4 client.16617 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:52.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.031+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56445bfab930 con 0x56445c0c1050 2024-04-22T06:47:52.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.032+0000 7f4b0c8ac640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4af00a3d20 0x7f4af00a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x56445c0aebf0 tx=0x7f4af8006cb0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.032+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c0c1050 0x56445c0c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.036+0000 7f4b07fff640 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c0c1050 0x56445c0c34d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f1824b657c461cd server_cookie=938b9cbe97932d74 in_seq=0 out_seq=0 2024-04-22T06:47:52.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.036+0000 7f4ae7fff640 10 client.16617 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:52.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4ae7fff640 1 -- 172.21.15.79:0/1268849986 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56445bfab930 con 0x56445c0c1050 2024-04-22T06:47:52.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4ae7fff640 10 client.16617 handle_command_reply: tid=0 2024-04-22T06:47:52.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 client.16617 shutdown 2024-04-22T06:47:52.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 msgr2=0x56445c0b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 0x56445c0b0290 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009a40 tx=0x7f4b0003aae0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 shutdown_connections 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c0c1050 0x56445c0c34d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a7e80 0x56445c0ad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 0x56445c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 --2- 172.21.15.79:0/1268849986 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a74d0 0x56445c0b0290 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.040+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 >> 172.21.15.79:0/1268849986 conn(0x56445c0a1ba0 msgr2=0x56445c0a38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.041+0000 7f4ae7fff640 0 client.16617 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.041+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 shutdown_connections 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.041+0000 7f4b117d8740 1 -- 172.21.15.79:0/1268849986 wait complete. 2024-04-22T06:47:52.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.041+0000 7f4b117d8740 20 client.16617 trim_cache size 0 max 16384 2024-04-22T06:47:52.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.073+0000 7f4b117d8740 1 Processor -- start 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.073+0000 7f4b117d8740 1 -- start start 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.074+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c09cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.074+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0bc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.074+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.074+0000 7f4b117d8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081b6e10 con 0x56445c0b0a00 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.074+0000 7f4b117d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081b46e0 con 0x56445c0a74d0 2024-04-22T06:47:52.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.074+0000 7f4b117d8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081b2530 con 0x56445c0a7e80 2024-04-22T06:47:52.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.075+0000 7f4b0d0ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.075+0000 7f4b0d0ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47182/0 (socket says 172.21.15.79:47182) 2024-04-22T06:47:52.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.075+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/4198462536 learned_addr learned my addr 172.21.15.79:0/4198462536 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.077+0000 7f4b07fff640 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0bc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.078+0000 7f4b07fff640 1 -- 172.21.15.79:0/4198462536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 msgr2=0x56445c09cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:52.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.078+0000 7f4b07fff640 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c09cec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.078+0000 7f4b07fff640 1 -- 172.21.15.79:0/4198462536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 msgr2=0x56445c0a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.078+0000 7f4b07fff640 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.078+0000 7f4b07fff640 1 -- 172.21.15.79:0/4198462536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56445c0b3bc0 con 0x56445c0a7e80 2024-04-22T06:47:52.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.080+0000 7f4b07fff640 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0bc950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b00039920 tx=0x7f4b0000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=139774742a6126e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.080+0000 7f4aee7fc640 1 -- 172.21.15.79:0/4198462536 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0005e070 con 0x56445c0a7e80 2024-04-22T06:47:52.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.081+0000 7f4aee7fc640 1 -- 172.21.15.79:0/4198462536 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0003a3f0 con 0x56445c0a7e80 2024-04-22T06:47:52.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.083+0000 7f4b117d8740 1 -- 172.21.15.79:0/4198462536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 msgr2=0x56445c0bc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.083+0000 7f4b117d8740 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0bc950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b00039920 tx=0x7f4b0000b040 comp rx=0 tx=0).stop 2024-04-22T06:47:52.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.085+0000 7f4b117d8740 1 -- 172.21.15.79:0/4198462536 shutdown_connections 2024-04-22T06:47:52.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.085+0000 7f4b117d8740 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0b0a00 0x56445c0a89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.085+0000 7f4b117d8740 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0a7e80 0x56445c0bc950 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.085+0000 7f4b117d8740 1 --2- 172.21.15.79:0/4198462536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0a74d0 0x56445c09cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.085+0000 7f4b117d8740 1 -- 172.21.15.79:0/4198462536 >> 172.21.15.79:0/4198462536 conn(0x56445c0a1ba0 msgr2=0x56445c0a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.090+0000 7f4b117d8740 1 -- 172.21.15.79:0/4198462536 shutdown_connections 2024-04-22T06:47:52.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.095+0000 7f4b117d8740 1 -- 172.21.15.79:0/4198462536 wait complete. 2024-04-22T06:47:52.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.095+0000 7f4b117d8740 1 -- start start 2024-04-22T06:47:52.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.095+0000 7f4b117d8740 1 Processor -- start 2024-04-22T06:47:52.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.096+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a7e80 0x56445c0b92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 0x56445c0aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b117d8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 0x56445c0a9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b117d8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081b2ba0 con 0x56445c0a7e80 2024-04-22T06:47:52.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b117d8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081b4010 con 0x56445c0aaf40 2024-04-22T06:47:52.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b117d8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b081b3270 con 0x56445c0b0a00 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b117d8740 10 client.0 _ll_register_callbacks cb 0x7f4b0003a960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b0d0ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 0x56445c0a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b0d0ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 0x56445c0a9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:56698/0 (socket says 172.21.15.79:56698) 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.097+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/904245601 learned_addr learned my addr 172.21.15.79:0/904245601 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.098+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/904245601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 msgr2=0x56445c0aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.098+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 0x56445c0aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.098+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/904245601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a7e80 msgr2=0x56445c0b92f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:52.097 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.098+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a7e80 0x56445c0b92f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.098+0000 7f4b0d0ad640 1 -- 172.21.15.79:0/904245601 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56445c0ab690 con 0x56445c0aaf40 2024-04-22T06:47:52.098 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.099+0000 7f4b0d0ad640 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 0x56445c0a9890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4afc03d2b0 tx=0x7f4afc03df60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fcba0c6c423a6f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.104+0000 7f4ae7fff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:47:52.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.104+0000 7f4ae7fff640 1 -- 172.21.15.79:0/904245601 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc0cf200 con 0x56445c0aaf40 2024-04-22T06:47:52.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.104+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b0003a3f0 con 0x56445c0aaf40 2024-04-22T06:47:52.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.104+0000 7f4ae7fff640 1 -- 172.21.15.79:0/904245601 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f4afc0b8b70 con 0x56445c0aaf40 2024-04-22T06:47:52.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.107+0000 7f4ae7fff640 1 -- 172.21.15.79:0/904245601 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4afc0cd800 con 0x56445c0aaf40 2024-04-22T06:47:52.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.108+0000 7f4b117d8740 10 client.25667 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:52.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.108+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56445c126c50 con 0x56445c0aaf40 2024-04-22T06:47:52.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.109+0000 7f4ae7fff640 1 -- 172.21.15.79:0/904245601 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0003a3f0 con 0x56445c0aaf40 2024-04-22T06:47:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.109+0000 7f4ae7fff640 1 -- 172.21.15.79:0/904245601 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f4afc0ee420 con 0x56445c0aaf40 2024-04-22T06:47:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.110+0000 7f4b117d8740 10 client.25667 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.110+0000 7f4b117d8740 10 client.25667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.110+0000 7f4b117d8740 20 client.25667 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.110+0000 7f4b117d8740 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c1271c0 0x56445c129620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.110+0000 7f4b117d8740 4 client.25667 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:47:52.109 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.110+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56445c019c10 con 0x56445c1271c0 2024-04-22T06:47:52.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.112+0000 7f4b0c8ac640 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c1271c0 0x56445c129620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.126+0000 7f4b0c8ac640 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c1271c0 0x56445c129620 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cb3b452413ac3a1 server_cookie=1cd4a6db26daed2d in_seq=0 out_seq=0 2024-04-22T06:47:52.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.126+0000 7f4ae7fff640 10 client.25667 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.131+0000 7f4ae7fff640 1 -- 172.21.15.79:0/904245601 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56445c019c10 con 0x56445c1271c0 2024-04-22T06:47:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.132+0000 7f4ae7fff640 10 client.25667 handle_command_reply: tid=0 2024-04-22T06:47:52.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.134+0000 7f4b117d8740 1 client.25667 shutdown 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 msgr2=0x56445c0a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 0x56445c0a9890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4afc03d2b0 tx=0x7f4afc03df60 comp rx=0 tx=0).stop 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 shutdown_connections 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56445c1271c0 0x56445c129620 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56445c0aaf40 0x56445c0a9890 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56445c0b0a00 0x56445c0aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 --2- 172.21.15.79:0/904245601 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56445c0a7e80 0x56445c0b92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.136+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 >> 172.21.15.79:0/904245601 conn(0x56445c0a1ba0 msgr2=0x56445c0a8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.137+0000 7f4ae7fff640 0 client.25667 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:52.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.141+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 shutdown_connections 2024-04-22T06:47:52.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.144+0000 7f4b117d8740 1 -- 172.21.15.79:0/904245601 wait complete. 2024-04-22T06:47:52.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.144+0000 7f4b117d8740 20 client.25667 trim_cache size 0 max 16384 2024-04-22T06:47:52.158 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:47:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.152+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4af00a3d20 msgr2=0x7f4af00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.152+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4af00a3d20 0x7f4af00a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x56445c0aebf0 tx=0x7f4af8006cb0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.152+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b0811b730 msgr2=0x7f4b08117470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.153+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b0811b730 0x7f4b08117470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4afc01a860 tx=0x7f4afc004040 comp rx=0 tx=0).stop 2024-04-22T06:47:52.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.166+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 shutdown_connections 2024-04-22T06:47:52.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.166+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b0811b730 0x7f4b08117470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.166+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b081107a0 0x7f4b08116f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.166+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4af00a3d20 0x7f4af00a61e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.166+0000 7f4b0eb10640 1 --2- 172.21.15.79:0/2402243350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b0810e7d0 0x7f4b0811aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.166+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 >> 172.21.15.79:0/2402243350 conn(0x7f4b0806d060 msgr2=0x7f4b0811d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.169+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 shutdown_connections 2024-04-22T06:47:52.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.169+0000 7f4b0eb10640 1 -- 172.21.15.79:0/2402243350 wait complete. 2024-04-22T06:47:52.188 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.2 2024-04-22T06:47:52.188 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:47:52.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:51 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/370746445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:52.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:51 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/370746445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:52.233 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:47:52.234 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:51 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/370746445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:52.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.531+0000 7f9dccf5b640 1 -- 172.21.15.79:0/473259137 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc0099ca0 msgr2=0x7f9dc009a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.530 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.531+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/473259137 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc0099ca0 0x7f9dc009a080 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009930 tx=0x7f9dbc02bc50 comp rx=0 tx=0).stop 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 -- 172.21.15.79:0/473259137 shutdown_connections 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/473259137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 0x7f9dc0095370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/473259137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dc00958b0 0x7f9dc00a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/473259137 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc0099ca0 0x7f9dc009a080 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 -- 172.21.15.79:0/473259137 >> 172.21.15.79:0/473259137 conn(0x7f9dc000a860 msgr2=0x7f9dc000ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 -- 172.21.15.79:0/473259137 shutdown_connections 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 -- 172.21.15.79:0/473259137 wait complete. 2024-04-22T06:47:52.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 Processor -- start 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.532+0000 7f9dccf5b640 1 -- start start 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dccf5b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 0x7f9dc01325a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dccf5b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 0x7f9dc0132ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dccf5b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dc0099ca0 0x7f9dc012c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dccf5b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc00a5030 con 0x7f9dc00958b0 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dccf5b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc00a4eb0 con 0x7f9dc0094ef0 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dccf5b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc00a51b0 con 0x7f9dc0099ca0 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dc6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 0x7f9dc0132ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dc6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 0x7f9dc0132ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47770/0 (socket says 172.21.15.79:47770) 2024-04-22T06:47:52.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.533+0000 7f9dc6ffd640 1 -- 172.21.15.79:0/2493220879 learned_addr learned my addr 172.21.15.79:0/2493220879 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc77fe640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 0x7f9dc01325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc6ffd640 1 -- 172.21.15.79:0/2493220879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dc0099ca0 msgr2=0x7f9dc012c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc6ffd640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dc0099ca0 0x7f9dc012c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc6ffd640 1 -- 172.21.15.79:0/2493220879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 msgr2=0x7f9dc01325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc6ffd640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 0x7f9dc01325a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc6ffd640 1 -- 172.21.15.79:0/2493220879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc012ce50 con 0x7f9dc00958b0 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.534+0000 7f9dc77fe640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 0x7f9dc01325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.535+0000 7f9dc6ffd640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 0x7f9dc0132ae0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9db400b480 tx=0x7f9db400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.535+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db4014070 con 0x7f9dc00958b0 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.535+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc012d140 con 0x7f9dc00958b0 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.535+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc009bd30 con 0x7f9dc00958b0 2024-04-22T06:47:52.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.536+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9db4010d20 con 0x7f9dc00958b0 2024-04-22T06:47:52.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.536+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db4017360 con 0x7f9dc00958b0 2024-04-22T06:47:52.537 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.538+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc00963d0 con 0x7f9dc00958b0 2024-04-22T06:47:52.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.540+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9db4017070 con 0x7f9dc00958b0 2024-04-22T06:47:52.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.541+0000 7f9dc4ff9640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9db80a3da0 0x7f9db80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.542+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f9db40d02e0 con 0x7f9dc00958b0 2024-04-22T06:47:52.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.542+0000 7f9dc77fe640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9db80a3da0 0x7f9db80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.543+0000 7f9dc77fe640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9db80a3da0 0x7f9db80a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009900 tx=0x7f9dbc02c8e0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.553+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9db409ca00 con 0x7f9dc00958b0 2024-04-22T06:47:52.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 -- 172.21.15.79:0/2115597972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 msgr2=0x7f6c3c10d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 --2- 172.21.15.79:0/2115597972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 0x7f6c3c10d7e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009900 tx=0x7f6c3402bd10 comp rx=0 tx=0).stop 2024-04-22T06:47:52.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 -- 172.21.15.79:0/2115597972 shutdown_connections 2024-04-22T06:47:52.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 --2- 172.21.15.79:0/2115597972 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c3c108e30 0x7f6c3c1152f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 --2- 172.21.15.79:0/2115597972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108490 0x7f6c3c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 --2- 172.21.15.79:0/2115597972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 0x7f6c3c10d7e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.620+0000 7f6c413b2640 1 -- 172.21.15.79:0/2115597972 >> 172.21.15.79:0/2115597972 conn(0x7f6c3c077d70 msgr2=0x7f6c3c106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.622+0000 7f6c413b2640 1 -- 172.21.15.79:0/2115597972 shutdown_connections 2024-04-22T06:47:52.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.622+0000 7f6c413b2640 1 -- 172.21.15.79:0/2115597972 wait complete. 2024-04-22T06:47:52.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.623+0000 7f6c413b2640 1 Processor -- start 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.624+0000 7f6c413b2640 1 -- start start 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.624+0000 7f6c413b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c3c108490 0x7f6c3c082620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.624+0000 7f6c413b2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 0x7f6c3c07b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.624+0000 7f6c413b2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 0x7f6c3c07bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.624+0000 7f6c413b2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c118840 con 0x7f6c3c108490 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.625+0000 7f6c413b2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1186c0 con 0x7f6c3c10d400 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.625+0000 7f6c413b2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1189c0 con 0x7f6c3c108e30 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.625+0000 7f6c3a7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 0x7f6c3c07b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.625+0000 7f6c3a7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 0x7f6c3c07b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33588/0 (socket says 172.21.15.79:33588) 2024-04-22T06:47:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.625+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/1199340442 learned_addr learned my addr 172.21.15.79:0/1199340442 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.626+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/1199340442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 msgr2=0x7f6c3c07bc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.626+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 0x7f6c3c07bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.626+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c3c108490 msgr2=0x7f6c3c082620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.626+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c3c108490 0x7f6c3c082620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.626+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/1199340442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c07c2a0 con 0x7f6c3c108e30 2024-04-22T06:47:52.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.628+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 0x7f6c3c07b6f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00ba50 tx=0x7f6c2c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.631+0000 7f6c33fff640 1 -- 172.21.15.79:0/1199340442 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c019070 con 0x7f6c3c108e30 2024-04-22T06:47:52.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.631+0000 7f6c33fff640 1 -- 172.21.15.79:0/1199340442 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c2c00cbe0 con 0x7f6c3c108e30 2024-04-22T06:47:52.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.632+0000 7f6c33fff640 1 -- 172.21.15.79:0/1199340442 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c0075f0 con 0x7f6c3c108e30 2024-04-22T06:47:52.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.633+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c080140 con 0x7f6c3c108e30 2024-04-22T06:47:52.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.633+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c080610 con 0x7f6c3c108e30 2024-04-22T06:47:52.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.637+0000 7f6c43fa4740 1 Processor -- start 2024-04-22T06:47:52.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.637+0000 7f6c43fa4740 1 -- start start 2024-04-22T06:47:52.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.638+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8a6d0 0x55bb88e8aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.638+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8b080 0x55bb88e93630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.638+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e96050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.639+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bb88def1f0 con 0x55bb88e8a6d0 2024-04-22T06:47:52.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.639+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb88d8c4b0 con 0x55bb88e93c00 2024-04-22T06:47:52.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.639+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb88deb1b0 con 0x55bb88e8b080 2024-04-22T06:47:52.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.640+0000 7f6c3a7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e96050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.640+0000 7f6c3a7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e96050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57930/0 (socket says 172.21.15.79:57930) 2024-04-22T06:47:52.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.640+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/3709456521 learned_addr learned my addr 172.21.15.79:0/3709456521 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.642+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8b080 0x55bb88e93630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.645+0000 7f6c3affd640 1 -- 172.21.15.79:0/3709456521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 msgr2=0x55bb88e96050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.645+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e96050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.645+0000 7f6c3affd640 1 -- 172.21.15.79:0/3709456521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8a6d0 msgr2=0x55bb88e8aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.645+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8a6d0 0x55bb88e8aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.645+0000 7f6c3affd640 1 -- 172.21.15.79:0/3709456521 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb88d7e1b0 con 0x55bb88e8b080 2024-04-22T06:47:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.645+0000 7f6c33fff640 1 -- 172.21.15.79:0/1199340442 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6c2c014440 con 0x7f6c3c108e30 2024-04-22T06:47:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.647+0000 7f6c33fff640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f6c280a3d80 0x7f6c280a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.647+0000 7f6c33fff640 1 -- 172.21.15.79:0/1199340442 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f6c2c0d5800 con 0x7f6c3c108e30 2024-04-22T06:47:52.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.653+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8b080 0x55bb88e93630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009230 tx=0x7f6c34009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d94feb99b6ad57ea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.653+0000 7f6c3b7fe640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f6c280a3d80 0x7f6c280a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.654+0000 7f6c317fa640 1 -- 172.21.15.79:0/3709456521 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34041020 con 0x55bb88e8b080 2024-04-22T06:47:52.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.655+0000 7f6c317fa640 1 -- 172.21.15.79:0/3709456521 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c34005500 con 0x55bb88e8b080 2024-04-22T06:47:52.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.675+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3709456521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8b080 msgr2=0x55bb88e93630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.675+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8b080 0x55bb88e93630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009230 tx=0x7f6c34009400 comp rx=0 tx=0).stop 2024-04-22T06:47:52.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.676+0000 7f6c3b7fe640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f6c280a3d80 0x7f6c280a6240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6c24001550 tx=0x7f6c24007b00 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.681+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3709456521 shutdown_connections 2024-04-22T06:47:52.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.681+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e96050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.681+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8b080 0x55bb88e93630 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.681+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3709456521 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8a6d0 0x55bb88e8aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.681+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3709456521 >> 172.21.15.79:0/3709456521 conn(0x55bb88e84da0 msgr2=0x55bb88e85180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.682+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3709456521 shutdown_connections 2024-04-22T06:47:52.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.682+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3709456521 wait complete. 2024-04-22T06:47:52.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.683+0000 7f6c43fa4740 1 -- start start 2024-04-22T06:47:52.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.684+0000 7f6c43fa4740 1 Processor -- start 2024-04-22T06:47:52.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.684+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 0x55bb88e93500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.684+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 0x55bb88e90890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.685+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e93c00 0x55bb88e90e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.685+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55bb88e96b60 con 0x55bb88e8b080 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.685+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55bb88da8500 con 0x55bb88e8a6d0 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.685+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb88e96ce0 con 0x55bb88e93c00 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.685+0000 7f6c43fa4740 10 client.0 _ll_register_callbacks cb 0x55bb88d76b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.686+0000 7f6c3affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 0x55bb88e93500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.686+0000 7f6c3affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 0x55bb88e93500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57946/0 (socket says 172.21.15.79:57946) 2024-04-22T06:47:52.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.686+0000 7f6c3affd640 1 -- 172.21.15.79:0/3437089321 learned_addr learned my addr 172.21.15.79:0/3437089321 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.687+0000 7f6c3affd640 1 -- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e93c00 msgr2=0x55bb88e90e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.687+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e93c00 0x55bb88e90e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.687+0000 7f6c3affd640 1 -- 172.21.15.79:0/3437089321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 msgr2=0x55bb88e90890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.687+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 0x55bb88e90890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.687+0000 7f6c3affd640 1 -- 172.21.15.79:0/3437089321 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb88b013f0 con 0x55bb88e8a6d0 2024-04-22T06:47:52.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.688+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 0x55bb88e93500 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c3403f000 tx=0x7f6c3402ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e6498f06357f9e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.689+0000 7f6c227fc640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:47:52.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.689+0000 7f6c227fc640 1 -- 172.21.15.79:0/3437089321 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3404a070 con 0x55bb88e8a6d0 2024-04-22T06:47:52.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.689+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bb88d8e930 con 0x55bb88e8a6d0 2024-04-22T06:47:52.688 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.690+0000 7f6c227fc640 1 -- 172.21.15.79:0/3437089321 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f6c34005460 con 0x55bb88e8a6d0 2024-04-22T06:47:52.688 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.690+0000 7f6c227fc640 1 -- 172.21.15.79:0/3437089321 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6c34056050 con 0x55bb88e8a6d0 2024-04-22T06:47:52.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.690+0000 7f6c43fa4740 10 client.25679 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:52.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.690+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55bb88bcb7d0 con 0x55bb88e8a6d0 2024-04-22T06:47:52.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.700+0000 7f6c227fc640 1 -- 172.21.15.79:0/3437089321 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bb88d8e930 con 0x55bb88e8a6d0 2024-04-22T06:47:52.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.701+0000 7f6c227fc640 1 -- 172.21.15.79:0/3437089321 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f6c34003820 con 0x55bb88e8a6d0 2024-04-22T06:47:52.701 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.703+0000 7f6c43fa4740 10 client.25679 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:52.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.703+0000 7f6c43fa4740 10 client.25679 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:47:52.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.703+0000 7f6c43fa4740 20 client.25679 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:52.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.703+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88ea3fc0 0x55bb88ea6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.703+0000 7f6c43fa4740 4 client.25679 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:52.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.703+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb88dfcc10 con 0x55bb88ea3fc0 2024-04-22T06:47:52.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.704+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88ea3fc0 0x55bb88ea6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.705+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88ea3fc0 0x55bb88ea6430 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4984337fe6bcc36 server_cookie=222a2ee8f6332f0c in_seq=0 out_seq=0 2024-04-22T06:47:52.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.708+0000 7f6c227fc640 10 client.25679 ms_handle_connect on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:52.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.710+0000 7f6c227fc640 1 -- 172.21.15.79:0/3437089321 <== mds.2 v2:172.21.15.193:6838/1588799940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb88dfcc10 con 0x55bb88ea3fc0 2024-04-22T06:47:52.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.710+0000 7f6c227fc640 10 client.25679 handle_command_reply: tid=0 2024-04-22T06:47:52.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.710+0000 7f6c43fa4740 1 client.25679 shutdown 2024-04-22T06:47:52.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 msgr2=0x55bb88e93500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 0x55bb88e93500 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c3403f000 tx=0x7f6c3402ca90 comp rx=0 tx=0).stop 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 shutdown_connections 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88ea3fc0 0x55bb88ea6430 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e93c00 0x55bb88e90e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 0x55bb88e90890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3437089321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8a6d0 0x55bb88e93500 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 >> 172.21.15.79:0/3437089321 conn(0x55bb88e84da0 msgr2=0x55bb88e86af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.716+0000 7f6c227fc640 0 client.25679 ms_handle_reset on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:52.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.720+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 shutdown_connections 2024-04-22T06:47:52.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.720+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3437089321 wait complete. 2024-04-22T06:47:52.722 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.723+0000 7f6c43fa4740 20 client.25679 trim_cache size 0 max 16384 2024-04-22T06:47:52.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.826+0000 7f6c43fa4740 1 Processor -- start 2024-04-22T06:47:52.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.826+0000 7f6c43fa4740 1 -- start start 2024-04-22T06:47:52.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.827+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e9faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.827+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 0x55bb88ea5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e92fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1bf2c0 con 0x55bb88e8b080 2024-04-22T06:47:52.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1bcb90 con 0x55bb88e93c00 2024-04-22T06:47:52.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1ba9e0 con 0x55bb88e8a6d0 2024-04-22T06:47:52.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c3affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e9faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c3affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e9faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33618/0 (socket says 172.21.15.79:33618) 2024-04-22T06:47:52.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.828+0000 7f6c3affd640 1 -- 172.21.15.79:0/1541094034 learned_addr learned my addr 172.21.15.79:0/1541094034 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.835+0000 7f6c3affd640 1 -- 172.21.15.79:0/1541094034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 msgr2=0x55bb88e92fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.835+0000 7f6c3affd640 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e92fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.835+0000 7f6c3affd640 1 -- 172.21.15.79:0/1541094034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 msgr2=0x55bb88ea5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.835+0000 7f6c3affd640 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 0x55bb88ea5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.835+0000 7f6c3affd640 1 -- 172.21.15.79:0/1541094034 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb88e964b0 con 0x55bb88e8a6d0 2024-04-22T06:47:52.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.836+0000 7f6c3affd640 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e9faa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009850 tx=0x7f6c34009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1b657730d77d34f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.841+0000 7f6c30ff9640 1 -- 172.21.15.79:0/1541094034 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34053020 con 0x55bb88e8a6d0 2024-04-22T06:47:52.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.842+0000 7f6c30ff9640 1 -- 172.21.15.79:0/1541094034 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c34003e10 con 0x55bb88e8a6d0 2024-04-22T06:47:52.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.842+0000 7f6c30ff9640 1 -- 172.21.15.79:0/1541094034 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34041040 con 0x55bb88e8a6d0 2024-04-22T06:47:52.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.844+0000 7f6c43fa4740 1 -- 172.21.15.79:0/1541094034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 msgr2=0x55bb88e9faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.845+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e9faa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009850 tx=0x7f6c34009400 comp rx=0 tx=0).stop 2024-04-22T06:47:52.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.850+0000 7f6c43fa4740 1 -- 172.21.15.79:0/1541094034 shutdown_connections 2024-04-22T06:47:52.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.850+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e93c00 0x55bb88e92fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.850+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e8b080 0x55bb88ea5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.850+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/1541094034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e9faa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.849 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.850+0000 7f6c43fa4740 1 -- 172.21.15.79:0/1541094034 >> 172.21.15.79:0/1541094034 conn(0x55bb88e84da0 msgr2=0x55bb88e90660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.852+0000 7f6c43fa4740 1 -- 172.21.15.79:0/1541094034 shutdown_connections 2024-04-22T06:47:52.850 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.852+0000 7f6c43fa4740 1 -- 172.21.15.79:0/1541094034 wait complete. 2024-04-22T06:47:52.851 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.853+0000 7f6c43fa4740 1 -- start start 2024-04-22T06:47:52.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.854+0000 7f6c43fa4740 1 Processor -- start 2024-04-22T06:47:52.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.855+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e96f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.855+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8b080 0x55bb88e974b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.855+0000 7f6c43fa4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e93c00 0x55bb88e979f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.856+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1bb050 con 0x55bb88e93c00 2024-04-22T06:47:52.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.856+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1bc4c0 con 0x55bb88e8b080 2024-04-22T06:47:52.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.856+0000 7f6c43fa4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1bb720 con 0x55bb88e8a6d0 2024-04-22T06:47:52.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.856+0000 7f6c43fa4740 10 client.0 _ll_register_callbacks cb 0x7f6c34004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.857+0000 7f6c3affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e96f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.857+0000 7f6c3affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e96f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33622/0 (socket says 172.21.15.79:33622) 2024-04-22T06:47:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.857+0000 7f6c3affd640 1 -- 172.21.15.79:0/3349605400 learned_addr learned my addr 172.21.15.79:0/3349605400 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.857+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8b080 0x55bb88e974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.858+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 msgr2=0x55bb88e96f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.858+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e96f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.858+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/3349605400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e93c00 msgr2=0x55bb88e979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:52.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.858+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e93c00 0x55bb88e979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.859+0000 7f6c3a7fc640 1 -- 172.21.15.79:0/3349605400 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb88f086f0 con 0x55bb88e8b080 2024-04-22T06:47:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.860+0000 7f6c3a7fc640 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8b080 0x55bb88e974b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0d2ee0 tx=0x7f6c2c09f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d8a3f3cc86af99 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:52.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.864+0000 7f6c227fc640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:47:52.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.864+0000 7f6c227fc640 1 -- 172.21.15.79:0/3349605400 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c0a1730 con 0x55bb88e8b080 2024-04-22T06:47:52.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.866+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c34003e10 con 0x55bb88e8b080 2024-04-22T06:47:52.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.866+0000 7f6c227fc640 1 -- 172.21.15.79:0/3349605400 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f6c2c09dd10 con 0x55bb88e8b080 2024-04-22T06:47:52.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.867+0000 7f6c227fc640 1 -- 172.21.15.79:0/3349605400 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6c2c0a5ae0 con 0x55bb88e8b080 2024-04-22T06:47:52.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.867+0000 7f6c43fa4740 10 client.25682 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:52.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.868+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55bb88f099f0 con 0x55bb88e8b080 2024-04-22T06:47:52.872 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.872+0000 7f6c227fc640 1 -- 172.21.15.79:0/3349605400 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c2c0bf030 con 0x55bb88e8b080 2024-04-22T06:47:52.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.872+0000 7f6c227fc640 1 -- 172.21.15.79:0/3349605400 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f6c2c01c020 con 0x55bb88e8b080 2024-04-22T06:47:52.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.873+0000 7f6c43fa4740 10 client.25682 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:52.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.873+0000 7f6c43fa4740 10 client.25682 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:47:52.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.873+0000 7f6c43fa4740 20 client.25682 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:52.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.873+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88f0a0d0 0x55bb88f0c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:52.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.873+0000 7f6c43fa4740 4 client.25682 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:47:52.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.873+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6c34003e10 con 0x55bb88f0a0d0 2024-04-22T06:47:52.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.879+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88f0a0d0 0x55bb88f0c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:52.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.890+0000 7f6c3affd640 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88f0a0d0 0x55bb88f0c4f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1223706f60fe408 server_cookie=1cf8e6dbbeab4 in_seq=0 out_seq=0 2024-04-22T06:47:52.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.891+0000 7f6c227fc640 10 client.25682 ms_handle_connect on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:52.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.893+0000 7f6c227fc640 1 -- 172.21.15.79:0/3349605400 <== mds.2 v2:172.21.15.193:6838/1588799940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6c34003e10 con 0x55bb88f0a0d0 2024-04-22T06:47:52.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.893+0000 7f6c227fc640 10 client.25682 handle_command_reply: tid=0 2024-04-22T06:47:52.891 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.893+0000 7f6c43fa4740 1 client.25682 shutdown 2024-04-22T06:47:52.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.895+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8b080 msgr2=0x55bb88e974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.893 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.895+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8b080 0x55bb88e974b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0d2ee0 tx=0x7f6c2c09f930 comp rx=0 tx=0).stop 2024-04-22T06:47:52.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 shutdown_connections 2024-04-22T06:47:52.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x55bb88f0a0d0 0x55bb88f0c4f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c227fc640 0 client.25682 ms_handle_reset on v2:172.21.15.193:6838/1588799940 2024-04-22T06:47:52.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bb88e8b080 0x55bb88e974b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55bb88e93c00 0x55bb88e979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c43fa4740 1 --2- 172.21.15.79:0/3349605400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb88e8a6d0 0x55bb88e96f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.899+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 >> 172.21.15.79:0/3349605400 conn(0x55bb88e84da0 msgr2=0x55bb88e8b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.900+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 shutdown_connections 2024-04-22T06:47:52.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.903+0000 7f6c43fa4740 1 -- 172.21.15.79:0/3349605400 wait complete. 2024-04-22T06:47:52.902 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.904+0000 7f6c43fa4740 20 client.25682 trim_cache size 0 max 16384 2024-04-22T06:47:52.906 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:47:52.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.912+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f6c280a3d80 msgr2=0x7f6c280a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.912+0000 7f6c413b2640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f6c280a3d80 0x7f6c280a6240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6c24001550 tx=0x7f6c24007b00 comp rx=0 tx=0).stop 2024-04-22T06:47:52.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.912+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 msgr2=0x7f6c3c07b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:52.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.912+0000 7f6c413b2640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 0x7f6c3c07b6f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00ba50 tx=0x7f6c2c00bf10 comp rx=0 tx=0).stop 2024-04-22T06:47:52.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.921+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 shutdown_connections 2024-04-22T06:47:52.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.921+0000 7f6c413b2640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f6c280a3d80 0x7f6c280a6240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.921+0000 7f6c413b2640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c3c10d400 0x7f6c3c07bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.921+0000 7f6c413b2640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c3c108e30 0x7f6c3c07b6f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.921+0000 7f6c413b2640 1 --2- 172.21.15.79:0/1199340442 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c3c108490 0x7f6c3c082620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:52.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.921+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 >> 172.21.15.79:0/1199340442 conn(0x7f6c3c077d70 msgr2=0x7f6c3c078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:52.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.925+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 shutdown_connections 2024-04-22T06:47:52.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:52.925+0000 7f6c413b2640 1 -- 172.21.15.79:0/1199340442 wait complete. 2024-04-22T06:47:52.983 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:47:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:52 smithi193 ceph-mon[31815]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 969 MiB/s wr, 1.87k op/s 2024-04-22T06:47:53.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:52 smithi190 ceph-mon[31421]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 969 MiB/s wr, 1.87k op/s 2024-04-22T06:47:53.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:52 smithi079 ceph-mon[27946]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 969 MiB/s wr, 1.87k op/s 2024-04-22T06:47:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 -- 172.21.15.79:0/3777998399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01105f0 msgr2=0x7f1af01109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 --2- 172.21.15.79:0/3777998399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01105f0 0x7f1af01109d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009860 tx=0x7f1ae002bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 -- 172.21.15.79:0/3777998399 shutdown_connections 2024-04-22T06:47:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 --2- 172.21.15.79:0/3777998399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1af01099d0 0x7f1af011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 --2- 172.21.15.79:0/3777998399 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 0x7f1af01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 --2- 172.21.15.79:0/3777998399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01105f0 0x7f1af01109d0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009860 tx=0x7f1ae002bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:53.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.421+0000 7f1af571c640 1 -- 172.21.15.79:0/3777998399 >> 172.21.15.79:0/3777998399 conn(0x7f1af006ced0 msgr2=0x7f1af006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.427 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.429+0000 7f1af571c640 1 -- 172.21.15.79:0/3777998399 shutdown_connections 2024-04-22T06:47:53.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.431+0000 7f1af571c640 1 -- 172.21.15.79:0/3777998399 wait complete. 2024-04-22T06:47:53.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.432+0000 7f1af571c640 1 Processor -- start 2024-04-22T06:47:53.432 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.432+0000 7f1af571c640 1 -- start start 2024-04-22T06:47:53.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.433+0000 7f1af571c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 0x7f1af01a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.433+0000 7f1af571c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01099d0 0x7f1af01a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.433+0000 7f1af571c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1af01abf40 0x7f1af01ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.433+0000 7f1af571c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0120ee0 con 0x7f1af0108f50 2024-04-22T06:47:53.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.433+0000 7f1af571c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0120d60 con 0x7f1af01abf40 2024-04-22T06:47:53.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.433+0000 7f1af571c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0121060 con 0x7f1af01099d0 2024-04-22T06:47:53.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.437+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 0x7f1af01a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.437+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 0x7f1af01a7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47832/0 (socket says 172.21.15.79:47832) 2024-04-22T06:47:53.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.437+0000 7f1aeffff640 1 -- 172.21.15.79:0/466124551 learned_addr learned my addr 172.21.15.79:0/466124551 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:53.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.438+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1af01abf40 0x7f1af01ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.444+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01099d0 0x7f1af01a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.445+0000 7f1aef7fe640 1 -- 172.21.15.79:0/466124551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1af01abf40 msgr2=0x7f1af01ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.445+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1af01abf40 0x7f1af01ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.445+0000 7f1aef7fe640 1 -- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 msgr2=0x7f1af01a7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.445+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 0x7f1af01a7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.445+0000 7f1aef7fe640 1 -- 172.21.15.79:0/466124551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af01aca30 con 0x7f1af01099d0 2024-04-22T06:47:53.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.447+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01099d0 0x7f1af01a7bc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4005fd0 tx=0x7f1ae400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:53.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.447+0000 7f1aed7fa640 1 -- 172.21.15.79:0/466124551 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae400cd00 con 0x7f1af01099d0 2024-04-22T06:47:53.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.448+0000 7f1aed7fa640 1 -- 172.21.15.79:0/466124551 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ae40035f0 con 0x7f1af01099d0 2024-04-22T06:47:53.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.448+0000 7f1aed7fa640 1 -- 172.21.15.79:0/466124551 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4010690 con 0x7f1af01099d0 2024-04-22T06:47:53.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.452+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af01acd20 con 0x7f1af01099d0 2024-04-22T06:47:53.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.452+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af01b4490 con 0x7f1af01099d0 2024-04-22T06:47:53.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.465+0000 7f1aed7fa640 1 -- 172.21.15.79:0/466124551 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1ae4003a80 con 0x7f1af01099d0 2024-04-22T06:47:53.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.467+0000 7f1aed7fa640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ad80a3e60 0x7f1ad80a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.468+0000 7f1aed7fa640 1 -- 172.21.15.79:0/466124551 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f1ae40d0b80 con 0x7f1af01099d0 2024-04-22T06:47:53.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.470+0000 7f1af9568740 1 Processor -- start 2024-04-22T06:47:53.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.471+0000 7f1aeffff640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ad80a3e60 0x7f1ad80a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.471+0000 7f1af9568740 1 -- start start 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.471+0000 7f1af9568740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e9147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.472+0000 7f1af9568740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 0x5574e9150630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.472+0000 7f1af9568740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9150c00 0x5574e9153050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.472+0000 7f1aeffff640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ad80a3e60 0x7f1ad80a6320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ae002c240 tx=0x7f1ae0005a90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.472+0000 7f1af9568740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5574e90ac1f0 con 0x5574e9148080 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.472+0000 7f1af9568740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5574e90494b0 con 0x5574e9150c00 2024-04-22T06:47:53.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.472+0000 7f1af9568740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5574e90a81b0 con 0x5574e91476d0 2024-04-22T06:47:53.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.473+0000 7f1aef7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 0x5574e9150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.473+0000 7f1aef7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 0x5574e9150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47834/0 (socket says 172.21.15.79:47834) 2024-04-22T06:47:53.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.473+0000 7f1aef7fe640 1 -- 172.21.15.79:0/3567349741 learned_addr learned my addr 172.21.15.79:0/3567349741 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.473+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9150c00 0x5574e9153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.473+0000 7f1aef7fe640 1 -- 172.21.15.79:0/3567349741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 msgr2=0x5574e9147ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.473+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e9147ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.474+0000 7f1aef7fe640 1 -- 172.21.15.79:0/3567349741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9150c00 msgr2=0x5574e9153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.474+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9150c00 0x5574e9153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.474+0000 7f1aef7fe640 1 -- 172.21.15.79:0/3567349741 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e903b1b0 con 0x5574e9148080 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.474+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9150c00 0x5574e9153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:47:53.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.474+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 0x5574e9150630 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4099370 tx=0x7f1ae409c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac1fda898e0c45fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:53.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.475+0000 7f1ad67fc640 1 -- 172.21.15.79:0/3567349741 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae40a0d50 con 0x5574e9148080 2024-04-22T06:47:53.479 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.475+0000 7f1ad67fc640 1 -- 172.21.15.79:0/3567349741 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ae40a5610 con 0x5574e9148080 2024-04-22T06:47:53.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.475+0000 7f1ad67fc640 1 -- 172.21.15.79:0/3567349741 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae40870c0 con 0x5574e9148080 2024-04-22T06:47:53.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.477+0000 7f1af9568740 1 -- 172.21.15.79:0/3567349741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 msgr2=0x5574e9150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.477+0000 7f1af9568740 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 0x5574e9150630 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4099370 tx=0x7f1ae409c400 comp rx=0 tx=0).stop 2024-04-22T06:47:53.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 -- 172.21.15.79:0/3567349741 shutdown_connections 2024-04-22T06:47:53.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9150c00 0x5574e9153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9148080 0x5574e9150630 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 --2- 172.21.15.79:0/3567349741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e9147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 -- 172.21.15.79:0/3567349741 >> 172.21.15.79:0/3567349741 conn(0x5574e9141da0 msgr2=0x5574e9142180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 -- 172.21.15.79:0/3567349741 shutdown_connections 2024-04-22T06:47:53.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.484+0000 7f1af9568740 1 -- 172.21.15.79:0/3567349741 wait complete. 2024-04-22T06:47:53.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.485+0000 7f1af9568740 1 -- start start 2024-04-22T06:47:53.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.485+0000 7f1af9568740 1 Processor -- start 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e91476d0 0x5574e9150620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9148080 0x5574e914d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e914e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5574e9153b70 con 0x5574e9150c00 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5574e9065500 con 0x5574e91476d0 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5574e9153cf0 con 0x5574e9148080 2024-04-22T06:47:53.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.486+0000 7f1af9568740 10 client.0 _ll_register_callbacks cb 0x5574e9033b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:53.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.487+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e91476d0 0x5574e9150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.487+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e91476d0 0x5574e9150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57998/0 (socket says 172.21.15.79:57998) 2024-04-22T06:47:53.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.487+0000 7f1aeffff640 1 -- 172.21.15.79:0/732266546 learned_addr learned my addr 172.21.15.79:0/732266546 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:53.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.487+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e914e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.487+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9148080 0x5574e914d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.488+0000 7f1aef7fe640 1 -- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e91476d0 msgr2=0x5574e9150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.488+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e91476d0 0x5574e9150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.488+0000 7f1aef7fe640 1 -- 172.21.15.79:0/732266546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 msgr2=0x5574e914e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.488+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e914e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.488+0000 7f1aef7fe640 1 -- 172.21.15.79:0/732266546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e8dbe3f0 con 0x5574e9148080 2024-04-22T06:47:53.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.489+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9148080 0x5574e914d9b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x5574e9148480 tx=0x7f1ae409cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b6b7289c8b542a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:53.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.499+0000 7f1acf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:47:53.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.499+0000 7f1acf7fe640 1 -- 172.21.15.79:0/732266546 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae40a0d50 con 0x5574e9148080 2024-04-22T06:47:53.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.499+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae40a5610 con 0x5574e9148080 2024-04-22T06:47:53.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.499+0000 7f1acf7fe640 1 -- 172.21.15.79:0/732266546 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f1ae40ce440 con 0x5574e9148080 2024-04-22T06:47:53.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.501+0000 7f1acf7fe640 1 -- 172.21.15.79:0/732266546 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1ae40bbb60 con 0x5574e9148080 2024-04-22T06:47:53.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.505+0000 7f1af9568740 10 client.25693 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:53.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.505+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5574e8e887d0 con 0x5574e9148080 2024-04-22T06:47:53.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.507+0000 7f1acf7fe640 1 -- 172.21.15.79:0/732266546 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ae40a5610 con 0x5574e9148080 2024-04-22T06:47:53.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.507+0000 7f1acf7fe640 1 -- 172.21.15.79:0/732266546 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f1ae40986f0 con 0x5574e9148080 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.514+0000 7f1af9568740 10 client.25693 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.514+0000 7f1af9568740 10 client.25693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.514+0000 7f1af9568740 20 client.25693 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.514+0000 7f1af9568740 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e9171db0 0x5574e91741a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.514+0000 7f1af9568740 4 client.25693 mds_command: new command op to 25265 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.514+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 --> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574e904b930 con 0x5574e9171db0 2024-04-22T06:47:53.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.515+0000 7f1aeffff640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e9171db0 0x5574e91741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.521+0000 7f1aeffff640 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e9171db0 0x5574e91741a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cbd45a9195be0420 server_cookie=9fe11ecc69d7fc27 in_seq=0 out_seq=0 2024-04-22T06:47:53.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.525+0000 7f1acf7fe640 10 client.25693 ms_handle_connect on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:53.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.525+0000 7f1acf7fe640 1 -- 172.21.15.79:0/732266546 <== mds.1 v2:172.21.15.190:6836/1635214680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574e904b930 con 0x5574e9171db0 2024-04-22T06:47:53.524 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.525+0000 7f1acf7fe640 10 client.25693 handle_command_reply: tid=0 2024-04-22T06:47:53.527 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.528+0000 7f1af9568740 1 client.25693 shutdown 2024-04-22T06:47:53.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9148080 msgr2=0x5574e914d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9148080 0x5574e914d9b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x5574e9148480 tx=0x7f1ae409cad0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 shutdown_connections 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e9171db0 0x5574e91741a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e914e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9148080 0x5574e914d9b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 --2- 172.21.15.79:0/732266546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e91476d0 0x5574e9150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 >> 172.21.15.79:0/732266546 conn(0x5574e9141da0 msgr2=0x5574e9143af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1acf7fe640 0 client.25693 ms_handle_reset on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 shutdown_connections 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.532+0000 7f1af9568740 1 -- 172.21.15.79:0/732266546 wait complete. 2024-04-22T06:47:53.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.533+0000 7f1af9568740 20 client.25693 trim_cache size 0 max 16384 2024-04-22T06:47:53.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.541+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dc0005a00 con 0x7f9dc00958b0 2024-04-22T06:47:53.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:47:53.544 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:46:45.384403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25265,"mds_2":25261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":35,"state":"up:active","state_seq":22,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25261,24467,25265]},"id":1}]} 2024-04-22T06:47:53.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.543+0000 7f9dc4ff9640 1 -- 172.21.15.79:0/2493220879 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7f9db409c300 con 0x7f9dc00958b0 2024-04-22T06:47:53.545 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 40 2024-04-22T06:47:53.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.551+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9db80a3da0 msgr2=0x7f9db80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.551+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9db80a3da0 0x7f9db80a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009900 tx=0x7f9dbc02c8e0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.551+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 msgr2=0x7f9dc0132ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.551+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 0x7f9dc0132ae0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9db400b480 tx=0x7f9db400b940 comp rx=0 tx=0).stop 2024-04-22T06:47:53.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.552+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 shutdown_connections 2024-04-22T06:47:53.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.552+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dc0099ca0 0x7f9dc012c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.552+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc00958b0 0x7f9dc0132ae0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.552+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9db80a3da0 0x7f9db80a6260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.552+0000 7f9dccf5b640 1 --2- 172.21.15.79:0/2493220879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dc0094ef0 0x7f9dc01325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.552+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 >> 172.21.15.79:0/2493220879 conn(0x7f9dc000a860 msgr2=0x7f9dc00988d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.556+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 shutdown_connections 2024-04-22T06:47:53.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.558+0000 7f9dccf5b640 1 -- 172.21.15.79:0/2493220879 wait complete. 2024-04-22T06:47:53.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.585+0000 7f1af9568740 1 Processor -- start 2024-04-22T06:47:53.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.585+0000 7f1af9568740 1 -- start start 2024-04-22T06:47:53.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.586+0000 7f1af9568740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e91500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.586+0000 7f1af9568740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 0x5574e9150620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.586+0000 7f1af9568740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e9176de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.586+0000 7f1af9568740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1af01b72a0 con 0x5574e9150c00 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.586+0000 7f1af9568740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1af01b4b70 con 0x5574e9148080 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.586+0000 7f1af9568740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0117930 con 0x5574e91476d0 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.587+0000 7f1af4f1b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e9176de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.587+0000 7f1af4f1b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e9176de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47860/0 (socket says 172.21.15.79:47860) 2024-04-22T06:47:53.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.587+0000 7f1af4f1b640 1 -- 172.21.15.79:0/287197735 learned_addr learned my addr 172.21.15.79:0/287197735 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:53.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.588+0000 7f1aeffff640 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e91500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.588+0000 7f1af4f1b640 1 -- 172.21.15.79:0/287197735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 msgr2=0x5574e91500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:53.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.588+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e91500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.588+0000 7f1af4f1b640 1 -- 172.21.15.79:0/287197735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 msgr2=0x5574e9150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:53.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.588+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 0x5574e9150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.589+0000 7f1af4f1b640 1 -- 172.21.15.79:0/287197735 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e91426e0 con 0x5574e9150c00 2024-04-22T06:47:53.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.590+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e9176de0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800ade0 tx=0x7f1ae8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c7433e4302c38f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:53.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.591+0000 7f1ad5ffb640 1 -- 172.21.15.79:0/287197735 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae801e020 con 0x5574e9150c00 2024-04-22T06:47:53.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.591+0000 7f1ad5ffb640 1 -- 172.21.15.79:0/287197735 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ae8003e20 con 0x5574e9150c00 2024-04-22T06:47:53.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.591+0000 7f1ad5ffb640 1 -- 172.21.15.79:0/287197735 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae800f890 con 0x5574e9150c00 2024-04-22T06:47:53.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.592+0000 7f1af9568740 1 -- 172.21.15.79:0/287197735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 msgr2=0x5574e9176de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.592+0000 7f1af9568740 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e9176de0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800ade0 tx=0x7f1ae8011810 comp rx=0 tx=0).stop 2024-04-22T06:47:53.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.592+0000 7f1af9568740 1 -- 172.21.15.79:0/287197735 shutdown_connections 2024-04-22T06:47:53.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.592+0000 7f1af9568740 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 0x5574e9150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.592+0000 7f1af9568740 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e9150c00 0x5574e9176de0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.593+0000 7f1af9568740 1 --2- 172.21.15.79:0/287197735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e91476d0 0x5574e91500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.593+0000 7f1af9568740 1 -- 172.21.15.79:0/287197735 >> 172.21.15.79:0/287197735 conn(0x5574e9141da0 msgr2=0x5574e9140e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.593+0000 7f1af9568740 1 -- 172.21.15.79:0/287197735 shutdown_connections 2024-04-22T06:47:53.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.593+0000 7f1af9568740 1 -- 172.21.15.79:0/287197735 wait complete. 2024-04-22T06:47:53.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.594+0000 7f1af9568740 1 -- start start 2024-04-22T06:47:53.594 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 86.2 --await' 2024-04-22T06:47:53.594 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 86.2 --await 2024-04-22T06:47:53.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.595+0000 7f1af9568740 1 Processor -- start 2024-04-22T06:47:53.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.596+0000 7f1af9568740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 0x5574e9177240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.597+0000 7f1af9568740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 0x5574e91745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.597+0000 7f1af9568740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9150c00 0x5574e9174b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.597+0000 7f1af9568740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0117fa0 con 0x5574e91476d0 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.597+0000 7f1af9568740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0119410 con 0x5574e9148080 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.598+0000 7f1af9568740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0118670 con 0x5574e9150c00 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.598+0000 7f1af9568740 10 client.0 _ll_register_callbacks cb 0x7f1ae400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.598+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 0x5574e9177240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.598+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 0x5574e9177240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47862/0 (socket says 172.21.15.79:47862) 2024-04-22T06:47:53.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.598+0000 7f1aeffff640 1 -- 172.21.15.79:0/1191266814 learned_addr learned my addr 172.21.15.79:0/1191266814 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.599+0000 7f1af4f1b640 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9150c00 0x5574e9174b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.600+0000 7f1aeffff640 1 -- 172.21.15.79:0/1191266814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9150c00 msgr2=0x5574e9174b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.600+0000 7f1aeffff640 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9150c00 0x5574e9174b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.600+0000 7f1aeffff640 1 -- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 msgr2=0x5574e91745d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.600+0000 7f1aeffff640 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 0x5574e91745d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.600+0000 7f1aeffff640 1 -- 172.21.15.79:0/1191266814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e9175380 con 0x5574e91476d0 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.601+0000 7f1aeffff640 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 0x5574e9177240 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009830 tx=0x7f1ae002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81796b838f1faedd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.604+0000 7f1acf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.604+0000 7f1acf7fe640 1 -- 172.21.15.79:0/1191266814 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae005a070 con 0x5574e91476d0 2024-04-22T06:47:53.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.604+0000 7f1acf7fe640 1 -- 172.21.15.79:0/1191266814 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f1ae0034b10 con 0x5574e91476d0 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.605+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae8003e20 con 0x5574e91476d0 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.605+0000 7f1acf7fe640 1 -- 172.21.15.79:0/1191266814 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1ae0063050 con 0x5574e91476d0 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.605+0000 7f1af9568740 10 client.16644 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.605+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5574e91d8090 con 0x5574e91476d0 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.606+0000 7f1acf7fe640 1 -- 172.21.15.79:0/1191266814 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ae0044040 con 0x5574e91476d0 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1acf7fe640 1 -- 172.21.15.79:0/1191266814 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f1ae0003a00 con 0x5574e91476d0 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1af9568740 10 client.16644 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1af9568740 10 client.16644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1af9568740 20 client.16644 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1af9568740 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e91d8790 0x5574e91dabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1af9568740 4 client.16644 mds_command: new command op to 25265 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:47:53.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.607+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 --> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f1ae8003e20 con 0x5574e91d8790 2024-04-22T06:47:53.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.609+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e91d8790 0x5574e91dabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:53.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.612+0000 7f1aef7fe640 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e91d8790 0x5574e91dabb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc595c52ac16fdfa server_cookie=860521285d756293 in_seq=0 out_seq=0 2024-04-22T06:47:53.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.612+0000 7f1acf7fe640 10 client.16644 ms_handle_connect on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:53.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.613+0000 7f1acf7fe640 1 -- 172.21.15.79:0/1191266814 <== mds.1 v2:172.21.15.190:6836/1635214680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f1ae8003e20 con 0x5574e91d8790 2024-04-22T06:47:53.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.613+0000 7f1acf7fe640 10 client.16644 handle_command_reply: tid=0 2024-04-22T06:47:53.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.613+0000 7f1af9568740 1 client.16644 shutdown 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.613+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 msgr2=0x5574e9177240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.613+0000 7f1af9568740 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 0x5574e9177240 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009830 tx=0x7f1ae002e050 comp rx=0 tx=0).stop 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 shutdown_connections 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1af9568740 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:6836/1635214680,v1:172.21.15.190:6837/1635214680] conn(0x5574e91d8790 0x5574e91dabb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1af9568740 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5574e9148080 0x5574e91745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1af9568740 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574e9150c00 0x5574e9174b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1af9568740 1 --2- 172.21.15.79:0/1191266814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5574e91476d0 0x5574e9177240 secure :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009830 tx=0x7f1ae002e050 comp rx=0 tx=0).stop 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 >> 172.21.15.79:0/1191266814 conn(0x5574e9141da0 msgr2=0x5574e915bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.614+0000 7f1acf7fe640 0 client.16644 ms_handle_reset on v2:172.21.15.190:6836/1635214680 2024-04-22T06:47:53.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.623+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 shutdown_connections 2024-04-22T06:47:53.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.623+0000 7f1af9568740 1 -- 172.21.15.79:0/1191266814 wait complete. 2024-04-22T06:47:53.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.624+0000 7f1af9568740 20 client.16644 trim_cache size 0 max 16384 2024-04-22T06:47:53.622 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ad80a3e60 msgr2=0x7f1ad80a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ad80a3e60 0x7f1ad80a6320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ae002c240 tx=0x7f1ae0005a90 comp rx=0 tx=0).stop 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01099d0 msgr2=0x7f1af01a7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01099d0 0x7f1af01a7bc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4005fd0 tx=0x7f1ae400ee50 comp rx=0 tx=0).stop 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 shutdown_connections 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1ad80a3e60 0x7f1ad80a6320 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1af01abf40 0x7f1af01ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af01099d0 0x7f1af01a7bc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 --2- 172.21.15.79:0/466124551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1af0108f50 0x7f1af01a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 >> 172.21.15.79:0/466124551 conn(0x7f1af006ced0 msgr2=0x7f1af0070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:53.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 shutdown_connections 2024-04-22T06:47:53.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:53.629+0000 7f1af571c640 1 -- 172.21.15.79:0/466124551 wait complete. 2024-04-22T06:47:54.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.159+0000 7f1a58b2c640 1 -- 172.21.15.79:0/312806542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 msgr2=0x7f1a5411d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.159+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/312806542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 0x7f1a5411d740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a4801a910 tx=0x7f1a4803cca0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.163+0000 7f1a58b2c640 1 -- 172.21.15.79:0/312806542 shutdown_connections 2024-04-22T06:47:54.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.163+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/312806542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 0x7f1a5411d740 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.163+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/312806542 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a5410e030 0x7f1a5410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.163+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/312806542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.163+0000 7f1a58b2c640 1 -- 172.21.15.79:0/312806542 >> 172.21.15.79:0/312806542 conn(0x7f1a5406ce60 msgr2=0x7f1a5406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.168+0000 7f1a58b2c640 1 -- 172.21.15.79:0/312806542 shutdown_connections 2024-04-22T06:47:54.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.168+0000 7f1a58b2c640 1 -- 172.21.15.79:0/312806542 wait complete. 2024-04-22T06:47:54.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.169+0000 7f1a58b2c640 1 Processor -- start 2024-04-22T06:47:54.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 -- start start 2024-04-22T06:47:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a5410e030 0x7f1a5411d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 0x7f1a54119460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541199a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1a541137f0 con 0x7f1a5410e030 2024-04-22T06:47:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54113670 con 0x7f1a5410eb50 2024-04-22T06:47:54.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.173+0000 7f1a58b2c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54113970 con 0x7f1a541104d0 2024-04-22T06:47:54.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.175+0000 7f1a53fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.175+0000 7f1a53fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541199a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33700/0 (socket says 172.21.15.79:33700) 2024-04-22T06:47:54.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.175+0000 7f1a53fff640 1 -- 172.21.15.79:0/643857690 learned_addr learned my addr 172.21.15.79:0/643857690 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.179 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.177+0000 7f1a53fff640 1 -- 172.21.15.79:0/643857690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 msgr2=0x7f1a54119460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:54.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.177+0000 7f1a53fff640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 0x7f1a54119460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.177+0000 7f1a53fff640 1 -- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a5410e030 msgr2=0x7f1a5411d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.177+0000 7f1a53fff640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a5410e030 0x7f1a5411d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.177+0000 7f1a53fff640 1 -- 172.21.15.79:0/643857690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5411a0d0 con 0x7f1a541104d0 2024-04-22T06:47:54.180 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.182+0000 7f1a53fff640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541199a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a4801a860 tx=0x7f1a48004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.185+0000 7f1a50ff9640 1 -- 172.21.15.79:0/643857690 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48052020 con 0x7f1a541104d0 2024-04-22T06:47:54.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.185+0000 7f1a50ff9640 1 -- 172.21.15.79:0/643857690 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4803dd00 con 0x7f1a541104d0 2024-04-22T06:47:54.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.185+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a541ab350 con 0x7f1a541104d0 2024-04-22T06:47:54.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.185+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a541ab750 con 0x7f1a541104d0 2024-04-22T06:47:54.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.185+0000 7f1a50ff9640 1 -- 172.21.15.79:0/643857690 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48003c90 con 0x7f1a541104d0 2024-04-22T06:47:54.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:53 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2493220879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:54.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.192+0000 7f1a50ff9640 1 -- 172.21.15.79:0/643857690 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a48055020 con 0x7f1a541104d0 2024-04-22T06:47:54.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.193+0000 7f1a5c977740 1 Processor -- start 2024-04-22T06:47:54.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.193+0000 7f1a5c977740 1 -- start start 2024-04-22T06:47:54.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.193+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 0x560553f0bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.193+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f14b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.193+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f17480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.193+0000 7f1a5c977740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560553e6c0b0 con 0x560553f15090 2024-04-22T06:47:54.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.194+0000 7f1a5c977740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560553e1d680 con 0x560553f0c4b0 2024-04-22T06:47:54.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.194+0000 7f1a5c977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560553e29c60 con 0x560553f0bb90 2024-04-22T06:47:54.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.194+0000 7f1a50ff9640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a3c0a3f20 0x7f1a3c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.195+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 0x560553f0bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.195+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 0x560553f0bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33710/0 (socket says 172.21.15.79:33710) 2024-04-22T06:47:54.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.195+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1321401269 learned_addr learned my addr 172.21.15.79:0/1321401269 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.195+0000 7f1a50ff9640 1 -- 172.21.15.79:0/643857690 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f1a481046c0 con 0x7f1a541104d0 2024-04-22T06:47:54.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.196+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1321401269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 msgr2=0x560553f14b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:54.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.196+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f14b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.196+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1321401269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 msgr2=0x560553f17480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.196+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f17480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.196+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1321401269 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560553c4ec90 con 0x560553f0bb90 2024-04-22T06:47:54.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.198+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 0x560553f0bf70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00bc40 tx=0x7f1a4c00d760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f206452f2bcc3e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.199+0000 7f1a39ffb640 1 -- 172.21.15.79:0/1321401269 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c004dc0 con 0x560553f0bb90 2024-04-22T06:47:54.199 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.201+0000 7f1a53fff640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a3c0a3f20 0x7f1a3c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.200 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.201+0000 7f1a39ffb640 1 -- 172.21.15.79:0/1321401269 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4c012070 con 0x560553f0bb90 2024-04-22T06:47:54.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.203+0000 7f1a5c977740 1 -- 172.21.15.79:0/1321401269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 msgr2=0x560553f0bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.203+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 0x560553f0bf70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00bc40 tx=0x7f1a4c00d760 comp rx=0 tx=0).stop 2024-04-22T06:47:54.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:53 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2493220879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:54.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.205+0000 7f1a5c977740 1 -- 172.21.15.79:0/1321401269 shutdown_connections 2024-04-22T06:47:54.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.205+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f14b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.205+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f17480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.205+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1321401269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0bb90 0x560553f0bf70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.206+0000 7f1a5c977740 1 -- 172.21.15.79:0/1321401269 >> 172.21.15.79:0/1321401269 conn(0x560553f062c0 msgr2=0x560553f066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.210+0000 7f1a53fff640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a3c0a3f20 0x7f1a3c0a63e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1a480cc9c0 tx=0x7f1a480ce600 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.221+0000 7f1a5c977740 1 -- 172.21.15.79:0/1321401269 shutdown_connections 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.223+0000 7f1a5c977740 1 -- 172.21.15.79:0/1321401269 wait complete. 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.224+0000 7f1a5c977740 1 -- start start 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.224+0000 7f1a5c977740 1 Processor -- start 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f0bb90 0x560553f12080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f125c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f15090 0x560553f12b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560553e6f400 con 0x560553f0bb90 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560553e2b840 con 0x560553f0c4b0 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560553e74720 con 0x560553f15090 2024-04-22T06:47:54.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.225+0000 7f1a5c977740 10 client.0 _ll_register_callbacks cb 0x560553e7cea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:54.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f125c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58052/0 (socket says 172.21.15.79:58052) 2024-04-22T06:47:54.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 -- 172.21.15.79:0/1865506214 learned_addr learned my addr 172.21.15.79:0/1865506214 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 -- 172.21.15.79:0/1865506214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f15090 msgr2=0x560553f12b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f15090 0x560553f12b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f15090 0x560553f12b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 -- 172.21.15.79:0/1865506214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f0bb90 msgr2=0x560553f12080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f0bb90 0x560553f12080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.226+0000 7f1a537fe640 1 -- 172.21.15.79:0/1865506214 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560553caf420 con 0x560553f0c4b0 2024-04-22T06:47:54.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.227+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f15090 0x560553f12b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:54.226 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.227+0000 7f1a537fe640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f125c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009e40 tx=0x7f1a4400b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e5cbfdadcfe3fc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.239+0000 7f1a377f6640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:47:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.240+0000 7f1a377f6640 1 -- 172.21.15.79:0/1865506214 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44015020 con 0x560553f0c4b0 2024-04-22T06:47:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.240+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a4c012070 con 0x560553f0c4b0 2024-04-22T06:47:54.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.240+0000 7f1a377f6640 1 -- 172.21.15.79:0/1865506214 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f1a44011620 con 0x560553f0c4b0 2024-04-22T06:47:54.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.240+0000 7f1a377f6640 1 -- 172.21.15.79:0/1865506214 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1a44020050 con 0x560553f0c4b0 2024-04-22T06:47:54.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.241+0000 7f1a5c977740 10 client.25697 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:54.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.241+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560553b42580 con 0x560553f0c4b0 2024-04-22T06:47:54.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.241+0000 7f1a377f6640 1 -- 172.21.15.79:0/1865506214 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4c012070 con 0x560553f0c4b0 2024-04-22T06:47:54.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.241+0000 7f1a377f6640 1 -- 172.21.15.79:0/1865506214 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f1a440198e0 con 0x560553f0c4b0 2024-04-22T06:47:54.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:53 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2493220879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:47:54.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a5c977740 10 client.25697 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a5c977740 10 client.25697 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a5c977740 20 client.25697 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f25390 0x560553f27850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a5c977740 4 client.25697 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560553e7db10 con 0x560553f25390 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.242+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f25390 0x560553f27850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.243+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f25390 0x560553f27850 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c589f592b294b83 server_cookie=6d6a1c92ed4d33f8 in_seq=0 out_seq=0 2024-04-22T06:47:54.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.243+0000 7f1a377f6640 10 client.25697 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:54.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.245+0000 7f1a377f6640 1 -- 172.21.15.79:0/1865506214 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560553e7db10 con 0x560553f25390 2024-04-22T06:47:54.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.246+0000 7f1a377f6640 10 client.25697 handle_command_reply: tid=0 2024-04-22T06:47:54.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.246+0000 7f1a5c977740 1 client.25697 shutdown 2024-04-22T06:47:54.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 msgr2=0x560553f125c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f125c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009e40 tx=0x7f1a4400b4b0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 shutdown_connections 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f25390 0x560553f27850 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0c4b0 0x560553f125c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f15090 0x560553f12b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1865506214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f0bb90 0x560553f12080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 >> 172.21.15.79:0/1865506214 conn(0x560553f062c0 msgr2=0x560553f08010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.249+0000 7f1a377f6640 0 client.25697 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.250+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 shutdown_connections 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.250+0000 7f1a5c977740 1 -- 172.21.15.79:0/1865506214 wait complete. 2024-04-22T06:47:54.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.250+0000 7f1a5c977740 20 client.25697 trim_cache size 0 max 16384 2024-04-22T06:47:54.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.307+0000 7f1a5c977740 1 Processor -- start 2024-04-22T06:47:54.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.307+0000 7f1a5c977740 1 -- start start 2024-04-22T06:47:54.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0bb90 0x560553f20bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f2d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f2d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f2d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f2d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33724/0 (socket says 172.21.15.79:33724) 2024-04-22T06:47:54.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a537fe640 1 -- 172.21.15.79:0/1282196679 learned_addr learned my addr 172.21.15.79:0/1282196679 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4000ec50 con 0x560553f15090 2024-04-22T06:47:54.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4000caa0 con 0x560553f0bb90 2024-04-22T06:47:54.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.308+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4000e580 con 0x560553f0c4b0 2024-04-22T06:47:54.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0bb90 0x560553f20bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f2d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1282196679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 msgr2=0x560553f2d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f2d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1282196679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 msgr2=0x560553f2d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f2d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.309+0000 7f1a52ffd640 1 -- 172.21.15.79:0/1282196679 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560553f0c2e0 con 0x560553f0bb90 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.310+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0bb90 0x560553f20bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x560553f280d0 tx=0x7f1a4c005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac0e519b1015445 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.310+0000 7f1a397fa640 1 -- 172.21.15.79:0/1282196679 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c003c00 con 0x560553f0bb90 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.310+0000 7f1a397fa640 1 -- 172.21.15.79:0/1282196679 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4c012030 con 0x560553f0bb90 2024-04-22T06:47:54.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.310+0000 7f1a397fa640 1 -- 172.21.15.79:0/1282196679 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c01f490 con 0x560553f0bb90 2024-04-22T06:47:54.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0bb90 msgr2=0x560553f20bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0bb90 0x560553f20bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x560553f280d0 tx=0x7f1a4c005040 comp rx=0 tx=0).stop 2024-04-22T06:47:54.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 shutdown_connections 2024-04-22T06:47:54.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f15090 0x560553f2d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f2d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 --2- 172.21.15.79:0/1282196679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f0bb90 0x560553f20bc0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x560553f280d0 tx=0x7f1a4c005040 comp rx=0 tx=0).stop 2024-04-22T06:47:54.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.312+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 >> 172.21.15.79:0/1282196679 conn(0x560553f062c0 msgr2=0x560553f04b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.317+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 shutdown_connections 2024-04-22T06:47:54.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.318+0000 7f1a5c977740 1 -- 172.21.15.79:0/1282196679 wait complete. 2024-04-22T06:47:54.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.318+0000 7f1a5c977740 1 -- start start 2024-04-22T06:47:54.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.318+0000 7f1a5c977740 1 Processor -- start 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f1d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 0x560553f194c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f19a00 0x560553f19e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4000d7e0 con 0x560553f19a00 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4000d110 con 0x560553f15090 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4000deb0 con 0x560553f0c4b0 2024-04-22T06:47:54.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.319+0000 7f1a5c977740 10 client.0 _ll_register_callbacks cb 0x7f1a44006ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.320+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 0x560553f194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.320+0000 7f1a52ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f19a00 0x560553f19e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.320+0000 7f1a537fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 0x560553f194c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58058/0 (socket says 172.21.15.79:58058) 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.320+0000 7f1a537fe640 1 -- 172.21.15.79:0/66886839 learned_addr learned my addr 172.21.15.79:0/66886839 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a537fe640 1 -- 172.21.15.79:0/66886839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 msgr2=0x560553f1d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a537fe640 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f1d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a537fe640 1 -- 172.21.15.79:0/66886839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f19a00 msgr2=0x560553f19e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a537fe640 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f19a00 0x560553f19e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a537fe640 1 -- 172.21.15.79:0/66886839 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560553f0ce60 con 0x560553f15090 2024-04-22T06:47:54.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f19a00 0x560553f19e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.321+0000 7f1a537fe640 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 0x560553f194c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009e40 tx=0x7f1a44006b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb85f3266042446 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.322+0000 7f1a377f6640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:47:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.322+0000 7f1a377f6640 1 -- 172.21.15.79:0/66886839 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44015040 con 0x560553f15090 2024-04-22T06:47:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.322+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a4c012030 con 0x560553f15090 2024-04-22T06:47:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.322+0000 7f1a377f6640 1 -- 172.21.15.79:0/66886839 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f1a440197b0 con 0x560553f15090 2024-04-22T06:47:54.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.323+0000 7f1a377f6640 1 -- 172.21.15.79:0/66886839 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1a44033050 con 0x560553f15090 2024-04-22T06:47:54.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.325+0000 7f1a5c977740 10 client.25703 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:54.323 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.325+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560553f8c940 con 0x560553f15090 2024-04-22T06:47:54.325 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a377f6640 1 -- 172.21.15.79:0/66886839 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a4c012030 con 0x560553f15090 2024-04-22T06:47:54.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a377f6640 1 -- 172.21.15.79:0/66886839 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f1a44011420 con 0x560553f15090 2024-04-22T06:47:54.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a5c977740 10 client.25703 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:54.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a5c977740 10 client.25703 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:47:54.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a5c977740 20 client.25703 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:54.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a5c977740 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f8d100 0x560553f8f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a5c977740 4 client.25703 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.2, "await": true}] 2024-04-22T06:47:54.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.327+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.2, "await": true}) -- 0x560553e77980 con 0x560553f8d100 2024-04-22T06:47:54.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.328+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f8d100 0x560553f8f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.329+0000 7f1a377f6640 10 client.25703 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:54.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.329+0000 7f1a52ffd640 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f8d100 0x560553f8f540 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d01063b287d0193 server_cookie=5bb1c39c787d0564 in_seq=0 out_seq=0 2024-04-22T06:47:54.699 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:47:54.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:47:54.699 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:47:54.804 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 -- 172.21.15.79:0/1007102750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 msgr2=0x7f05541035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 --2- 172.21.15.79:0/1007102750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f05541035c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0544009900 tx=0x7f054402bd60 comp rx=0 tx=0).stop 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 -- 172.21.15.79:0/1007102750 shutdown_connections 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 --2- 172.21.15.79:0/1007102750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0554103b00 0x7f055410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 --2- 172.21.15.79:0/1007102750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f055410c520 0x7f055410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 --2- 172.21.15.79:0/1007102750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f05541035c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 -- 172.21.15.79:0/1007102750 >> 172.21.15.79:0/1007102750 conn(0x7f05540771d0 msgr2=0x7f05540775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 -- 172.21.15.79:0/1007102750 shutdown_connections 2024-04-22T06:47:54.805 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.806+0000 7f0558b6d640 1 -- 172.21.15.79:0/1007102750 wait complete. 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 Processor -- start 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 -- start start 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f055419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554103b00 0x7f055419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f055410c520 0x7f0554198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0554112210 con 0x7f05541031e0 2024-04-22T06:47:54.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0554112090 con 0x7f055410c520 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0558b6d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0554112390 con 0x7f0554103b00 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f055419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f055419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47922/0 (socket says 172.21.15.79:47922) 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 -- 172.21.15.79:0/3674548716 learned_addr learned my addr 172.21.15.79:0/3674548716 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0551d9b640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554103b00 0x7f055419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f055410c520 0x7f0554198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 -- 172.21.15.79:0/3674548716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554103b00 msgr2=0x7f055419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554103b00 0x7f055419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 -- 172.21.15.79:0/3674548716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f055410c520 msgr2=0x7f0554198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f055410c520 0x7f0554198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.807+0000 7f055259c640 1 -- 172.21.15.79:0/3674548716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0554198d10 con 0x7f05541031e0 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.808+0000 7f055259c640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f055419e3e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0544005940 tx=0x7f054402d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.808+0000 7f05437fe640 1 -- 172.21.15.79:0/3674548716 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544043020 con 0x7f05541031e0 2024-04-22T06:47:54.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.808+0000 7f05437fe640 1 -- 172.21.15.79:0/3674548716 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054402dd20 con 0x7f05541031e0 2024-04-22T06:47:54.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.808+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0554198fa0 con 0x7f05541031e0 2024-04-22T06:47:54.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.808+0000 7f05437fe640 1 -- 172.21.15.79:0/3674548716 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544003b80 con 0x7f05541031e0 2024-04-22T06:47:54.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.808+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05541a52a0 con 0x7f05541031e0 2024-04-22T06:47:54.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.810+0000 7f05437fe640 1 -- 172.21.15.79:0/3674548716 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0544046020 con 0x7f05541031e0 2024-04-22T06:47:54.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.810+0000 7f055b834740 1 Processor -- start 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 -- start start 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf62cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf6b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf6be10 0x55b86cf6e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b86cec6f00 con 0x55b86cf628e0 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b86ce64590 con 0x55b86cf63290 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055b834740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b86cec2ec0 con 0x55b86cf6be10 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f05437fe640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f053c0a3d20 0x7f053c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf62cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf62cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47928/0 (socket says 172.21.15.79:47928) 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f055259c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf6b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 -- 172.21.15.79:0/3987774485 learned_addr learned my addr 172.21.15.79:0/3987774485 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0551d9b640 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf6be10 0x55b86cf6e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f053c0a3d20 0x7f053c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f05437fe640 1 -- 172.21.15.79:0/3674548716 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f05440f29d0 con 0x7f05541031e0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 -- 172.21.15.79:0/3987774485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf6be10 msgr2=0x55b86cf6e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf6be10 0x55b86cf6e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 -- 172.21.15.79:0/3987774485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 msgr2=0x55b86cf6b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf6b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 -- 172.21.15.79:0/3987774485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b86cca17e0 con 0x55b86cf628e0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.811+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf62cc0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f054c00af30 tx=0x7f054c016700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e0a5039163ea69 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f053c0a3d20 0x7f053c0a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f054c009fd0 tx=0x7f054c01a000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f0540ff9640 1 -- 172.21.15.79:0/3987774485 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c026070 con 0x55b86cf628e0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f0540ff9640 1 -- 172.21.15.79:0/3987774485 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054c00a590 con 0x55b86cf628e0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f0540ff9640 1 -- 172.21.15.79:0/3987774485 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c013040 con 0x55b86cf628e0 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 -- 172.21.15.79:0/3987774485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 msgr2=0x55b86cf62cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf62cc0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f054c00af30 tx=0x7f054c016700 comp rx=0 tx=0).stop 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 -- 172.21.15.79:0/3987774485 shutdown_connections 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf6b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf6be10 0x55b86cf6e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 --2- 172.21.15.79:0/3987774485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf62cc0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 -- 172.21.15.79:0/3987774485 >> 172.21.15.79:0/3987774485 conn(0x55b86cf5cfb0 msgr2=0x55b86cf5d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 -- 172.21.15.79:0/3987774485 shutdown_connections 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.812+0000 7f055b834740 1 -- 172.21.15.79:0/3987774485 wait complete. 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 -- start start 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 Processor -- start 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf628e0 0x55b86cf68d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf69270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 0x55b86cf697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b86cf6ed20 con 0x55b86cf6be10 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b86ce80580 con 0x55b86cf63290 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b86cf6eea0 con 0x55b86cf628e0 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055b834740 10 client.0 _ll_register_callbacks cb 0x55b86ce86650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:54.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 0x55b86cf697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 0x55b86cf697b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47930/0 (socket says 172.21.15.79:47930) 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 -- 172.21.15.79:0/3295775628 learned_addr learned my addr 172.21.15.79:0/3295775628 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0551d9b640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf69270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f055259c640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf628e0 0x55b86cf68d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 -- 172.21.15.79:0/3295775628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf628e0 msgr2=0x55b86cf68d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf628e0 0x55b86cf68d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 -- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 msgr2=0x55b86cf69270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf69270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.813+0000 7f0552d9d640 1 -- 172.21.15.79:0/3295775628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b86cd01f70 con 0x55b86cf6be10 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 0x55b86cf697b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f054c004b90 tx=0x7f054c009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62af87a0594f8e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0539ffb640 1 -- 172.21.15.79:0/3295775628 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c030070 con 0x55b86cf6be10 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0539ffb640 1 -- 172.21.15.79:0/3295775628 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f054c00a880 con 0x55b86cf6be10 2024-04-22T06:47:54.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f054c00a590 con 0x55b86cf6be10 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0539ffb640 1 -- 172.21.15.79:0/3295775628 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f054c03c050 con 0x55b86cf6be10 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 10 client.16659 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b86cb99170 con 0x55b86cf6be10 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0539ffb640 1 -- 172.21.15.79:0/3295775628 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054c00a590 con 0x55b86cf6be10 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f0539ffb640 1 -- 172.21.15.79:0/3295775628 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f054c02b480 con 0x55b86cf6be10 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 10 client.16659 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 20 client.16659 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cf7c1d0 0x55b86cf7e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 4 client.16659 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b86ced4920 con 0x55b86cf7c1d0 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.814+0000 7f055259c640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cf7c1d0 0x55b86cf7e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.815+0000 7f0539ffb640 10 client.16659 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.815+0000 7f055259c640 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cf7c1d0 0x55b86cf7e5c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e214fb8616d2773 server_cookie=20dadef17f5d7d8 in_seq=0 out_seq=0 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f0539ffb640 1 -- 172.21.15.79:0/3295775628 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b86ced4920 con 0x55b86cf7c1d0 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f0539ffb640 10 client.16659 handle_command_reply: tid=0 2024-04-22T06:47:54.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 client.16659 shutdown 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 msgr2=0x55b86cf697b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 0x55b86cf697b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f054c004b90 tx=0x7f054c009d30 comp rx=0 tx=0).stop 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 shutdown_connections 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cf7c1d0 0x55b86cf7e5c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf63290 0x55b86cf69270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf6be10 0x55b86cf697b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 --2- 172.21.15.79:0/3295775628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf628e0 0x55b86cf68d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f0539ffb640 0 client.16659 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 >> 172.21.15.79:0/3295775628 conn(0x55b86cf5cfb0 msgr2=0x55b86cf5ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 shutdown_connections 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 1 -- 172.21.15.79:0/3295775628 wait complete. 2024-04-22T06:47:54.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.817+0000 7f055b834740 20 client.16659 trim_cache size 0 max 16384 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 Processor -- start 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 -- start start 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf77be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf695d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cf83c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f05541b0630 con 0x55b86cf628e0 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05541adf00 con 0x55b86cf6be10 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055b834740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05541abd50 con 0x55b86cf63290 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf77be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf77be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47946/0 (socket says 172.21.15.79:47946) 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.834+0000 7f055259c640 1 -- 172.21.15.79:0/1597397216 learned_addr learned my addr 172.21.15.79:0/1597397216 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f0551d9b640 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f0552d9d640 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cf83c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055259c640 1 -- 172.21.15.79:0/1597397216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 msgr2=0x55b86cf695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055259c640 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf695d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055259c640 1 -- 172.21.15.79:0/1597397216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 msgr2=0x55b86cf83c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055259c640 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cf83c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055259c640 1 -- 172.21.15.79:0/1597397216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b86cf5d8f0 con 0x55b86cf628e0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f0552d9d640 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cf83c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055259c640 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf77be0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f05440bad70 tx=0x7f0544091450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721b0c4cd31eb28c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f053bfff640 1 -- 172.21.15.79:0/1597397216 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544089470 con 0x55b86cf628e0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f053bfff640 1 -- 172.21.15.79:0/1597397216 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05440be960 con 0x55b86cf628e0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f053bfff640 1 -- 172.21.15.79:0/1597397216 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544005460 con 0x55b86cf628e0 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055b834740 1 -- 172.21.15.79:0/1597397216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 msgr2=0x55b86cf77be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055b834740 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf77be0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f05440bad70 tx=0x7f0544091450 comp rx=0 tx=0).stop 2024-04-22T06:47:54.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055b834740 1 -- 172.21.15.79:0/1597397216 shutdown_connections 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055b834740 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cf83c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055b834740 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.835+0000 7f055b834740 1 --2- 172.21.15.79:0/1597397216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf77be0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- 172.21.15.79:0/1597397216 >> 172.21.15.79:0/1597397216 conn(0x55b86cf5cfb0 msgr2=0x55b86cf5ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- 172.21.15.79:0/1597397216 shutdown_connections 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- 172.21.15.79:0/1597397216 wait complete. 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- start start 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 Processor -- start 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf6ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf70450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cfe14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f05541ac3c0 con 0x55b86cf628e0 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05541ad830 con 0x55b86cf6be10 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05541aca90 con 0x55b86cf63290 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055b834740 10 client.0 _ll_register_callbacks cb 0x7f054c024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:54.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf6ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf6ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47960/0 (socket says 172.21.15.79:47960) 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 -- 172.21.15.79:0/3249305691 learned_addr learned my addr 172.21.15.79:0/3249305691 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f0551d9b640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf70450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cfe14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 -- 172.21.15.79:0/3249305691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 msgr2=0x55b86cf70450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf70450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 -- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 msgr2=0x55b86cfe14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cfe14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f055259c640 1 -- 172.21.15.79:0/3249305691 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b86cfe1a00 con 0x55b86cf628e0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.836+0000 7f0552d9d640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cfe14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055259c640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf6ff10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0544006010 tx=0x7f0544009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd8dbebcd3d1e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0539ffb640 1 -- 172.21.15.79:0/3249305691 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440c75d0 con 0x55b86cf628e0 2024-04-22T06:47:54.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0539ffb640 1 -- 172.21.15.79:0/3249305691 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f05440f0280 con 0x55b86cf628e0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05440be960 con 0x55b86cf628e0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0539ffb640 1 -- 172.21.15.79:0/3249305691 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f05440bcc60 con 0x55b86cf628e0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 10 client.16665 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b86cfe2b50 con 0x55b86cf628e0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0539ffb640 1 -- 172.21.15.79:0/3249305691 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05440be960 con 0x55b86cf628e0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0539ffb640 1 -- 172.21.15.79:0/3249305691 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f05440a90b0 con 0x55b86cf628e0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 10 client.16665 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 20 client.16665 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cfe3210 0x55b86cfe5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 4 client.16665 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b86cece790 con 0x55b86cfe3210 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.837+0000 7f0551d9b640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cfe3210 0x55b86cfe5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.838+0000 7f0539ffb640 10 client.16665 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:54.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:54.838+0000 7f0551d9b640 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cfe3210 0x55b86cfe5600 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dde959fe8e84f4d4 server_cookie=d6370ec5187c802a in_seq=0 out_seq=0 2024-04-22T06:47:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:54 smithi079 ceph-mon[27946]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 866 MiB/s wr, 1.54k op/s 2024-04-22T06:47:55.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f0539ffb640 1 -- 172.21.15.79:0/3249305691 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b86cece790 con 0x55b86cfe3210 2024-04-22T06:47:55.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f0539ffb640 10 client.16665 handle_command_reply: tid=0 2024-04-22T06:47:55.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 client.16665 shutdown 2024-04-22T06:47:55.000 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 msgr2=0x55b86cf6ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf6ff10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0544006010 tx=0x7f0544009650 comp rx=0 tx=0).stop 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 shutdown_connections 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b86cfe3210 0x55b86cfe5600 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b86cf6be10 0x55b86cfe14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f0539ffb640 0 client.16665 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b86cf63290 0x55b86cf70450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 --2- 172.21.15.79:0/3249305691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b86cf628e0 0x55b86cf6ff10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 >> 172.21.15.79:0/3249305691 conn(0x55b86cf5cfb0 msgr2=0x55b86cf5b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.001+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 shutdown_connections 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f055b834740 1 -- 172.21.15.79:0/3249305691 wait complete. 2024-04-22T06:47:55.001 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f055b834740 20 client.16665 trim_cache size 0 max 16384 2024-04-22T06:47:55.002 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:55.002 INFO:teuthology.orchestra.run.smithi079.stdout: "return_code": 0, 2024-04-22T06:47:55.002 INFO:teuthology.orchestra.run.smithi079.stdout: "scrub_tag": "46a66088-178b-4809-abcb-007a9cd7cea4", 2024-04-22T06:47:55.002 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": "asynchronous" 2024-04-22T06:47:55.002 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f053c0a3d20 msgr2=0x7f053c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f053c0a3d20 0x7f053c0a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f054c009fd0 tx=0x7f054c01a000 comp rx=0 tx=0).stop 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 msgr2=0x7f055419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f055419e3e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0544005940 tx=0x7f054402d920 comp rx=0 tx=0).stop 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 shutdown_connections 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f053c0a3d20 0x7f053c0a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f055410c520 0x7f0554198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.003 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554103b00 0x7f055419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 --2- 172.21.15.79:0/3674548716 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05541031e0 0x7f055419e3e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 >> 172.21.15.79:0/3674548716 conn(0x7f05540771d0 msgr2=0x7f05540fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 shutdown_connections 2024-04-22T06:47:55.004 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.002+0000 7f0558b6d640 1 -- 172.21.15.79:0/3674548716 wait complete. 2024-04-22T06:47:55.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:47:55.011 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 -- 172.21.15.79:0/1406540195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6730101a30 msgr2=0x7f6730105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 --2- 172.21.15.79:0/1406540195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6730101a30 0x7f6730105a70 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6718009900 tx=0x7f671802bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 -- 172.21.15.79:0/1406540195 shutdown_connections 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 --2- 172.21.15.79:0/1406540195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730106150 0x7f673010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 --2- 172.21.15.79:0/1406540195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6730101a30 0x7f6730105a70 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 --2- 172.21.15.79:0/1406540195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6730101080 0x7f6730101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.119+0000 7f67355db640 1 -- 172.21.15.79:0/1406540195 >> 172.21.15.79:0/1406540195 conn(0x7f6730078000 msgr2=0x7f6730078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 -- 172.21.15.79:0/1406540195 shutdown_connections 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 -- 172.21.15.79:0/1406540195 wait complete. 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 Processor -- start 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 -- start start 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 0x7f67301986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6730106150 0x7f6730198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 0x7f673019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6730111070 con 0x7f673019cf90 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6730110ef0 con 0x7f6730101080 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.120+0000 7f67355db640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67301111f0 con 0x7f6730106150 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 0x7f673019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 0x7f67301986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 0x7f673019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47978/0 (socket says 172.21.15.79:47978) 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 0x7f67301986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:58124/0 (socket says 172.21.15.79:58124) 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 -- 172.21.15.79:0/16783092 learned_addr learned my addr 172.21.15.79:0/16783092 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:55.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f67267fc640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6730106150 0x7f6730198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 -- 172.21.15.79:0/16783092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6730106150 msgr2=0x7f6730198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6730106150 0x7f6730198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 -- 172.21.15.79:0/16783092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 msgr2=0x7f67301986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 0x7f67301986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 -- 172.21.15.79:0/16783092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673019da80 con 0x7f673019cf90 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672effd640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 0x7f67301986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672f7fe640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 0x7f673019d440 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f672800eb10 tx=0x7f672800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672cff9640 1 -- 172.21.15.79:0/16783092 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728004b20 con 0x7f673019cf90 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672cff9640 1 -- 172.21.15.79:0/16783092 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6728004cc0 con 0x7f673019cf90 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f672cff9640 1 -- 172.21.15.79:0/16783092 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728010670 con 0x7f673019cf90 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673019dd70 con 0x7f673019cf90 2024-04-22T06:47:55.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.121+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67301a54e0 con 0x7f673019cf90 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.124+0000 7f672cff9640 1 -- 172.21.15.79:0/16783092 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6728005020 con 0x7f673019cf90 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.124+0000 7f67381ce740 1 Processor -- start 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.124+0000 7f67381ce740 1 -- start start 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67381ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0b580 0x55870ac0b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67381ce740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac14540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67381ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac16e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67381ce740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55870ab701e0 con 0x55870ac0bea0 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67381ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55870ab0d390 con 0x55870ac0b580 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67381ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55870ab6c1a0 con 0x55870ac14a80 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac14540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac14540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47984/0 (socket says 172.21.15.79:47984) 2024-04-22T06:47:55.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 -- 172.21.15.79:0/899443846 learned_addr learned my addr 172.21.15.79:0/899443846 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67267fc640 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0b580 0x55870ac0b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672cff9640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f671c0a3d20 0x7f671c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67267fc640 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac16e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 -- 172.21.15.79:0/899443846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 msgr2=0x55870ac16e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac16e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672f7fe640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f671c0a3d20 0x7f671c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 -- 172.21.15.79:0/899443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0b580 msgr2=0x55870ac0b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0b580 0x55870ac0b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 -- 172.21.15.79:0/899443846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870a94c7d0 con 0x55870ac0bea0 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67267fc640 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0b580 0x55870ac0b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672cff9640 1 -- 172.21.15.79:0/16783092 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f67280d0ed0 con 0x7f673019cf90 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f672effd640 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac14540 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6720010480 tx=0x7f6720010940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a50ba9bead62eed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.125+0000 7f67257fa640 1 -- 172.21.15.79:0/899443846 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720011760 con 0x55870ac0bea0 2024-04-22T06:47:55.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f672f7fe640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f671c0a3d20 0x7f671c0a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6728099720 tx=0x7f6728098ff0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67257fa640 1 -- 172.21.15.79:0/899443846 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6720011db0 con 0x55870ac0bea0 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67257fa640 1 -- 172.21.15.79:0/899443846 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672001ce60 con 0x55870ac0bea0 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 -- 172.21.15.79:0/899443846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 msgr2=0x55870ac14540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac14540 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6720010480 tx=0x7f6720010940 comp rx=0 tx=0).stop 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 -- 172.21.15.79:0/899443846 shutdown_connections 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac16e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac14540 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 --2- 172.21.15.79:0/899443846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0b580 0x55870ac0b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.126+0000 7f67381ce740 1 -- 172.21.15.79:0/899443846 >> 172.21.15.79:0/899443846 conn(0x55870ac05cb0 msgr2=0x55870ac06090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 -- 172.21.15.79:0/899443846 shutdown_connections 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 -- 172.21.15.79:0/899443846 wait complete. 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 -- start start 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 Processor -- start 2024-04-22T06:47:55.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0bea0 0x55870ac11ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac12530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55870ac17870 con 0x55870ac0b580 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55870ab292f0 con 0x55870ac14a80 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55870ac179f0 con 0x55870ac0bea0 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67381ce740 10 client.0 _ll_register_callbacks cb 0x55870ab36580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67267fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67267fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac11ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47998/0 (socket says 172.21.15.79:47998) 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67267fc640 1 -- 172.21.15.79:0/494566907 learned_addr learned my addr 172.21.15.79:0/494566907 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67267fc640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac12530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f672effd640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0bea0 0x55870ac11ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67267fc640 1 -- 172.21.15.79:0/494566907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0bea0 msgr2=0x55870ac11ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.127+0000 7f67267fc640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0bea0 0x55870ac11ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67267fc640 1 -- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 msgr2=0x55870ac12530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67267fc640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac12530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67267fc640 1 -- 172.21.15.79:0/494566907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870a8823f0 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67267fc640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac11ab0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6718035000 tx=0x7f6718005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640669f6b2157399 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67167fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67167fc640 1 -- 172.21.15.79:0/494566907 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671802cba0 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67167fc640 1 -- 172.21.15.79:0/494566907 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f6718005050 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6720011db0 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67167fc640 1 -- 172.21.15.79:0/494566907 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6718053050 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 10 client.16677 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55870a8420f0 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67167fc640 1 -- 172.21.15.79:0/494566907 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6720011db0 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67167fc640 1 -- 172.21.15.79:0/494566907 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f6718040040 con 0x55870ac0b580 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 10 client.16677 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:55.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 20 client.16677 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:55.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.128+0000 7f67381ce740 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac24a70 0x55870ac26e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.129+0000 7f67381ce740 4 client.16677 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:55.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.129+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55870ab7dc00 con 0x55870ac24a70 2024-04-22T06:47:55.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.129+0000 7f672effd640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac24a70 0x55870ac26e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.129+0000 7f672effd640 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac24a70 0x55870ac26e80 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b86fd6f38b558743 server_cookie=52525c9d7134b911 in_seq=0 out_seq=0 2024-04-22T06:47:55.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.129+0000 7f67167fc640 10 client.16677 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67167fc640 1 -- 172.21.15.79:0/494566907 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55870ab7dc00 con 0x55870ac24a70 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67167fc640 10 client.16677 handle_command_reply: tid=0 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 client.16677 shutdown 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 msgr2=0x55870ac11ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac11ab0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6718035000 tx=0x7f6718005be0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 shutdown_connections 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac24a70 0x55870ac26e80 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.134 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac12530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0bea0 0x55870ac11ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67167fc640 0 client.16677 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 --2- 172.21.15.79:0/494566907 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac11ab0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 >> 172.21.15.79:0/494566907 conn(0x55870ac05cb0 msgr2=0x55870ac07a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 shutdown_connections 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 1 -- 172.21.15.79:0/494566907 wait complete. 2024-04-22T06:47:55.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.130+0000 7f67381ce740 20 client.16677 trim_cache size 0 max 16384 2024-04-22T06:47:55.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 Processor -- start 2024-04-22T06:47:55.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 -- start start 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0bea0 0x55870ac20730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac10c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f67301b05e0 con 0x55870ac0b580 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f67301adeb0 con 0x55870ac0bea0 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67381ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67301abd00 con 0x55870ac14a80 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f672effd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0bea0 0x55870ac20730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac141e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48012/0 (socket says 172.21.15.79:48012) 2024-04-22T06:47:55.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 -- 172.21.15.79:0/3911638855 learned_addr learned my addr 172.21.15.79:0/3911638855 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 -- 172.21.15.79:0/3911638855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 msgr2=0x55870ac10c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac10c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 -- 172.21.15.79:0/3911638855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0bea0 msgr2=0x55870ac20730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0bea0 0x55870ac20730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.148+0000 7f67267fc640 1 -- 172.21.15.79:0/3911638855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870ac0bcd0 con 0x55870ac0b580 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.149+0000 7f67267fc640 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac141e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6718006010 tx=0x7f6718000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fd96f99d8ffdf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.149+0000 7f6724ff9640 1 -- 172.21.15.79:0/3911638855 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671805b020 con 0x55870ac0b580 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.149+0000 7f6724ff9640 1 -- 172.21.15.79:0/3911638855 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671804a070 con 0x55870ac0b580 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.149+0000 7f6724ff9640 1 -- 172.21.15.79:0/3911638855 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180056a0 con 0x55870ac0b580 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.149+0000 7f67381ce740 1 -- 172.21.15.79:0/3911638855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 msgr2=0x55870ac141e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.149+0000 7f67381ce740 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac141e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6718006010 tx=0x7f6718000bc0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- 172.21.15.79:0/3911638855 shutdown_connections 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac0bea0 0x55870ac20730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac14a80 0x55870ac10c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 --2- 172.21.15.79:0/3911638855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0b580 0x55870ac141e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- 172.21.15.79:0/3911638855 >> 172.21.15.79:0/3911638855 conn(0x55870ac05cb0 msgr2=0x55870ac070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- 172.21.15.79:0/3911638855 shutdown_connections 2024-04-22T06:47:55.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- 172.21.15.79:0/3911638855 wait complete. 2024-04-22T06:47:55.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- start start 2024-04-22T06:47:55.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 Processor -- start 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0b580 0x55870ac1cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac18280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f67301ac370 con 0x55870ac0bea0 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f67301ad7e0 con 0x55870ac14a80 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67301aca40 con 0x55870ac0b580 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67381ce740 10 client.0 _ll_register_callbacks cb 0x7f67280093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f672effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac18280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f672effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac18280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48024/0 (socket says 172.21.15.79:48024) 2024-04-22T06:47:55.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f672effd640 1 -- 172.21.15.79:0/3309148216 learned_addr learned my addr 172.21.15.79:0/3309148216 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67267fc640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.150+0000 7f67267fc640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0b580 0x55870ac1cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f672effd640 1 -- 172.21.15.79:0/3309148216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0b580 msgr2=0x55870ac1cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f672effd640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0b580 0x55870ac1cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f672effd640 1 -- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 msgr2=0x55870ac187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f672effd640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f672effd640 1 -- 172.21.15.79:0/3309148216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870ac18e60 con 0x55870ac0bea0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f672effd640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac18280 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6720013010 tx=0x7f6720005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98af6515e3a4748 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67167fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67167fc640 1 -- 172.21.15.79:0/3309148216 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720005380 con 0x55870ac0bea0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67167fc640 1 -- 172.21.15.79:0/3309148216 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f6720005520 con 0x55870ac0bea0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f671804a070 con 0x55870ac0bea0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67167fc640 1 -- 172.21.15.79:0/3309148216 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6720004660 con 0x55870ac0bea0 2024-04-22T06:47:55.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 10 client.16683 fetch_fsmap learned FSMap version 40 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55870ac8aeb0 con 0x55870ac0bea0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67167fc640 1 -- 172.21.15.79:0/3309148216 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671804a070 con 0x55870ac0bea0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67167fc640 1 -- 172.21.15.79:0/3309148216 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f6720027a80 con 0x55870ac0bea0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 10 client.16683 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 20 client.16683 populate_metadata read hostname 'smithi079' 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac8b4d0 0x55870ac8d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 4 client.16683 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.151+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55870ab77a70 con 0x55870ac8b4d0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67267fc640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac8b4d0 0x55870ac8d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67167fc640 10 client.16683 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67267fc640 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac8b4d0 0x55870ac8d8f0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbb6430847002bd1 server_cookie=624413933db38e88 in_seq=0 out_seq=0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67167fc640 1 -- 172.21.15.79:0/3309148216 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55870ab77a70 con 0x55870ac8b4d0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67167fc640 10 client.16683 handle_command_reply: tid=0 2024-04-22T06:47:55.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67381ce740 1 client.16683 shutdown 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 msgr2=0x55870ac18280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.152+0000 7f67381ce740 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac18280 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6720013010 tx=0x7f6720005040 comp rx=0 tx=0).stop 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 shutdown_connections 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55870ac8b4d0 0x55870ac8d8f0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55870ac14a80 0x55870ac187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55870ac0bea0 0x55870ac18280 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 --2- 172.21.15.79:0/3309148216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55870ac0b580 0x55870ac1cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67167fc640 0 client.16683 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 >> 172.21.15.79:0/3309148216 conn(0x55870ac05cb0 msgr2=0x55870ac07970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 shutdown_connections 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 1 -- 172.21.15.79:0/3309148216 wait complete. 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67381ce740 20 client.16683 trim_cache size 0 max 16384 2024-04-22T06:47:55.156 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": { 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: "46a66088-178b-4809-abcb-007a9cd7cea4": { 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: "path": "/", 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: "tag": "46a66088-178b-4809-abcb-007a9cd7cea4", 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: "options": "recursive,force" 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.157 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f671c0a3d20 msgr2=0x7f671c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f671c0a3d20 0x7f671c0a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6728099720 tx=0x7f6728098ff0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 msgr2=0x7f673019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 0x7f673019d440 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f672800eb10 tx=0x7f672800c2d0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 shutdown_connections 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f673019cf90 0x7f673019d440 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6730106150 0x7f6730198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f671c0a3d20 0x7f671c0a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 --2- 172.21.15.79:0/16783092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6730101080 0x7f67301986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 >> 172.21.15.79:0/16783092 conn(0x7f6730078000 msgr2=0x7f6730100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 shutdown_connections 2024-04-22T06:47:55.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.153+0000 7f67355db640 1 -- 172.21.15.79:0/16783092 wait complete. 2024-04-22T06:47:55.162 INFO:tasks.cephfs.filesystem:scrub status for tag:46a66088-178b-4809-abcb-007a9cd7cea4 - {'path': '/', 'tag': '46a66088-178b-4809-abcb-007a9cd7cea4', 'options': 'recursive,force'} 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a377f6640 1 -- 172.21.15.79:0/66886839 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x560553e77980 con 0x560553f8d100 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a377f6640 10 client.25703 handle_command_reply: tid=0 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 client.25703 shutdown 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 msgr2=0x560553f194c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 0x560553f194c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009e40 tx=0x7f1a44006b80 comp rx=0 tx=0).stop 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 shutdown_connections 2024-04-22T06:47:55.188 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560553f8d100 0x560553f8f540 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560553f15090 0x560553f194c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560553f19a00 0x560553f19e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a377f6640 0 client.25703 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 --2- 172.21.15.79:0/66886839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560553f0c4b0 0x560553f1d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 >> 172.21.15.79:0/66886839 conn(0x560553f062c0 msgr2=0x560553f053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 shutdown_connections 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.189+0000 7f1a5c977740 1 -- 172.21.15.79:0/66886839 wait complete. 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a5c977740 20 client.25703 trim_cache size 0 max 16384 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 40, 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 12, 2024-04-22T06:47:55.189 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "a5377597": { 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 12, 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 86.2, 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:47:55.190 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:47:55.191 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a3c0a3f20 msgr2=0x7f1a3c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a3c0a3f20 0x7f1a3c0a63e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1a480cc9c0 tx=0x7f1a480ce600 comp rx=0 tx=0).stop 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 msgr2=0x7f1a541199a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541199a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a4801a860 tx=0x7f1a48004040 comp rx=0 tx=0).stop 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 shutdown_connections 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a3c0a3f20 0x7f1a3c0a63e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5410eb50 0x7f1a54119460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a541104d0 0x7f1a541199a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 --2- 172.21.15.79:0/643857690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a5410e030 0x7f1a5411d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 >> 172.21.15.79:0/643857690 conn(0x7f1a5406ce60 msgr2=0x7f1a540715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 shutdown_connections 2024-04-22T06:47:55.192 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:47:55.190+0000 7f1a58b2c640 1 -- 172.21.15.79:0/643857690 wait complete. 2024-04-22T06:47:55.193 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:54 smithi193 ceph-mon[31815]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 866 MiB/s wr, 1.54k op/s 2024-04-22T06:47:55.200 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: a5377597, quiesce took 3.0 seconds. Will release after: 26.2 2024-04-22T06:47:55.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:54 smithi190 ceph-mon[31421]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 866 MiB/s wr, 1.54k op/s 2024-04-22T06:47:56.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:55 smithi190 ceph-mon[31421]: scrub queued for path: / 2024-04-22T06:47:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:55 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:55 smithi190 ceph-mon[31421]: scrub summary: active paths [/] 2024-04-22T06:47:56.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:55 smithi190 ceph-mon[31421]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:55 smithi079 ceph-mon[27946]: scrub queued for path: / 2024-04-22T06:47:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:55 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:55 smithi079 ceph-mon[27946]: scrub summary: active paths [/] 2024-04-22T06:47:56.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:55 smithi079 ceph-mon[27946]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:55 smithi193 ceph-mon[31815]: scrub queued for path: / 2024-04-22T06:47:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:55 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:55 smithi193 ceph-mon[31815]: scrub summary: active paths [/] 2024-04-22T06:47:56.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:55 smithi193 ceph-mon[31815]: scrub summary: idle+waiting paths [/] 2024-04-22T06:47:57.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:56 smithi190 ceph-mon[31421]: pgmap v112: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 726 MiB/s wr, 1.24k op/s 2024-04-22T06:47:57.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:56 smithi079 ceph-mon[27946]: pgmap v112: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 726 MiB/s wr, 1.24k op/s 2024-04-22T06:47:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:56 smithi193 ceph-mon[31815]: pgmap v112: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 726 MiB/s wr, 1.24k op/s 2024-04-22T06:47:59.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:58 smithi190 ceph-mon[31421]: pgmap v113: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 953 MiB/s wr, 2.06k op/s 2024-04-22T06:47:59.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:47:58 smithi190 ceph-mon[31421]: scrub summary: idle 2024-04-22T06:47:59.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:58 smithi079 ceph-mon[27946]: pgmap v113: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 953 MiB/s wr, 2.06k op/s 2024-04-22T06:47:59.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:47:58 smithi079 ceph-mon[27946]: scrub summary: idle 2024-04-22T06:47:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:58 smithi193 ceph-mon[31815]: pgmap v113: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 953 MiB/s wr, 2.06k op/s 2024-04-22T06:47:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:47:58 smithi193 ceph-mon[31815]: scrub summary: idle 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 -- 172.21.15.79:0/1400667111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 msgr2=0x7fa62c106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 --2- 172.21.15.79:0/1400667111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c106a60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009930 tx=0x7fa61c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 -- 172.21.15.79:0/1400667111 shutdown_connections 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 --2- 172.21.15.79:0/1400667111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa62c1005e0 0x7fa62c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 --2- 172.21.15.79:0/1400667111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c100fa0 0x7fa62c10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 --2- 172.21.15.79:0/1400667111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c106a60 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 -- 172.21.15.79:0/1400667111 >> 172.21.15.79:0/1400667111 conn(0x7fa62c077390 msgr2=0x7fa62c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 -- 172.21.15.79:0/1400667111 shutdown_connections 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.656+0000 7fa632904640 1 -- 172.21.15.79:0/1400667111 wait complete. 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 Processor -- start 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 -- start start 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c1005e0 0x7fa62c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa62c100fa0 0x7fa62c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c110030 con 0x7fa62c106680 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c10feb0 con 0x7fa62c100fa0 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa632904640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1101b0 con 0x7fa62c1005e0 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa630ea1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa630ea1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48034/0 (socket says 172.21.15.79:48034) 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa630ea1640 1 -- 172.21.15.79:0/3067305447 learned_addr learned my addr 172.21.15.79:0/3067305447 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.657+0000 7fa62bfff640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c1005e0 0x7fa62c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa62c100fa0 0x7fa62c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa630ea1640 1 -- 172.21.15.79:0/3067305447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c1005e0 msgr2=0x7fa62c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa630ea1640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c1005e0 0x7fa62c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa630ea1640 1 -- 172.21.15.79:0/3067305447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa62c100fa0 msgr2=0x7fa62c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa630ea1640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa62c100fa0 0x7fa62c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa630ea1640 1 -- 172.21.15.79:0/3067305447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c19d680 con 0x7fa62c106680 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa630ea1640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c19cf50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa62400eb10 tx=0x7fa62400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa6297fa640 1 -- 172.21.15.79:0/3067305447 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624004b20 con 0x7fa62c106680 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c19d970 con 0x7fa62c106680 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa6297fa640 1 -- 172.21.15.79:0/3067305447 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa624004cc0 con 0x7fa62c106680 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa6297fa640 1 -- 172.21.15.79:0/3067305447 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624010690 con 0x7fa62c106680 2024-04-22T06:48:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.658+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1a5200 con 0x7fa62c106680 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6297fa640 1 -- 172.21.15.79:0/3067305447 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa624005020 con 0x7fa62c106680 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 Processor -- start 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 -- start start 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e552620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e552b60 0x55e97e552fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 0x55e97e55e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e97e4b78d0 con 0x55e97e55c0e0 2024-04-22T06:48:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e97e454e80 con 0x55e97e552b60 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6355d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e97e4b3890 con 0x55e97e5538d0 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 0x55e97e55e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa6297fa640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6140a3dc0 0x7fa6140a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 0x55e97e55e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48040/0 (socket says 172.21.15.79:48040) 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa62b7fe640 1 -- 172.21.15.79:0/1128795954 learned_addr learned my addr 172.21.15.79:0/1128795954 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa62bfff640 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e552b60 0x55e97e552fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e552620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.661+0000 7fa630ea1640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6140a3dc0 0x7fa6140a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa62b7fe640 1 -- 172.21.15.79:0/1128795954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 msgr2=0x55e97e552620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e552620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa62b7fe640 1 -- 172.21.15.79:0/1128795954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e552b60 msgr2=0x55e97e552fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e552b60 0x55e97e552fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa62b7fe640 1 -- 172.21.15.79:0/1128795954 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97e476320 con 0x55e97e55c0e0 2024-04-22T06:48:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa6297fa640 1 -- 172.21.15.79:0/3067305447 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fa6240d0f10 con 0x7fa62c106680 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 0x55e97e55e4d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa62000cf80 tx=0x7fa62000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c095aa384168e3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa630ea1640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6140a3dc0 0x7fa6140a6280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa6240ceb40 tx=0x7fa6240878c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa6127fc640 1 -- 172.21.15.79:0/1128795954 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000f110 con 0x55e97e55c0e0 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa6127fc640 1 -- 172.21.15.79:0/1128795954 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa62000f2b0 con 0x55e97e55c0e0 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.662+0000 7fa6127fc640 1 -- 172.21.15.79:0/1128795954 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000d800 con 0x55e97e55c0e0 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- 172.21.15.79:0/1128795954 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 msgr2=0x55e97e55e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 0x55e97e55e4d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa62000cf80 tx=0x7fa62000c7b0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- 172.21.15.79:0/1128795954 shutdown_connections 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e552b60 0x55e97e552fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e55c0e0 0x55e97e55e4d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- 172.21.15.79:0/1128795954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e552620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- 172.21.15.79:0/1128795954 >> 172.21.15.79:0/1128795954 conn(0x55e97e519c90 msgr2=0x55e97e54d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- 172.21.15.79:0/1128795954 shutdown_connections 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- 172.21.15.79:0/1128795954 wait complete. 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- start start 2024-04-22T06:48:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 Processor -- start 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e55bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e5538d0 0x55e97e558e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e55c0e0 0x55e97e5594a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e97e55ee10 con 0x55e97e552b60 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e97e43feb0 con 0x55e97e5538d0 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.663+0000 7fa6355d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e97e55ef90 con 0x55e97e55c0e0 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa6355d2740 10 client.0 _ll_register_callbacks cb 0x55e97e46cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e55bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e55bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48054/0 (socket says 172.21.15.79:48054) 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 -- 172.21.15.79:0/4205989940 learned_addr learned my addr 172.21.15.79:0/4205989940 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 -- 172.21.15.79:0/4205989940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e55c0e0 msgr2=0x55e97e5594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e55c0e0 0x55e97e5594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 -- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e5538d0 msgr2=0x55e97e558e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e5538d0 0x55e97e558e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 -- 172.21.15.79:0/4205989940 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97e1ca210 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e55bb00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa620019040 tx=0x7fa62000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fe4d06dceff2f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa60b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa60b7fe640 1 -- 172.21.15.79:0/4205989940 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000f110 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa60b7fe640 1 -- 172.21.15.79:0/4205989940 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fa620005460 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa62000f2b0 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa60b7fe640 1 -- 172.21.15.79:0/4205989940 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa62002d050 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa6355d2740 10 client.16695 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e97e296620 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.664+0000 7fa60b7fe640 1 -- 172.21.15.79:0/4205989940 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa62000f2b0 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa60b7fe640 1 -- 172.21.15.79:0/4205989940 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fa62000b040 con 0x55e97e552b60 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa6355d2740 10 client.16695 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa6355d2740 10 client.16695 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa6355d2740 20 client.16695 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa6355d2740 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e56c120 0x55e97e56e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa6355d2740 4 client.16695 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e97e4cb880 con 0x55e97e56c120 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.665+0000 7fa62bfff640 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e56c120 0x55e97e56e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.666+0000 7fa62bfff640 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e56c120 0x55e97e56e510 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c1aae08c6da90f server_cookie=74b3b6063219b155 in_seq=0 out_seq=0 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.666+0000 7fa60b7fe640 10 client.16695 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa60b7fe640 1 -- 172.21.15.79:0/4205989940 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e97e4cb880 con 0x55e97e56c120 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa60b7fe640 10 client.16695 handle_command_reply: tid=0 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 client.16695 shutdown 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 msgr2=0x55e97e55bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e55bb00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa620019040 tx=0x7fa62000c900 comp rx=0 tx=0).stop 2024-04-22T06:48:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 shutdown_connections 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e56c120 0x55e97e56e510 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e5538d0 0x55e97e558e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e55c0e0 0x55e97e5594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 --2- 172.21.15.79:0/4205989940 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e55bb00 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa60b7fe640 0 client.16695 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 >> 172.21.15.79:0/4205989940 conn(0x55e97e519c90 msgr2=0x55e97e54f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.667+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 shutdown_connections 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.668+0000 7fa6355d2740 1 -- 172.21.15.79:0/4205989940 wait complete. 2024-04-22T06:48:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.668+0000 7fa6355d2740 20 client.16695 trim_cache size 0 max 16384 2024-04-22T06:48:00.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 Processor -- start 2024-04-22T06:48:00.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 -- start start 2024-04-22T06:48:00.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e552b60 0x55e97e567d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 0x55e97e55b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 0x55e97e55bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1b06c0 con 0x55e97e5538d0 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1adf90 con 0x55e97e55c0e0 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa6355d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1abde0 con 0x55e97e552b60 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 0x55e97e55b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 0x55e97e55b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48070/0 (socket says 172.21.15.79:48070) 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 -- 172.21.15.79:0/3068085502 learned_addr learned my addr 172.21.15.79:0/3068085502 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e552b60 0x55e97e567d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 -- 172.21.15.79:0/3068085502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e552b60 msgr2=0x55e97e567d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e552b60 0x55e97e567d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 -- 172.21.15.79:0/3068085502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 msgr2=0x55e97e55bb00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.682+0000 7fa62bfff640 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 0x55e97e55bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa62bfff640 1 -- 172.21.15.79:0/3068085502 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97e551e30 con 0x55e97e5538d0 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa62bfff640 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 0x55e97e55b5c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa61c005ab0 tx=0x7fa61c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c88dd08c8c606b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa611ffb640 1 -- 172.21.15.79:0/3068085502 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c046c10 con 0x55e97e5538d0 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa611ffb640 1 -- 172.21.15.79:0/3068085502 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa61c046db0 con 0x55e97e5538d0 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa611ffb640 1 -- 172.21.15.79:0/3068085502 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c046720 con 0x55e97e5538d0 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 -- 172.21.15.79:0/3068085502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 msgr2=0x55e97e55b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 0x55e97e55b5c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa61c005ab0 tx=0x7fa61c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 -- 172.21.15.79:0/3068085502 shutdown_connections 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 0x55e97e55bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e5538d0 0x55e97e55b5c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3068085502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e552b60 0x55e97e567d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 -- 172.21.15.79:0/3068085502 >> 172.21.15.79:0/3068085502 conn(0x55e97e519c90 msgr2=0x55e97e54d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.683+0000 7fa6355d2740 1 -- 172.21.15.79:0/3068085502 shutdown_connections 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 -- 172.21.15.79:0/3068085502 wait complete. 2024-04-22T06:48:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 -- start start 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 Processor -- start 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e5644d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e55cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 0x55e97e55d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1ac450 con 0x55e97e552b60 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1ad8c0 con 0x55e97e55c0e0 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1acb20 con 0x55e97e5538d0 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa6355d2740 10 client.0 _ll_register_callbacks cb 0x7fa62000f460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e5644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa62b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e5644d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48086/0 (socket says 172.21.15.79:48086) 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.684+0000 7fa62b7fe640 1 -- 172.21.15.79:0/3406181936 learned_addr learned my addr 172.21.15.79:0/3406181936 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa62b7fe640 1 -- 172.21.15.79:0/3406181936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 msgr2=0x55e97e55cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e55cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa62b7fe640 1 -- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 msgr2=0x55e97e55d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 0x55e97e55d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa62b7fe640 1 -- 172.21.15.79:0/3406181936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97e55d760 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa62b7fe640 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e5644d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa620005020 tx=0x7fa620004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b87d10c757a3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa60b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa60b7fe640 1 -- 172.21.15.79:0/3406181936 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000c970 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa60b7fe640 1 -- 172.21.15.79:0/3406181936 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fa62000f0d0 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa61c046db0 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa60b7fe640 1 -- 172.21.15.79:0/3406181936 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa62000c510 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 10 client.16701 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e97e5d2350 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa60b7fe640 1 -- 172.21.15.79:0/3406181936 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa61c046db0 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa60b7fe640 1 -- 172.21.15.79:0/3406181936 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fa6200269f0 con 0x55e97e552b60 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 10 client.16701 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 20 client.16701 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e5d28c0 0x55e97e5d4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 4 client.16701 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.685+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e97e4573d0 con 0x55e97e5d28c0 2024-04-22T06:48:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.686+0000 7fa62bfff640 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e5d28c0 0x55e97e5d4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.686+0000 7fa60b7fe640 10 client.16701 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.686+0000 7fa62bfff640 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e5d28c0 0x55e97e5d4ce0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55077463e4b82eb9 server_cookie=834098a97ecd2fac in_seq=0 out_seq=0 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.702+0000 7fa60b7fe640 1 -- 172.21.15.79:0/3406181936 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e97e4573d0 con 0x55e97e5d28c0 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.702+0000 7fa60b7fe640 10 client.16701 handle_command_reply: tid=0 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 client.16701 shutdown 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 msgr2=0x55e97e5644d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e5644d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa620005020 tx=0x7fa620004bc0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 shutdown_connections 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e97e5d28c0 0x55e97e5d4ce0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e97e55c0e0 0x55e97e55d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e97e5538d0 0x55e97e55cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 --2- 172.21.15.79:0/3406181936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e97e552b60 0x55e97e5644d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 >> 172.21.15.79:0/3406181936 conn(0x55e97e519c90 msgr2=0x55e97e553ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.703+0000 7fa60b7fe640 0 client.16701 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 shutdown_connections 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa6355d2740 1 -- 172.21.15.79:0/3406181936 wait complete. 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa6355d2740 20 client.16701 trim_cache size 0 max 16384 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:48:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6140a3dc0 msgr2=0x7fa6140a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa632904640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6140a3dc0 0x7fa6140a6280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa6240ceb40 tx=0x7fa6240878c0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 msgr2=0x7fa62c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.704+0000 7fa632904640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c19cf50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa62400eb10 tx=0x7fa62400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 shutdown_connections 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa6140a3dc0 0x7fa6140a6280 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa62c100fa0 0x7fa62c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa62c106680 0x7fa62c19cf50 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 --2- 172.21.15.79:0/3067305447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c1005e0 0x7fa62c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 >> 172.21.15.79:0/3067305447 conn(0x7fa62c077390 msgr2=0x7fa62c104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 shutdown_connections 2024-04-22T06:48:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:00.705+0000 7fa632904640 1 -- 172.21.15.79:0/3067305447 wait complete. 2024-04-22T06:48:01.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:00 smithi190 ceph-mon[31421]: pgmap v114: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 1.42k op/s 2024-04-22T06:48:01.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:00 smithi079 ceph-mon[27946]: pgmap v114: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 1.42k op/s 2024-04-22T06:48:01.241 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:00 smithi193 ceph-mon[31815]: pgmap v114: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 1.42k op/s 2024-04-22T06:48:02.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:02 smithi193 ceph-mon[31815]: pgmap v115: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 624 MiB/s wr, 1.48k op/s 2024-04-22T06:48:02.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:02 smithi190 ceph-mon[31421]: pgmap v115: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 624 MiB/s wr, 1.48k op/s 2024-04-22T06:48:02.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:02 smithi079 ceph-mon[27946]: pgmap v115: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 624 MiB/s wr, 1.48k op/s 2024-04-22T06:48:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:04 smithi193 ceph-mon[31815]: pgmap v116: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 398 MiB/s wr, 1.14k op/s 2024-04-22T06:48:04.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:04 smithi190 ceph-mon[31421]: pgmap v116: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 398 MiB/s wr, 1.14k op/s 2024-04-22T06:48:04.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:04 smithi079 ceph-mon[27946]: pgmap v116: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 398 MiB/s wr, 1.14k op/s 2024-04-22T06:48:06.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:06 smithi193 ceph-mon[31815]: pgmap v117: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 266 MiB/s wr, 929 op/s 2024-04-22T06:48:06.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:06 smithi190 ceph-mon[31421]: pgmap v117: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 266 MiB/s wr, 929 op/s 2024-04-22T06:48:06.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:06 smithi079 ceph-mon[27946]: pgmap v117: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 266 MiB/s wr, 929 op/s 2024-04-22T06:48:08.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:08 smithi193 ceph-mon[31815]: pgmap v118: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 302 MiB/s wr, 994 op/s 2024-04-22T06:48:08.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:08 smithi190 ceph-mon[31421]: pgmap v118: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 302 MiB/s wr, 994 op/s 2024-04-22T06:48:08.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:08 smithi079 ceph-mon[27946]: pgmap v118: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 302 MiB/s wr, 994 op/s 2024-04-22T06:48:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:10 smithi193 ceph-mon[31815]: pgmap v119: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 74 MiB/s wr, 176 op/s 2024-04-22T06:48:10.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:10 smithi190 ceph-mon[31421]: pgmap v119: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 74 MiB/s wr, 176 op/s 2024-04-22T06:48:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:10 smithi079 ceph-mon[27946]: pgmap v119: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 74 MiB/s wr, 176 op/s 2024-04-22T06:48:12.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:12 smithi193 ceph-mon[31815]: pgmap v120: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 94 MiB/s wr, 210 op/s 2024-04-22T06:48:12.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:12 smithi190 ceph-mon[31421]: pgmap v120: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 94 MiB/s wr, 210 op/s 2024-04-22T06:48:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:12 smithi079 ceph-mon[27946]: pgmap v120: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 94 MiB/s wr, 210 op/s 2024-04-22T06:48:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1431878463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8105e80 msgr2=0x7f7fb8106260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1431878463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8105e80 0x7f7fb8106260 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009ca0 tx=0x7f7fb402c120 comp rx=0 tx=0).stop 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1431878463 shutdown_connections 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1431878463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fb8101f20 0x7f7fb810dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1431878463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fb8101560 0x7f7fb81019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1431878463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8105e80 0x7f7fb8106260 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1431878463 >> 172.21.15.79:0/1431878463 conn(0x7f7fb8077370 msgr2=0x7f7fb80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1431878463 shutdown_connections 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1431878463 wait complete. 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 Processor -- start 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.801+0000 7f7fbffb6640 1 -- start start 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fb8101560 0x7f7fb8198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 0x7f7fb8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fb8105e80 0x7f7fb819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8111130 con 0x7f7fb8101f20 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8110fb0 con 0x7f7fb8101560 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81112b0 con 0x7f7fb8105e80 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 0x7f7fb8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 0x7f7fb8198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40048/0 (socket says 172.21.15.79:40048) 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 -- 172.21.15.79:0/1194805737 learned_addr learned my addr 172.21.15.79:0/1194805737 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 -- 172.21.15.79:0/1194805737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fb8105e80 msgr2=0x7f7fb819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fb8105e80 0x7f7fb819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 -- 172.21.15.79:0/1194805737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fb8101560 msgr2=0x7f7fb8198660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fb8101560 0x7f7fb8198660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 -- 172.21.15.79:0/1194805737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb819d650 con 0x7f7fb8101f20 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbd551640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 0x7f7fb8198ba0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800b450 tx=0x7f7fa800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7faeffd640 1 -- 172.21.15.79:0/1194805737 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8015020 con 0x7f7fb8101f20 2024-04-22T06:48:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb819d940 con 0x7f7fb8101f20 2024-04-22T06:48:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7faeffd640 1 -- 172.21.15.79:0/1194805737 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fa8011d40 con 0x7f7fb8101f20 2024-04-22T06:48:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7faeffd640 1 -- 172.21.15.79:0/1194805737 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa800f340 con 0x7f7fb8101f20 2024-04-22T06:48:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.802+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb81a51d0 con 0x7f7fb8101f20 2024-04-22T06:48:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7faeffd640 1 -- 172.21.15.79:0/1194805737 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7fa8018020 con 0x7f7fb8101f20 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 Processor -- start 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 -- start start 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b986620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b986fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7faeffd640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7fa40a3dc0 0x7f7fa40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 0x56303b9968e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56303b8eb8d0 con 0x56303b9944f0 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56303b888e80 con 0x56303b986b60 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.805+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56303b8e7890 con 0x56303b9878d0 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 0x56303b9968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbd551640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7fa40a3dc0 0x7f7fa40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 0x56303b9968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40064/0 (socket says 172.21.15.79:40064) 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 -- 172.21.15.79:0/2444423104 learned_addr learned my addr 172.21.15.79:0/2444423104 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7faeffd640 1 -- 172.21.15.79:0/1194805737 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f7fa80d03d0 con 0x7f7fb8101f20 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbdd52640 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b986fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 -- 172.21.15.79:0/2444423104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 msgr2=0x56303b986620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b986620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 -- 172.21.15.79:0/2444423104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 msgr2=0x56303b986fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b986fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 -- 172.21.15.79:0/2444423104 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56303b8aa320 con 0x56303b9944f0 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbd551640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7fa40a3dc0 0x7f7fa40a6280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80cdbc0 tx=0x7f7fa801eb80 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 0x56303b9968e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000ba50 tx=0x7f7fb000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5930f0bdd358331 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fa3fff640 1 -- 172.21.15.79:0/2444423104 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0014030 con 0x56303b9944f0 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fa3fff640 1 -- 172.21.15.79:0/2444423104 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0005020 con 0x56303b9944f0 2024-04-22T06:48:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.806+0000 7f7fa3fff640 1 -- 172.21.15.79:0/2444423104 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0004b70 con 0x56303b9944f0 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2444423104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 msgr2=0x56303b9968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 0x56303b9968e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000ba50 tx=0x7f7fb000bf10 comp rx=0 tx=0).stop 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2444423104 shutdown_connections 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b986fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9944f0 0x56303b9968e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2444423104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b986620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2444423104 >> 172.21.15.79:0/2444423104 conn(0x56303b94dc90 msgr2=0x56303b981850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2444423104 shutdown_connections 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2444423104 wait complete. 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.807+0000 7f7fc2bab740 1 -- start start 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 Processor -- start 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b993ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 0x56303b991260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9944f0 0x56303b991870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56303b997220 con 0x56303b9878d0 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56303b873eb0 con 0x56303b986b60 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56303b9973a0 con 0x56303b9944f0 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fc2bab740 10 client.0 _ll_register_callbacks cb 0x56303b8a0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 0x56303b991260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 0x56303b991260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40066/0 (socket says 172.21.15.79:40066) 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 -- 172.21.15.79:0/2268777661 learned_addr learned my addr 172.21.15.79:0/2268777661 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 -- 172.21.15.79:0/2268777661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9944f0 msgr2=0x56303b991870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9944f0 0x56303b991870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 -- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 msgr2=0x56303b993ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b993ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.808+0000 7f7fbdd52640 1 -- 172.21.15.79:0/2268777661 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56303b5fe210 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fbdd52640 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 0x56303b991260 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7fb402c660 tx=0x7f7fb4032520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6146d9045f89732e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2268777661 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4043020 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb0005020 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2268777661 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f7fb4032890 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2268777661 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7fb404e050 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 10 client.16713 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56303b6ca620 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2268777661 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0005020 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2268777661 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f7fb4004650 con 0x56303b9878d0 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 10 client.16713 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 10 client.16713 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 20 client.16713 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303b9a4570 0x56303b9a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 4 client.16713 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.809+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56303b8ff880 con 0x56303b9a4570 2024-04-22T06:48:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.810+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303b9a4570 0x56303b9a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.813+0000 7f7fa17fa640 10 client.16713 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.813+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303b9a4570 0x56303b9a69c0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d8e1c372386a9a4 server_cookie=c6fb7ed6369d8874 in_seq=0 out_seq=0 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.815+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2268777661 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56303b8ff880 con 0x56303b9a4570 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.815+0000 7f7fa17fa640 10 client.16713 handle_command_reply: tid=0 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 client.16713 shutdown 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 msgr2=0x56303b991260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 0x56303b991260 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7fb402c660 tx=0x7f7fb4032520 comp rx=0 tx=0).stop 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 shutdown_connections 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303b9a4570 0x56303b9a69c0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9944f0 0x56303b991870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b9878d0 0x56303b991260 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fa17fa640 0 client.16713 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2268777661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b986b60 0x56303b993ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 >> 172.21.15.79:0/2268777661 conn(0x56303b94dc90 msgr2=0x56303b9835a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 shutdown_connections 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2268777661 wait complete. 2024-04-22T06:48:13.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.816+0000 7f7fc2bab740 20 client.16713 trim_cache size 0 max 16384 2024-04-22T06:48:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 Processor -- start 2024-04-22T06:48:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 -- start start 2024-04-22T06:48:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9a0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b990240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 0x56303b993990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81ae420 con 0x56303b986b60 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81abcf0 con 0x56303b9944f0 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81a9b40 con 0x56303b9878d0 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40070/0 (socket says 172.21.15.79:40070) 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 -- 172.21.15.79:0/37770439 learned_addr learned my addr 172.21.15.79:0/37770439 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 -- 172.21.15.79:0/37770439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 msgr2=0x56303b990240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b990240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 -- 172.21.15.79:0/37770439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 msgr2=0x56303b993990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.831+0000 7f7fbe553640 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 0x56303b993990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fbe553640 1 -- 172.21.15.79:0/37770439 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56303b985e30 con 0x56303b986b60 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fbe553640 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9a0150 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x56303b987600 tx=0x7f7fb000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a9c05e488252cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fa37fe640 1 -- 172.21.15.79:0/37770439 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0014040 con 0x56303b986b60 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fa37fe640 1 -- 172.21.15.79:0/37770439 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0015040 con 0x56303b986b60 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fa37fe640 1 -- 172.21.15.79:0/37770439 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0022520 con 0x56303b986b60 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 -- 172.21.15.79:0/37770439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 msgr2=0x56303b9a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9a0150 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x56303b987600 tx=0x7f7fb000b7c0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 -- 172.21.15.79:0/37770439 shutdown_connections 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 0x56303b993990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b990240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/37770439 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9a0150 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.832+0000 7f7fc2bab740 1 -- 172.21.15.79:0/37770439 >> 172.21.15.79:0/37770439 conn(0x56303b94dc90 msgr2=0x56303b98fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 -- 172.21.15.79:0/37770439 shutdown_connections 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 -- 172.21.15.79:0/37770439 wait complete. 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 -- start start 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 Processor -- start 2024-04-22T06:48:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9953b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b9958f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 0x56303b995e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81aa1b0 con 0x56303b986b60 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81ab620 con 0x56303b9944f0 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81aa880 con 0x56303b9878d0 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fc2bab740 10 client.0 _ll_register_callbacks cb 0x7f7fb4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fbe553640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fbe553640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40086/0 (socket says 172.21.15.79:40086) 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.833+0000 7f7fbe553640 1 -- 172.21.15.79:0/2521123527 learned_addr learned my addr 172.21.15.79:0/2521123527 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fbe553640 1 -- 172.21.15.79:0/2521123527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 msgr2=0x56303b9958f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b9958f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fbe553640 1 -- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 msgr2=0x56303b995e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 0x56303b995e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fbe553640 1 -- 172.21.15.79:0/2521123527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56303b99ad80 con 0x56303b986b60 2024-04-22T06:48:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fbe553640 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9953b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000bf50 tx=0x7f7fb000d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e972e31745878e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2521123527 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb002e070 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2521123527 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7f7fb0004220 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb0015040 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2521123527 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7fb0014040 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fc2bab740 10 client.16719 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56303ba0b0e0 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2521123527 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0015040 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2521123527 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7f7fb0004bb0 con 0x56303b986b60 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.834+0000 7f7fc2bab740 10 client.16719 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.835+0000 7f7fc2bab740 10 client.16719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.835+0000 7f7fc2bab740 20 client.16719 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.835+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303ba0b580 0x56303ba0d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.835+0000 7f7fc2bab740 4 client.16719 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.835+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56303b88b3d0 con 0x56303ba0b580 2024-04-22T06:48:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.835+0000 7f7fbdd52640 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303ba0b580 0x56303ba0d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.837+0000 7f7fa17fa640 10 client.16719 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.837+0000 7f7fbdd52640 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303ba0b580 0x56303ba0d9a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c048ded0df1fdf5 server_cookie=8950d3068d5babc6 in_seq=0 out_seq=0 2024-04-22T06:48:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.851+0000 7f7fa17fa640 1 -- 172.21.15.79:0/2521123527 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56303b88b3d0 con 0x56303ba0b580 2024-04-22T06:48:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.851+0000 7f7fa17fa640 10 client.16719 handle_command_reply: tid=0 2024-04-22T06:48:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.851+0000 7f7fc2bab740 1 client.16719 shutdown 2024-04-22T06:48:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.851+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 msgr2=0x56303b9953b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.851+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9953b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000bf50 tx=0x7f7fb000d830 comp rx=0 tx=0).stop 2024-04-22T06:48:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 shutdown_connections 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56303ba0b580 0x56303ba0d9a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56303b9944f0 0x56303b995e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56303b9878d0 0x56303b9958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 --2- 172.21.15.79:0/2521123527 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56303b986b60 0x56303b9953b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fa17fa640 0 client.16719 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 >> 172.21.15.79:0/2521123527 conn(0x56303b94dc90 msgr2=0x56303b980040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 shutdown_connections 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 1 -- 172.21.15.79:0/2521123527 wait complete. 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fc2bab740 20 client.16719 trim_cache size 0 max 16384 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:48:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:48:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:48:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:48:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7fa40a3dc0 msgr2=0x7f7fa40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7fa40a3dc0 0x7f7fa40a6280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80cdbc0 tx=0x7f7fa801eb80 comp rx=0 tx=0).stop 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 msgr2=0x7f7fb8198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 0x7f7fb8198ba0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800b450 tx=0x7f7fa800b910 comp rx=0 tx=0).stop 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 shutdown_connections 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fb8105e80 0x7f7fb819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb8101f20 0x7f7fb8198ba0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7fa40a3dc0 0x7f7fa40a6280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 --2- 172.21.15.79:0/1194805737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fb8101560 0x7f7fb8198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 >> 172.21.15.79:0/1194805737 conn(0x7f7fb8077370 msgr2=0x7f7fb80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 shutdown_connections 2024-04-22T06:48:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:13.852+0000 7f7fbffb6640 1 -- 172.21.15.79:0/1194805737 wait complete. 2024-04-22T06:48:14.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:14 smithi193 ceph-mon[31815]: pgmap v121: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 86 MiB/s wr, 171 op/s 2024-04-22T06:48:14.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:14 smithi190 ceph-mon[31421]: pgmap v121: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 86 MiB/s wr, 171 op/s 2024-04-22T06:48:14.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:14 smithi079 ceph-mon[27946]: pgmap v121: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 86 MiB/s wr, 171 op/s 2024-04-22T06:48:16.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:16 smithi193 ceph-mon[31815]: pgmap v122: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 71 MiB/s wr, 124 op/s 2024-04-22T06:48:16.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:16 smithi190 ceph-mon[31421]: pgmap v122: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 71 MiB/s wr, 124 op/s 2024-04-22T06:48:16.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:16 smithi079 ceph-mon[27946]: pgmap v122: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 71 MiB/s wr, 124 op/s 2024-04-22T06:48:18.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:18 smithi193 ceph-mon[31815]: pgmap v123: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 103 MiB/s wr, 179 op/s 2024-04-22T06:48:18.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:18 smithi190 ceph-mon[31421]: pgmap v123: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 103 MiB/s wr, 179 op/s 2024-04-22T06:48:18.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:18 smithi079 ceph-mon[27946]: pgmap v123: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 103 MiB/s wr, 179 op/s 2024-04-22T06:48:20.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:20 smithi193 ceph-mon[31815]: pgmap v124: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 67 MiB/s wr, 113 op/s 2024-04-22T06:48:20.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:20 smithi190 ceph-mon[31421]: pgmap v124: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 67 MiB/s wr, 113 op/s 2024-04-22T06:48:20.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:20 smithi079 ceph-mon[27946]: pgmap v124: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 67 MiB/s wr, 113 op/s 2024-04-22T06:48:21.401 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:21.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 -- 172.21.15.79:0/3835876227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 msgr2=0x7fc82c10dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 --2- 172.21.15.79:0/3835876227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c10dca0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc824009ca0 tx=0x7fc82402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 -- 172.21.15.79:0/3835876227 shutdown_connections 2024-04-22T06:48:21.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 --2- 172.21.15.79:0/3835876227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc82c101130 0x7fc82c101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 --2- 172.21.15.79:0/3835876227 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c10dca0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 --2- 172.21.15.79:0/3835876227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc82c106f00 0x7fc82c1072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 -- 172.21.15.79:0/3835876227 >> 172.21.15.79:0/3835876227 conn(0x7fc82c077e30 msgr2=0x7fc82c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 -- 172.21.15.79:0/3835876227 shutdown_connections 2024-04-22T06:48:21.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.500+0000 7fc832a4e640 1 -- 172.21.15.79:0/3835876227 wait complete. 2024-04-22T06:48:21.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 Processor -- start 2024-04-22T06:48:21.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 -- start start 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc82c101130 0x7fc82c199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc82c106f00 0x7fc82c1a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c1110e0 con 0x7fc82c101ad0 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c110f60 con 0x7fc82c106f00 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc832a4e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c111260 con 0x7fc82c101130 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc82b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc82b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40102/0 (socket says 172.21.15.79:40102) 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc82b7fe640 1 -- 172.21.15.79:0/2963918553 learned_addr learned my addr 172.21.15.79:0/2963918553 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc830feb640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc82c106f00 0x7fc82c1a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.501+0000 7fc82bfff640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc82c101130 0x7fc82c199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc82b7fe640 1 -- 172.21.15.79:0/2963918553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc82c101130 msgr2=0x7fc82c199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc82b7fe640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc82c101130 0x7fc82c199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc82b7fe640 1 -- 172.21.15.79:0/2963918553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc82c106f00 msgr2=0x7fc82c1a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc82b7fe640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc82c106f00 0x7fc82c1a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc82b7fe640 1 -- 172.21.15.79:0/2963918553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c1a4b00 con 0x7fc82c101ad0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc82b7fe640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c199a80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc82000d950 tx=0x7fc82000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820017070 con 0x7fc82c101ad0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc820009c30 con 0x7fc82c101ad0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82c1a4cd0 con 0x7fc82c101ad0 2024-04-22T06:48:21.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82000f620 con 0x7fc82c101ad0 2024-04-22T06:48:21.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.502+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc82c1a51d0 con 0x7fc82c101ad0 2024-04-22T06:48:21.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.503+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c069230 con 0x7fc82c101ad0 2024-04-22T06:48:21.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.504+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc820010460 con 0x7fc82c101ad0 2024-04-22T06:48:21.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.505+0000 7fc8297fa640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc8140a3d20 0x7fc8140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.505+0000 7fc82bfff640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc8140a3d20 0x7fc8140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.505+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc8200d0540 con 0x7fc82c101ad0 2024-04-22T06:48:21.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.505+0000 7fc82bfff640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc8140a3d20 0x7fc8140a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc81c005d70 tx=0x7fc81c005d00 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.509+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc82009cac0 con 0x7fc82c101ad0 2024-04-22T06:48:21.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.722+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc82c109360 con 0x7fc82c101ad0 2024-04-22T06:48:21.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.723+0000 7fc8297fa640 1 -- 172.21.15.79:0/2963918553 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7fc820009de0 con 0x7fc82c101ad0 2024-04-22T06:48:21.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:48:21.722 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:46:45.384403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25265,"mds_2":25261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":35,"state":"up:active","state_seq":22,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25261,24467,25265]},"id":1}]} 2024-04-22T06:48:21.722 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 40 2024-04-22T06:48:21.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc8140a3d20 msgr2=0x7fc8140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc8140a3d20 0x7fc8140a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc81c005d70 tx=0x7fc81c005d00 comp rx=0 tx=0).stop 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 msgr2=0x7fc82c199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c199a80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fc82000d950 tx=0x7fc82000de10 comp rx=0 tx=0).stop 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 shutdown_connections 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc8140a3d20 0x7fc8140a61e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc82c106f00 0x7fc82c1a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc82c101ad0 0x7fc82c199a80 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 --2- 172.21.15.79:0/2963918553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc82c101130 0x7fc82c199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 >> 172.21.15.79:0/2963918553 conn(0x7fc82c077e30 msgr2=0x7fc82c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 shutdown_connections 2024-04-22T06:48:21.724 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.725+0000 7fc832a4e640 1 -- 172.21.15.79:0/2963918553 wait complete. 2024-04-22T06:48:21.735 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id a5377597 --release --await' 2024-04-22T06:48:21.735 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id a5377597 --release --await 2024-04-22T06:48:21.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 -- 172.21.15.79:0/2577976000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 msgr2=0x7efe3010dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 --2- 172.21.15.79:0/2577976000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe3010dc80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7efe28009cd0 tx=0x7efe2802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 -- 172.21.15.79:0/2577976000 shutdown_connections 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 --2- 172.21.15.79:0/2577976000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe301011a0 0x7efe30101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 --2- 172.21.15.79:0/2577976000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe3010dc80 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 --2- 172.21.15.79:0/2577976000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 0x7efe30107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 -- 172.21.15.79:0/2577976000 >> 172.21.15.79:0/2577976000 conn(0x7efe30076c50 msgr2=0x7efe30077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 -- 172.21.15.79:0/2577976000 shutdown_connections 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 -- 172.21.15.79:0/2577976000 wait complete. 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 Processor -- start 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.842+0000 7efe37ce0640 1 -- start start 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe301011a0 0x7efe30197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe30197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 0x7efe3019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efe301110c0 con 0x7efe30101b40 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efe30110f40 con 0x7efe301011a0 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe30111240 con 0x7efe30107030 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe30197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe30197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40116/0 (socket says 172.21.15.79:40116) 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 -- 172.21.15.79:0/3821241929 learned_addr learned my addr 172.21.15.79:0/3821241929 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3627d640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 0x7efe3019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe35a7c640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe301011a0 0x7efe30197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 -- 172.21.15.79:0/3821241929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 msgr2=0x7efe3019bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 0x7efe3019bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 -- 172.21.15.79:0/3821241929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe301011a0 msgr2=0x7efe30197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe301011a0 0x7efe30197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 -- 172.21.15.79:0/3821241929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3019c630 con 0x7efe30101b40 2024-04-22T06:48:21.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3627d640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 0x7efe3019bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:21.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe3527b640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe30197b80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7efe2000c920 tx=0x7efe2000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe26ffd640 1 -- 172.21.15.79:0/3821241929 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20007c70 con 0x7efe30101b40 2024-04-22T06:48:21.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe26ffd640 1 -- 172.21.15.79:0/3821241929 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe20007e10 con 0x7efe30101b40 2024-04-22T06:48:21.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3019c920 con 0x7efe30101b40 2024-04-22T06:48:21.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe26ffd640 1 -- 172.21.15.79:0/3821241929 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20018820 con 0x7efe30101b40 2024-04-22T06:48:21.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.843+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe301a41b0 con 0x7efe30101b40 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.845+0000 7efe26ffd640 1 -- 172.21.15.79:0/3821241929 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efe2000f450 con 0x7efe30101b40 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 Processor -- start 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 -- start start 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe26ffd640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efe1c0a3da0 0x7efe1c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d16a0 0x558f130dc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 0x558f130dee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558f13036500 con 0x558f130dca20 2024-04-22T06:48:21.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558f12fdc6a0 con 0x558f130d16a0 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3a9af740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f130324c0 con 0x558f130d0cf0 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.846+0000 7efe3627d640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efe1c0a3da0 0x7efe1c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe26ffd640 1 -- 172.21.15.79:0/3821241929 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7efe200d0aa0 con 0x7efe30101b40 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 0x558f130dee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 0x558f130dee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40128/0 (socket says 172.21.15.79:40128) 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 -- 172.21.15.79:0/2292860524 learned_addr learned my addr 172.21.15.79:0/2292860524 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe35a7c640 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3527b640 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d16a0 0x558f130dc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 -- 172.21.15.79:0/2292860524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 msgr2=0x558f130d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 -- 172.21.15.79:0/2292860524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d16a0 msgr2=0x558f130dc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d16a0 0x558f130dc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 -- 172.21.15.79:0/2292860524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f12ffb5e0 con 0x558f130dca20 2024-04-22T06:48:21.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 0x558f130dee10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7efe28009ca0 tx=0x7efe28032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21eba15961d7e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe3627d640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efe1c0a3da0 0x7efe1c0a6260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efe28009cd0 tx=0x7efe28039000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe1bfff640 1 -- 172.21.15.79:0/2292860524 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28051070 con 0x558f130dca20 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe1bfff640 1 -- 172.21.15.79:0/2292860524 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe2803a040 con 0x558f130dca20 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.847+0000 7efe1bfff640 1 -- 172.21.15.79:0/2292860524 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28039070 con 0x558f130dca20 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 -- 172.21.15.79:0/2292860524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 msgr2=0x558f130dee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 0x558f130dee10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7efe28009ca0 tx=0x7efe28032fc0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 -- 172.21.15.79:0/2292860524 shutdown_connections 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d16a0 0x558f130dc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130dca20 0x558f130dee10 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2292860524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 -- 172.21.15.79:0/2292860524 >> 172.21.15.79:0/2292860524 conn(0x558f130cba40 msgr2=0x558f130cbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 -- 172.21.15.79:0/2292860524 shutdown_connections 2024-04-22T06:48:21.854 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 -- 172.21.15.79:0/2292860524 wait complete. 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.848+0000 7efe3a9af740 1 -- start start 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 Processor -- start 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d0cf0 0x558f130dc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130d9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130dca20 0x558f130d9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558f130df780 con 0x558f130d16a0 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558f12fefa60 con 0x558f130d0cf0 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f130df900 con 0x558f130dca20 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3a9af740 10 client.0 _ll_register_callbacks cb 0x558f12d48110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130d9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130d9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40132/0 (socket says 172.21.15.79:40132) 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 -- 172.21.15.79:0/3545952670 learned_addr learned my addr 172.21.15.79:0/3545952670 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe35a7c640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d0cf0 0x558f130dc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3627d640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130dca20 0x558f130d9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 -- 172.21.15.79:0/3545952670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130dca20 msgr2=0x558f130d9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130dca20 0x558f130d9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 -- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d0cf0 msgr2=0x558f130dc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d0cf0 0x558f130dc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 -- 172.21.15.79:0/3545952670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f12d48920 con 0x558f130d16a0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe3527b640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130d9820 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efe2009c620 tx=0x7efe2009c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c542e3324079b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.849+0000 7efe197fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe197fa640 1 -- 172.21.15.79:0/3545952670 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe200a0cd0 con 0x558f130d16a0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe197fa640 1 -- 172.21.15.79:0/3545952670 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7efe2000f740 con 0x558f130d16a0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe2803a040 con 0x558f130d16a0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe197fa640 1 -- 172.21.15.79:0/3545952670 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efe200a5590 con 0x558f130d16a0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 10 client.16737 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558f12e14d30 con 0x558f130d16a0 2024-04-22T06:48:21.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe197fa640 1 -- 172.21.15.79:0/3545952670 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe2803a040 con 0x558f130d16a0 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe197fa640 1 -- 172.21.15.79:0/3545952670 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7efe200a5100 con 0x558f130d16a0 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 10 client.16737 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 10 client.16737 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 20 client.16737 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f130ecb20 0x558f130eef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 4 client.16737 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f13009a10 con 0x558f130ecb20 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.850+0000 7efe35a7c640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f130ecb20 0x558f130eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.851+0000 7efe197fa640 10 client.16737 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.851+0000 7efe35a7c640 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f130ecb20 0x558f130eef10 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92ebd4ea2621abe3 server_cookie=f73544b5b792ce52 in_seq=0 out_seq=0 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.851+0000 7efe197fa640 1 -- 172.21.15.79:0/3545952670 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f13009a10 con 0x558f130ecb20 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.851+0000 7efe197fa640 10 client.16737 handle_command_reply: tid=0 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 client.16737 shutdown 2024-04-22T06:48:21.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 msgr2=0x558f130d9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130d9820 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efe2009c620 tx=0x7efe2009c7f0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 shutdown_connections 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f130ecb20 0x558f130eef10 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130dca20 0x558f130d9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130d9820 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe197fa640 0 client.16737 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 --2- 172.21.15.79:0/3545952670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130d0cf0 0x558f130dc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 >> 172.21.15.79:0/3545952670 conn(0x558f130cba40 msgr2=0x558f130cd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 shutdown_connections 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 1 -- 172.21.15.79:0/3545952670 wait complete. 2024-04-22T06:48:21.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.852+0000 7efe3a9af740 20 client.16737 trim_cache size 0 max 16384 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 Processor -- start 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 -- start start 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 0x558f130cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d16a0 0x558f130dbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130dc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efe301ae420 con 0x558f130d0cf0 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efe301abcf0 con 0x558f130dca20 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3a9af740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe301a9b40 con 0x558f130d16a0 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe35a7c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 0x558f130cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe35a7c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 0x558f130cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40142/0 (socket says 172.21.15.79:40142) 2024-04-22T06:48:21.873 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe35a7c640 1 -- 172.21.15.79:0/2707621860 learned_addr learned my addr 172.21.15.79:0/2707621860 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3627d640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130dc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe3527b640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d16a0 0x558f130dbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.869+0000 7efe35a7c640 1 -- 172.21.15.79:0/2707621860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d16a0 msgr2=0x558f130dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe35a7c640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d16a0 0x558f130dbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe35a7c640 1 -- 172.21.15.79:0/2707621860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 msgr2=0x558f130dc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe35a7c640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130dc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe35a7c640 1 -- 172.21.15.79:0/2707621860 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f130ef280 con 0x558f130d0cf0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3527b640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d16a0 0x558f130dbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3627d640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130dc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe35a7c640 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 0x558f130cd5a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efe2c005bc0 tx=0x7efe2c00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734d0d49c21473e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe1b7fe640 1 -- 172.21.15.79:0/2707621860 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c003b20 con 0x558f130d0cf0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe1b7fe640 1 -- 172.21.15.79:0/2707621860 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe2c01d8f0 con 0x558f130d0cf0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe1b7fe640 1 -- 172.21.15.79:0/2707621860 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c0047a0 con 0x558f130d0cf0 2024-04-22T06:48:21.874 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 -- 172.21.15.79:0/2707621860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 msgr2=0x558f130cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 0x558f130cd5a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efe2c005bc0 tx=0x7efe2c00dc40 comp rx=0 tx=0).stop 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 -- 172.21.15.79:0/2707621860 shutdown_connections 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130dc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d16a0 0x558f130dbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 --2- 172.21.15.79:0/2707621860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d0cf0 0x558f130cd5a0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.870+0000 7efe3a9af740 1 -- 172.21.15.79:0/2707621860 >> 172.21.15.79:0/2707621860 conn(0x558f130cba40 msgr2=0x558f130cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 -- 172.21.15.79:0/2707621860 shutdown_connections 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 -- 172.21.15.79:0/2707621860 wait complete. 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 -- start start 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 Processor -- start 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130dfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130e00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efe301aa1b0 con 0x558f130d16a0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efe301ab620 con 0x558f130dca20 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe301aa880 con 0x558f130d0cf0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3a9af740 10 client.0 _ll_register_callbacks cb 0x7efe2009c280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3527b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130dfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3527b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130dfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40156/0 (socket says 172.21.15.79:40156) 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe3527b640 1 -- 172.21.15.79:0/1279807066 learned_addr learned my addr 172.21.15.79:0/1279807066 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.871+0000 7efe35a7c640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3527b640 1 -- 172.21.15.79:0/1279807066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 msgr2=0x558f130e4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3527b640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130e4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3527b640 1 -- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 msgr2=0x558f130e00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3627d640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130e00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3527b640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130e00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3527b640 1 -- 172.21.15.79:0/1279807066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f130e0600 con 0x558f130d16a0 2024-04-22T06:48:21.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3527b640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130dfb80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efe200992f0 tx=0x7efe2000fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffbd190fb0b2cbe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe197fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe197fa640 1 -- 172.21.15.79:0/1279807066 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe200a0cd0 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe197fa640 1 -- 172.21.15.79:0/1279807066 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7efe200bbbb0 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe2c01d8f0 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe197fa640 1 -- 172.21.15.79:0/1279807066 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efe200bbdb0 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 10 client.16743 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558f13153d40 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe197fa640 1 -- 172.21.15.79:0/1279807066 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe2c01d8f0 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe197fa640 1 -- 172.21.15.79:0/1279807066 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7efe20097070 con 0x558f130d16a0 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 10 client.16743 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 10 client.16743 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 20 client.16743 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f131544c0 0x558f131568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 4 client.16743 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a5377597", "release": true, "await": true}] 2024-04-22T06:48:21.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.872+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a5377597", "release": true, "await": true}) -- 0x558f13043f20 con 0x558f131544c0 2024-04-22T06:48:21.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.873+0000 7efe35a7c640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f131544c0 0x558f131568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.873+0000 7efe35a7c640 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f131544c0 0x558f131568e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=338bda4df723bcb8 server_cookie=43286c7619310798 in_seq=0 out_seq=0 2024-04-22T06:48:21.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.873+0000 7efe197fa640 10 client.16743 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.876+0000 7efe197fa640 1 -- 172.21.15.79:0/1279807066 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x558f13043f20 con 0x558f131544c0 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.876+0000 7efe197fa640 10 client.16743 handle_command_reply: tid=0 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.876+0000 7efe3a9af740 1 client.16743 shutdown 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 msgr2=0x558f130dfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130dfb80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efe200992f0 tx=0x7efe2000fe60 comp rx=0 tx=0).stop 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 shutdown_connections 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558f131544c0 0x558f131568e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558f130dca20 0x558f130e00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe197fa640 0 client.16743 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:21.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558f130d16a0 0x558f130dfb80 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 --2- 172.21.15.79:0/1279807066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f130d0cf0 0x558f130e4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 >> 172.21.15.79:0/1279807066 conn(0x558f130cba40 msgr2=0x558f130cab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 shutdown_connections 2024-04-22T06:48:21.880 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 1 -- 172.21.15.79:0/1279807066 wait complete. 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.877+0000 7efe3a9af740 20 client.16743 trim_cache size 0 max 16384 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 40, 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 14, 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "a5377597": { 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 14, 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:48:21.881 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 86.2, 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:48:21.882 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:48:21.883 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:48:21.883 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:48:21.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efe1c0a3da0 msgr2=0x7efe1c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efe1c0a3da0 0x7efe1c0a6260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efe28009cd0 tx=0x7efe28039000 comp rx=0 tx=0).stop 2024-04-22T06:48:21.883 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 msgr2=0x7efe30197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe30197b80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7efe2000c920 tx=0x7efe2000cde0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 shutdown_connections 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30107030 0x7efe3019bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efe30101b40 0x7efe30197b80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efe1c0a3da0 0x7efe1c0a6260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 --2- 172.21.15.79:0/3821241929 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe301011a0 0x7efe30197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 >> 172.21.15.79:0/3821241929 conn(0x7efe30076c50 msgr2=0x7efe300ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 shutdown_connections 2024-04-22T06:48:21.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:21.878+0000 7efe37ce0640 1 -- 172.21.15.79:0/3821241929 wait complete. 2024-04-22T06:48:21.897 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: a5377597, seconds elapsed: 29.7 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7bd9e92700 time 2024-04-22T06:48:21.874984+0000 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7bed4cd3dd] 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.165 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: clone() 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:21.874+0000 7f7bd9e92700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7bd9e92700 time 2024-04-22T06:48:21.874984+0000 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7bed4cd3dd] 2024-04-22T06:48:22.166 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: clone() 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: *** Caught signal (Aborted) ** 2024-04-22T06:48:22.168 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: in thread 7f7bd9e92700 thread_name:quiesce.agt 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7bec255d20] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: gsignal() 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: abort() 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7bed4cd437] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.169 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 13: clone() 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:21.876+0000 7f7bd9e92700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: in thread 7f7bd9e92700 thread_name:quiesce.agt 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7bec255d20] 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: gsignal() 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: abort() 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7bed4cd437] 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.170 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 13: clone() 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: -1> 2024-04-22T06:48:21.874+0000 7f7bd9e92700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7bd9e92700 time 2024-04-22T06:48:21.874984+0000 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7bed4cd3dd] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.171 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: clone() 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 0> 2024-04-22T06:48:21.876+0000 7f7bd9e92700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: in thread 7f7bd9e92700 thread_name:quiesce.agt 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7bec255d20] 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: gsignal() 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: abort() 2024-04-22T06:48:22.172 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7bed4cd437] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 13: clone() 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f12ca7d4700 time 2024-04-22T06:48:21.874919+0000 2024-04-22T06:48:22.173 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f12dde0f3dd] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: clone() 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:21.875+0000 7f12ca7d4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f12ca7d4700 time 2024-04-22T06:48:21.874919+0000 2024-04-22T06:48:22.174 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f12dde0f3dd] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: clone() 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.175 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: *** Caught signal (Aborted) ** 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: in thread 7f12ca7d4700 thread_name:quiesce.agt 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12dcb97d20] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: gsignal() 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: abort() 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f12dde0f437] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.176 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 13: clone() 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:21.877+0000 7f12ca7d4700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: in thread 7f12ca7d4700 thread_name:quiesce.agt 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12dcb97d20] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: gsignal() 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: abort() 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f12dde0f437] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.177 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 13: clone() 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: -2> 2024-04-22T06:48:21.875+0000 7f12ca7d4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f12ca7d4700 time 2024-04-22T06:48:21.874919+0000 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f12dde0f3dd] 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.178 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: clone() 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 0> 2024-04-22T06:48:21.877+0000 7f12ca7d4700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: in thread 7f12ca7d4700 thread_name:quiesce.agt 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12dcb97d20] 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: gsignal() 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: abort() 2024-04-22T06:48:22.179 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f12dde0f437] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 13: clone() 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:22.180 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.181 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: -2> 2024-04-22T06:48:21.875+0000 7f12ca7d4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f12ca7d4700 time 2024-04-22T06:48:21.874919+0000 2024-04-22T06:48:22.181 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.181 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.181 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.181 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f12dde0f3dd] 2024-04-22T06:48:22.181 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: clone() 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 0> 2024-04-22T06:48:21.877+0000 7f12ca7d4700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: in thread 7f12ca7d4700 thread_name:quiesce.agt 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.182 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f12dcb97d20] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2: gsignal() 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 3: abort() 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f12dde0f437] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f12dde0f5a3] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56199a83be9c] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56199a83bf9f] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56199a76a0d3] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56199a76a03c] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56199a9e9d3f] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56199aa52560] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12dcb8d1ca] 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 13: clone() 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:22.183 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[65460]: 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:21 smithi190 podman[65987]: 2024-04-22 06:48:21.936364839 +0000 UTC m=+0.023930112 container died 5436cf65340ec100dfc76cefc7948e0edf921cbeb5b89cb3759591b34ce9ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: -2> 2024-04-22T06:48:21.874+0000 7f7bd9e92700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7bd9e92700 time 2024-04-22T06:48:21.874984+0000 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7bed4cd3dd] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.184 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: clone() 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: -1> 2024-04-22T06:48:21.876+0000 7f7bd9e92700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: in thread 7f7bd9e92700 thread_name:quiesce.agt 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7bec255d20] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2: gsignal() 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 3: abort() 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7bed4cd437] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7bed4cd5a3] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56256e354e9c] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56256e354f9f] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56256e2830d3] 2024-04-22T06:48:22.185 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56256e28303c] 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56256e502d3f] 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56256e56b560] 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7bec24b1ca] 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 13: clone() 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[66383]: 2024-04-22T06:48:22.186 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:21 smithi193 podman[67010]: 2024-04-22 06:48:21.940384442 +0000 UTC m=+0.025816256 container died 3de8b91a108f4e85ea2257de0be1348b3df44cd4faef4252d4c63b5ff26928b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:48:22.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:22 smithi193 ceph-mon[31815]: pgmap v125: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 79 MiB/s wr, 145 op/s 2024-04-22T06:48:22.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:22 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2963918553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:22 smithi190 ceph-mon[31421]: pgmap v125: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 79 MiB/s wr, 145 op/s 2024-04-22T06:48:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:22 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2963918553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:22 smithi079 ceph-mon[27946]: pgmap v125: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 79 MiB/s wr, 145 op/s 2024-04-22T06:48:22.492 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:22 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2963918553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:22 smithi193 podman[67010]: 2024-04-22 06:48:22.657793173 +0000 UTC m=+0.743224988 container remove 3de8b91a108f4e85ea2257de0be1348b3df44cd4faef4252d4c63b5ff26928b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:48:22.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:22 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:48:23.049 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:22 smithi190 podman[65987]: 2024-04-22 06:48:22.713361126 +0000 UTC m=+0.800926393 container remove 5436cf65340ec100dfc76cefc7948e0edf921cbeb5b89cb3759591b34ce9ec94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:48:23.050 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:22 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:48:23.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:22 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-22T06:48:23.452 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:23 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:48:24.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:24 smithi193 ceph-mon[31815]: pgmap v126: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 79 MiB/s wr, 165 op/s 2024-04-22T06:48:24.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:24 smithi190 ceph-mon[31421]: pgmap v126: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 79 MiB/s wr, 165 op/s 2024-04-22T06:48:24.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:24 smithi079 ceph-mon[27946]: pgmap v126: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 79 MiB/s wr, 165 op/s 2024-04-22T06:48:25.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:48:25.164 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:48:25.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.456+0000 7fa30929e640 1 -- 172.21.15.79:0/3430227668 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f402c680 con 0x7fa304106120 2024-04-22T06:48:25.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 -- 172.21.15.79:0/3430227668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304106120 msgr2=0x7fa3041065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/3430227668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304106120 0x7fa3041065b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009900 tx=0x7fa2f402bd40 comp rx=0 tx=0).stop 2024-04-22T06:48:25.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 -- 172.21.15.79:0/3430227668 shutdown_connections 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/3430227668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa304101640 0x7fa304105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/3430227668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304106120 0x7fa3041065b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/3430227668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa304100d20 0x7fa304101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 -- 172.21.15.79:0/3430227668 >> 172.21.15.79:0/3430227668 conn(0x7fa304077a30 msgr2=0x7fa3040fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.457+0000 7fa30b2a2640 1 -- 172.21.15.79:0/3430227668 shutdown_connections 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.458+0000 7fa30b2a2640 1 -- 172.21.15.79:0/3430227668 wait complete. 2024-04-22T06:48:25.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 Processor -- start 2024-04-22T06:48:25.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 -- start start 2024-04-22T06:48:25.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 0x7fa3040737f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa304101640 0x7fa30406c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa304106120 0x7fa30406ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041186c0 con 0x7fa304106120 2024-04-22T06:48:25.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa304118540 con 0x7fa304101640 2024-04-22T06:48:25.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.459+0000 7fa30b2a2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa304118840 con 0x7fa304100d20 2024-04-22T06:48:25.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.460+0000 7fa30a2a0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 0x7fa3040737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.460+0000 7fa30a2a0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 0x7fa3040737f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:42678/0 (socket says 172.21.15.79:42678) 2024-04-22T06:48:25.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.460+0000 7fa30a2a0640 1 -- 172.21.15.79:0/508126615 learned_addr learned my addr 172.21.15.79:0/508126615 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:25.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.460+0000 7fa309a9f640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa304101640 0x7fa30406c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.460+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa304106120 0x7fa30406ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.461+0000 7fa30a2a0640 1 -- 172.21.15.79:0/508126615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa304101640 msgr2=0x7fa30406c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.461+0000 7fa30a2a0640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa304101640 0x7fa30406c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.461+0000 7fa30a2a0640 1 -- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa304106120 msgr2=0x7fa30406ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.461+0000 7fa30a2a0640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa304106120 0x7fa30406ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.461+0000 7fa30a2a0640 1 -- 172.21.15.79:0/508126615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30406d470 con 0x7fa304100d20 2024-04-22T06:48:25.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.461+0000 7fa30a2a0640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 0x7fa3040737f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00ba50 tx=0x7fa2fc00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.462+0000 7fa2fb7fe640 1 -- 172.21.15.79:0/508126615 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc019070 con 0x7fa304100d20 2024-04-22T06:48:25.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.462+0000 7fa2fb7fe640 1 -- 172.21.15.79:0/508126615 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2fc014440 con 0x7fa304100d20 2024-04-22T06:48:25.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.462+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa304071310 con 0x7fa304100d20 2024-04-22T06:48:25.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.462+0000 7fa2fb7fe640 1 -- 172.21.15.79:0/508126615 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0075b0 con 0x7fa304100d20 2024-04-22T06:48:25.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.462+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3040718b0 con 0x7fa304100d20 2024-04-22T06:48:25.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.466+0000 7fa30f1d4740 1 Processor -- start 2024-04-22T06:48:25.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.466+0000 7fa30f1d4740 1 -- start start 2024-04-22T06:48:25.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f150faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1510030 0x5597f15186d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f150faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f150faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47398/0 (socket says 172.21.15.79:47398) 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2464872478 learned_addr learned my addr 172.21.15.79:0/2464872478 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa2fb7fe640 1 -- 172.21.15.79:0/508126615 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa2fc01c070 con 0x7fa304100d20 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f151b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5597f14741e0 con 0x5597f150f710 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5597f1411390 con 0x5597f1518c10 2024-04-22T06:48:25.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.467+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5597f14701a0 con 0x5597f1510030 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2464872478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1510030 msgr2=0x5597f15186d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1510030 0x5597f15186d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2464872478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 msgr2=0x5597f151b000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f151b000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa2fb7fe640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa2f00a3e20 0x7fa2f00a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2464872478 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597f12507d0 con 0x5597f150f710 2024-04-22T06:48:25.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30a2a0640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa2f00a3e20 0x7fa2f00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa2fb7fe640 1 -- 172.21.15.79:0/508126615 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fa2fc0cfd20 con 0x7fa304100d20 2024-04-22T06:48:25.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.468+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f150faf0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa2f402c350 tx=0x7fa2f40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a276fc66d9901cf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.469+0000 7fa30a2a0640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa2f00a3e20 0x7fa2f00a62e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x5597f150f1e0 tx=0x7fa2fc0bc040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.470+0000 7fa2f8bfe640 1 -- 172.21.15.79:0/2464872478 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4043070 con 0x5597f150f710 2024-04-22T06:48:25.468 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.470+0000 7fa2f8bfe640 1 -- 172.21.15.79:0/2464872478 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2f402cb30 con 0x5597f150f710 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa2f8bfe640 1 -- 172.21.15.79:0/2464872478 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f403fb00 con 0x5597f150f710 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 msgr2=0x5597f150faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f150faf0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa2f402c350 tx=0x7fa2f40047d0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 shutdown_connections 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f151b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1510030 0x5597f15186d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2464872478 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f150faf0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.471+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 >> 172.21.15.79:0/2464872478 conn(0x5597f1509e40 msgr2=0x5597f150a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.472+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 shutdown_connections 2024-04-22T06:48:25.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.472+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2464872478 wait complete. 2024-04-22T06:48:25.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.472+0000 7fa30f1d4740 1 -- start start 2024-04-22T06:48:25.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.473+0000 7fa30f1d4740 1 Processor -- start 2024-04-22T06:48:25.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.473+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f150f710 0x5597f1515c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.473+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 0x5597f15161b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f15166f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5597f151b9f0 con 0x5597f1510030 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 0x5597f15161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 0x5597f15161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47412/0 (socket says 172.21.15.79:47412) 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 -- 172.21.15.79:0/893024812 learned_addr learned my addr 172.21.15.79:0/893024812 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5597f142d2f0 con 0x5597f150f710 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5597f151bb70 con 0x5597f1518c10 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f15166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa30f1d4740 10 client.0 _ll_register_callbacks cb 0x5597f143a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 -- 172.21.15.79:0/893024812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 msgr2=0x5597f15166f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f15166f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 -- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f150f710 msgr2=0x5597f1515c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f150f710 0x5597f1515c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.474+0000 7fa309a9f640 1 -- 172.21.15.79:0/893024812 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597f11863f0 con 0x5597f1510030 2024-04-22T06:48:25.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.475+0000 7fa309a9f640 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 0x5597f15161b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e450 tx=0x7fa30000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ce9e70f2d15e8d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.475+0000 7fa2edffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:25.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.475+0000 7fa2edffb640 1 -- 172.21.15.79:0/893024812 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300017070 con 0x5597f1510030 2024-04-22T06:48:25.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.475+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2f402cb30 con 0x5597f1510030 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.476+0000 7fa2edffb640 1 -- 172.21.15.79:0/893024812 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fa300013680 con 0x5597f1510030 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.476+0000 7fa2edffb640 1 -- 172.21.15.79:0/893024812 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa300021050 con 0x5597f1510030 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.476+0000 7fa30f1d4740 10 client.16752 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.476+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5597f11460f0 con 0x5597f1510030 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.476+0000 7fa2edffb640 1 -- 172.21.15.79:0/893024812 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2f402cb30 con 0x5597f1510030 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa2edffb640 1 -- 172.21.15.79:0/893024812 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fa300024020 con 0x5597f1510030 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa30f1d4740 10 client.16752 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa30f1d4740 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa30f1d4740 20 client.16752 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f15290e0 0x5597f152b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa30f1d4740 4 client.16752 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:25.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.477+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597f1481c00 con 0x5597f15290e0 2024-04-22T06:48:25.477 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.479+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f15290e0 0x5597f152b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.487+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f15290e0 0x5597f152b5a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217d5750d1ea146b server_cookie=52ab418b080c6e93 in_seq=0 out_seq=0 2024-04-22T06:48:25.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.487+0000 7fa2edffb640 10 client.16752 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:25.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.495+0000 7fa2edffb640 1 -- 172.21.15.79:0/893024812 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597f1481c00 con 0x5597f15290e0 2024-04-22T06:48:25.494 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.495+0000 7fa2edffb640 10 client.16752 handle_command_reply: tid=0 2024-04-22T06:48:25.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.499+0000 7fa30f1d4740 1 client.16752 shutdown 2024-04-22T06:48:25.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 msgr2=0x5597f15161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 0x5597f15161b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e450 tx=0x7fa30000e910 comp rx=0 tx=0).stop 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 shutdown_connections 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f15290e0 0x5597f152b5a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f15166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1510030 0x5597f15161b0 secure :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e450 tx=0x7fa30000e910 comp rx=0 tx=0).stop 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/893024812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f150f710 0x5597f1515c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.500+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 >> 172.21.15.79:0/893024812 conn(0x5597f1509e40 msgr2=0x5597f1509120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.501+0000 7fa2edffb640 0 client.16752 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:25.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.504+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 shutdown_connections 2024-04-22T06:48:25.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.510+0000 7fa30f1d4740 1 -- 172.21.15.79:0/893024812 wait complete. 2024-04-22T06:48:25.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.510+0000 7fa30f1d4740 20 client.16752 trim_cache size 0 max 16384 2024-04-22T06:48:25.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.548+0000 7fa30f1d4740 1 Processor -- start 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 -- start start 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 0x5597f15183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f1524900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1508fa0 0x5597f1530eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041bf360 con 0x5597f1508fa0 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041bcc30 con 0x5597f1518c10 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.551+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041baa80 con 0x5597f150f710 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.552+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1508fa0 0x5597f1530eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.555+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 0x5597f15183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.555+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 0x5597f15183a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:42698/0 (socket says 172.21.15.79:42698) 2024-04-22T06:48:25.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.555+0000 7fa30aaa1640 1 -- 172.21.15.79:0/94089104 learned_addr learned my addr 172.21.15.79:0/94089104 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:25.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.556+0000 7fa309a9f640 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f1524900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.558+0000 7fa30aaa1640 1 -- 172.21.15.79:0/94089104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 msgr2=0x5597f1524900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.558+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f1524900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.558+0000 7fa30aaa1640 1 -- 172.21.15.79:0/94089104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1508fa0 msgr2=0x5597f1530eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.558+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1508fa0 0x5597f1530eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.558+0000 7fa30aaa1640 1 -- 172.21.15.79:0/94089104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597f150fe60 con 0x5597f150f710 2024-04-22T06:48:25.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.560+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 0x5597f15183a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005e90 tx=0x7fa2f40047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b41f21c7ff4be1c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.563+0000 7fa2effff640 1 -- 172.21.15.79:0/94089104 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4050070 con 0x5597f150f710 2024-04-22T06:48:25.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.563+0000 7fa2effff640 1 -- 172.21.15.79:0/94089104 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2f402cb30 con 0x5597f150f710 2024-04-22T06:48:25.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.563+0000 7fa2effff640 1 -- 172.21.15.79:0/94089104 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4043040 con 0x5597f150f710 2024-04-22T06:48:25.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.565+0000 7fa30f1d4740 1 -- 172.21.15.79:0/94089104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 msgr2=0x5597f15183a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.563 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.565+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 0x5597f15183a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005e90 tx=0x7fa2f40047d0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.566+0000 7fa30f1d4740 1 -- 172.21.15.79:0/94089104 shutdown_connections 2024-04-22T06:48:25.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.566+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1518c10 0x5597f1524900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.566+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f1508fa0 0x5597f1530eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.566+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/94089104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f150f710 0x5597f15183a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.566+0000 7fa30f1d4740 1 -- 172.21.15.79:0/94089104 >> 172.21.15.79:0/94089104 conn(0x5597f1509e40 msgr2=0x5597f1508890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.568+0000 7fa30f1d4740 1 -- 172.21.15.79:0/94089104 shutdown_connections 2024-04-22T06:48:25.566 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.568+0000 7fa30f1d4740 1 -- 172.21.15.79:0/94089104 wait complete. 2024-04-22T06:48:25.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.568+0000 7fa30f1d4740 1 -- start start 2024-04-22T06:48:25.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 Processor -- start 2024-04-22T06:48:25.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1508fa0 0x5597f151ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f151cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f151d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041bb0f0 con 0x5597f150f710 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041bc560 con 0x5597f1508fa0 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3041bb7c0 con 0x5597f1518c10 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.569+0000 7fa30f1d4740 10 client.0 _ll_register_callbacks cb 0x7fa300009400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.570+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f151d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.570+0000 7fa30aaa1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f151d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:42700/0 (socket says 172.21.15.79:42700) 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.570+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2120332303 learned_addr learned my addr 172.21.15.79:0/2120332303 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.570+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1508fa0 0x5597f151ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.571+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1508fa0 msgr2=0x5597f151ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.571+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1508fa0 0x5597f151ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.571+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2120332303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 msgr2=0x5597f151cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.571+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f151cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.571+0000 7fa30aaa1640 1 -- 172.21.15.79:0/2120332303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597f159f980 con 0x5597f1518c10 2024-04-22T06:48:25.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.572+0000 7fa30aaa1640 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f151d4c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4051340 tx=0x7fa2f4039040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c62f943fa2a0b4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.572+0000 7fa2edffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:48:25.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.573+0000 7fa2edffb640 1 -- 172.21.15.79:0/2120332303 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4043070 con 0x5597f1518c10 2024-04-22T06:48:25.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.573+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2f402cb30 con 0x5597f1518c10 2024-04-22T06:48:25.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.573+0000 7fa2edffb640 1 -- 172.21.15.79:0/2120332303 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fa2f402c880 con 0x5597f1518c10 2024-04-22T06:48:25.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.573+0000 7fa2edffb640 1 -- 172.21.15.79:0/2120332303 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fa2f4050040 con 0x5597f1518c10 2024-04-22T06:48:25.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.574+0000 7fa30f1d4740 10 client.25783 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:25.572 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.574+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5597f158e870 con 0x5597f1518c10 2024-04-22T06:48:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.575+0000 7fa2edffb640 1 -- 172.21.15.79:0/2120332303 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2f402cb30 con 0x5597f1518c10 2024-04-22T06:48:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa2edffb640 1 -- 172.21.15.79:0/2120332303 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fa2f4051490 con 0x5597f1518c10 2024-04-22T06:48:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa30f1d4740 10 client.25783 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa30f1d4740 10 client.25783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa30f1d4740 20 client.25783 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:25.574 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f158ee60 0x5597f1591280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa30f1d4740 4 client.25783 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:48:25.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.576+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597f147ba70 con 0x5597f158ee60 2024-04-22T06:48:25.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.577+0000 7fa309a9f640 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f158ee60 0x5597f1591280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.577+0000 7fa2edffb640 10 client.25783 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:25.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.578+0000 7fa309a9f640 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f158ee60 0x5597f1591280 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cb79a5d4111dd3b server_cookie=1d775e2819b05116 in_seq=0 out_seq=0 2024-04-22T06:48:25.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.580+0000 7fa2edffb640 1 -- 172.21.15.79:0/2120332303 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5597f147ba70 con 0x5597f158ee60 2024-04-22T06:48:25.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.581+0000 7fa2edffb640 10 client.25783 handle_command_reply: tid=0 2024-04-22T06:48:25.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.581+0000 7fa30f1d4740 1 client.25783 shutdown 2024-04-22T06:48:25.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.582+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 msgr2=0x5597f151d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.582+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f151d4c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4051340 tx=0x7fa2f4039040 comp rx=0 tx=0).stop 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 shutdown_connections 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5597f158ee60 0x5597f1591280 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5597f1518c10 0x5597f151d4c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5597f150f710 0x5597f151cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa30f1d4740 1 --2- 172.21.15.79:0/2120332303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5597f1508fa0 0x5597f151ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 >> 172.21.15.79:0/2120332303 conn(0x5597f1509e40 msgr2=0x5597f1515b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.584+0000 7fa2edffb640 0 client.25783 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:25.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.587+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 shutdown_connections 2024-04-22T06:48:25.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.588+0000 7fa30f1d4740 1 -- 172.21.15.79:0/2120332303 wait complete. 2024-04-22T06:48:25.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.588+0000 7fa30f1d4740 20 client.25783 trim_cache size 0 max 16384 2024-04-22T06:48:25.587 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:48:25.587 INFO:teuthology.orchestra.run.smithi079.stdout: "status": "no active scrubs running", 2024-04-22T06:48:25.587 INFO:teuthology.orchestra.run.smithi079.stdout: "scrubs": {} 2024-04-22T06:48:25.587 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.596+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa2f00a3e20 msgr2=0x7fa2f00a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.596+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa2f00a3e20 0x7fa2f00a62e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x5597f150f1e0 tx=0x7fa2fc0bc040 comp rx=0 tx=0).stop 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.596+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 msgr2=0x7fa3040737f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.596+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 0x7fa3040737f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00ba50 tx=0x7fa2fc00be50 comp rx=0 tx=0).stop 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.597+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 shutdown_connections 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.597+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa2f00a3e20 0x7fa2f00a62e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.597+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa304101640 0x7fa30406c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.597+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa304106120 0x7fa30406ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.597+0000 7fa30b2a2640 1 --2- 172.21.15.79:0/508126615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa304100d20 0x7fa3040737f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.597+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 >> 172.21.15.79:0/508126615 conn(0x7fa304077a30 msgr2=0x7fa304103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.599+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 shutdown_connections 2024-04-22T06:48:25.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.600+0000 7fa30b2a2640 1 -- 172.21.15.79:0/508126615 wait complete. 2024-04-22T06:48:25.624 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:48:25.625 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:25.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.857+0000 7ff42caa7640 1 -- 172.21.15.79:0/1795690501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428109b70 msgr2=0x7ff42811d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.858+0000 7ff42caa7640 1 --2- 172.21.15.79:0/1795690501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428109b70 0x7ff42811d7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009900 tx=0x7ff41c02bc40 comp rx=0 tx=0).stop 2024-04-22T06:48:25.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.858+0000 7ff42caa7640 1 -- 172.21.15.79:0/1795690501 shutdown_connections 2024-04-22T06:48:25.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.858+0000 7ff42caa7640 1 --2- 172.21.15.79:0/1795690501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428109b70 0x7ff42811d7a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.858+0000 7ff42caa7640 1 --2- 172.21.15.79:0/1795690501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff428110b30 0x7ff428109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.858+0000 7ff42caa7640 1 --2- 172.21.15.79:0/1795690501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4281106c0 0x7ff428108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.858+0000 7ff42caa7640 1 -- 172.21.15.79:0/1795690501 >> 172.21.15.79:0/1795690501 conn(0x7ff42806e410 msgr2=0x7ff4280719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.860+0000 7ff42caa7640 1 -- 172.21.15.79:0/1795690501 shutdown_connections 2024-04-22T06:48:25.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.863+0000 7ff42caa7640 1 -- 172.21.15.79:0/1795690501 wait complete. 2024-04-22T06:48:25.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.864+0000 7ff42caa7640 1 Processor -- start 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 -- start start 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 0x7ff4281a7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428110b30 0x7ff4281a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4281abdd0 0x7ff4281ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff428120f90 con 0x7ff4281abdd0 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff428120e10 con 0x7ff428110b30 2024-04-22T06:48:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.865+0000 7ff42caa7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff428121110 con 0x7ff4281106c0 2024-04-22T06:48:25.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.866+0000 7ff42659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 0x7ff4281a7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.866+0000 7ff42659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 0x7ff4281a7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:42720/0 (socket says 172.21.15.79:42720) 2024-04-22T06:48:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.866+0000 7ff42659c640 1 -- 172.21.15.79:0/3445861895 learned_addr learned my addr 172.21.15.79:0/3445861895 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.866+0000 7ff426d9d640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4281abdd0 0x7ff4281ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.866+0000 7ff425d9b640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428110b30 0x7ff4281a7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.867+0000 7ff42659c640 1 -- 172.21.15.79:0/3445861895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428110b30 msgr2=0x7ff4281a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.867+0000 7ff42659c640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428110b30 0x7ff4281a7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.867+0000 7ff42659c640 1 -- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4281abdd0 msgr2=0x7ff4281ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.867+0000 7ff42659c640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4281abdd0 0x7ff4281ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.867+0000 7ff42659c640 1 -- 172.21.15.79:0/3445861895 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4281ac8c0 con 0x7ff4281106c0 2024-04-22T06:48:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.868+0000 7ff42659c640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 0x7ff4281a7510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff42000ebc0 tx=0x7ff42000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.872+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420010630 con 0x7ff4281106c0 2024-04-22T06:48:25.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.872+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff420010c80 con 0x7ff4281106c0 2024-04-22T06:48:25.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.872+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420018b60 con 0x7ff4281106c0 2024-04-22T06:48:25.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.872+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4281acbb0 con 0x7ff4281106c0 2024-04-22T06:48:25.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.872+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4281111e0 con 0x7ff4281106c0 2024-04-22T06:48:25.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.879+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff420018d00 con 0x7ff4281106c0 2024-04-22T06:48:25.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.880+0000 7ff4177fe640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4100a3f20 0x7ff4100a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.880+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7ff4200d0e90 con 0x7ff4281106c0 2024-04-22T06:48:25.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.881+0000 7ff425d9b640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4100a3f20 0x7ff4100a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.879 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.881+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff428073e40 con 0x7ff4281106c0 2024-04-22T06:48:25.882 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.884+0000 7ff425d9b640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4100a3f20 0x7ff4100a63e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff428072d30 tx=0x7ff41800a2d0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.890 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:25.892+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff42009d3b0 con 0x7ff4281106c0 2024-04-22T06:48:26.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.436+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff428068de0 con 0x7ff4281106c0 2024-04-22T06:48:26.435 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.437+0000 7ff4177fe640 1 -- 172.21.15.79:0/3445861895 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11881 (secure 0 0 0) 0x7ff42009ccb0 con 0x7ff4281106c0 2024-04-22T06:48:26.436 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:48:26.436 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:46:45.384403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25265,"mds_2":25261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25261":{"gid":25261,"name":"i","rank":2,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.193:6839/1588799940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1588799940},{"type":"v1","addr":"172.21.15.193:6839","nonce":1588799940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25265":{"gid":25265,"name":"h","rank":1,"incarnation":35,"state":"up:active","state_seq":22,"addr":"172.21.15.190:6837/1635214680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1635214680},{"type":"v1","addr":"172.21.15.190:6837","nonce":1635214680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25261,24467,25265]},"id":1}]} 2024-04-22T06:48:26.436 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 40 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4100a3f20 msgr2=0x7ff4100a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4100a3f20 0x7ff4100a63e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff428072d30 tx=0x7ff41800a2d0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 msgr2=0x7ff4281a7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 0x7ff4281a7510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff42000ebc0 tx=0x7ff42000c6a0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 shutdown_connections 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4100a3f20 0x7ff4100a63e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff428110b30 0x7ff4281a7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff4281abdd0 0x7ff4281ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 --2- 172.21.15.79:0/3445861895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4281106c0 0x7ff4281a7510 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 >> 172.21.15.79:0/3445861895 conn(0x7ff42806e410 msgr2=0x7ff42806fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 shutdown_connections 2024-04-22T06:48:26.440 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.440+0000 7ff42caa7640 1 -- 172.21.15.79:0/3445861895 wait complete. 2024-04-22T06:48:26.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:26 smithi190 ceph-mon[31421]: pgmap v127: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 63 MiB/s wr, 140 op/s 2024-04-22T06:48:26.462 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:26 smithi079 ceph-mon[27946]: pgmap v127: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 63 MiB/s wr, 140 op/s 2024-04-22T06:48:26.463 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:48:26.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.650+0000 7fc721bfa640 1 -- 172.21.15.79:0/187256195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 msgr2=0x7fc71c1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.650+0000 7fc721bfa640 1 --2- 172.21.15.79:0/187256195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 0x7fc71c1098f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc710009900 tx=0x7fc71002bc80 comp rx=0 tx=0).stop 2024-04-22T06:48:26.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.651+0000 7fc721bfa640 1 -- 172.21.15.79:0/187256195 shutdown_connections 2024-04-22T06:48:26.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.651+0000 7fc721bfa640 1 --2- 172.21.15.79:0/187256195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c116f50 0x7fc71c119340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.651+0000 7fc721bfa640 1 --2- 172.21.15.79:0/187256195 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 0x7fc71c1098f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.651+0000 7fc721bfa640 1 --2- 172.21.15.79:0/187256195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.651+0000 7fc721bfa640 1 -- 172.21.15.79:0/187256195 >> 172.21.15.79:0/187256195 conn(0x7fc71c06ced0 msgr2=0x7fc71c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.652+0000 7fc721bfa640 1 -- 172.21.15.79:0/187256195 shutdown_connections 2024-04-22T06:48:26.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.652+0000 7fc721bfa640 1 -- 172.21.15.79:0/187256195 wait complete. 2024-04-22T06:48:26.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.652+0000 7fc721bfa640 1 Processor -- start 2024-04-22T06:48:26.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.653+0000 7fc721bfa640 1 -- start start 2024-04-22T06:48:26.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc721bfa640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 0x7fc71c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc721bfa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c10d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc721bfa640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c116f50 0x7fc71c10d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc721bfa640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c11cbe0 con 0x7fc71c109490 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc721bfa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c11ca60 con 0x7fc71c10c500 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc721bfa640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c11cd60 con 0x7fc71c116f50 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c10d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c10d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57288/0 (socket says 172.21.15.79:57288) 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc71bfff640 1 -- 172.21.15.79:0/1670665746 learned_addr learned my addr 172.21.15.79:0/1670665746 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc720bf8640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 0x7fc71c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc7213f9640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c116f50 0x7fc71c10d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.654+0000 7fc71bfff640 1 -- 172.21.15.79:0/1670665746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c116f50 msgr2=0x7fc71c10d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc71bfff640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c116f50 0x7fc71c10d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc71bfff640 1 -- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 msgr2=0x7fc71c10cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc71bfff640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 0x7fc71c10cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc71bfff640 1 -- 172.21.15.79:0/1670665746 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71c1a2fd0 con 0x7fc71c10c500 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc71bfff640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c10d110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc71002c290 tx=0x7fc71002c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc719ffb640 1 -- 172.21.15.79:0/1670665746 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710043020 con 0x7fc71c10c500 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc719ffb640 1 -- 172.21.15.79:0/1670665746 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71002ccb0 con 0x7fc71c10c500 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.655+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71c1a3260 con 0x7fc71c10c500 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.656+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc71c1a3800 con 0x7fc71c10c500 2024-04-22T06:48:26.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.656+0000 7fc719ffb640 1 -- 172.21.15.79:0/1670665746 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710003bc0 con 0x7fc71c10c500 2024-04-22T06:48:26.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.660+0000 7fc725a50740 1 Processor -- start 2024-04-22T06:48:26.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.661+0000 7fc725a50740 1 -- start start 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.663+0000 7fc719ffb640 1 -- 172.21.15.79:0/1670665746 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc710046020 con 0x7fc71c10c500 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.664+0000 7fc725a50740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e56d0 0x5588ae7e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.664+0000 7fc725a50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 0x5588ae7ee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.664+0000 7fc725a50740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7eec00 0x5588ae7f1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.664+0000 7fc725a50740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5588ae74a1f0 con 0x5588ae7e56d0 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.664+0000 7fc725a50740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5588ae6e74b0 con 0x5588ae7eec00 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.664+0000 7fc725a50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588ae7461b0 con 0x5588ae7e6080 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.665+0000 7fc719ffb640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7040a3e20 0x7fc7040a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 0x5588ae7ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7eec00 0x5588ae7f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 0x5588ae7ee630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:42740/0 (socket says 172.21.15.79:42740) 2024-04-22T06:48:26.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 -- 172.21.15.79:0/2208885565 learned_addr learned my addr 172.21.15.79:0/2208885565 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc7213f9640 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e56d0 0x5588ae7e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 -- 172.21.15.79:0/2208885565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7eec00 msgr2=0x5588ae7f1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7eec00 0x5588ae7f1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 -- 172.21.15.79:0/2208885565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e56d0 msgr2=0x5588ae7e5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e56d0 0x5588ae7e5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.666+0000 7fc720bf8640 1 -- 172.21.15.79:0/2208885565 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588ae6d91b0 con 0x5588ae7e6080 2024-04-22T06:48:26.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.668+0000 7fc7213f9640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7040a3e20 0x7fc7040a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.668+0000 7fc719ffb640 1 -- 172.21.15.79:0/1670665746 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc7100f2930 con 0x7fc71c10c500 2024-04-22T06:48:26.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.668+0000 7fc720bf8640 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 0x5588ae7ee630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009a40 tx=0x7fc70c009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa48a39664b808dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.669+0000 7fc7213f9640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7040a3e20 0x7fc7040a62e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc71c109f30 tx=0x7fc714004b90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.670+0000 7fc702ffd640 1 -- 172.21.15.79:0/2208885565 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c017070 con 0x5588ae7e6080 2024-04-22T06:48:26.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.670+0000 7fc702ffd640 1 -- 172.21.15.79:0/2208885565 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70c013470 con 0x5588ae7e6080 2024-04-22T06:48:26.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.670+0000 7fc702ffd640 1 -- 172.21.15.79:0/2208885565 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c012630 con 0x5588ae7e6080 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 -- 172.21.15.79:0/2208885565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 msgr2=0x5588ae7ee630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 0x5588ae7ee630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009a40 tx=0x7fc70c009e40 comp rx=0 tx=0).stop 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 -- 172.21.15.79:0/2208885565 shutdown_connections 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7eec00 0x5588ae7f1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e6080 0x5588ae7ee630 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 --2- 172.21.15.79:0/2208885565 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e56d0 0x5588ae7e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 -- 172.21.15.79:0/2208885565 >> 172.21.15.79:0/2208885565 conn(0x5588ae7dfda0 msgr2=0x5588ae7e0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.671+0000 7fc725a50740 1 -- 172.21.15.79:0/2208885565 shutdown_connections 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.672+0000 7fc725a50740 1 -- 172.21.15.79:0/2208885565 wait complete. 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.672+0000 7fc725a50740 1 -- start start 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.672+0000 7fc725a50740 1 Processor -- start 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7ee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae7eb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5588ae7f1b70 con 0x5588ae7e6080 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5588ae703500 con 0x5588ae7e56d0 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588ae7f1cf0 con 0x5588ae7eec00 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc725a50740 10 client.0 _ll_register_callbacks cb 0x5588ae6d1b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc720bf8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc720bf8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7ee510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57304/0 (socket says 172.21.15.79:57304) 2024-04-22T06:48:26.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc720bf8640 1 -- 172.21.15.79:0/60269405 learned_addr learned my addr 172.21.15.79:0/60269405 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc71bfff640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae7eb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.673+0000 7fc7213f9640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc720bf8640 1 -- 172.21.15.79:0/60269405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 msgr2=0x5588ae7ebe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc720bf8640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7ebe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc720bf8640 1 -- 172.21.15.79:0/60269405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 msgr2=0x5588ae7eb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc720bf8640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae7eb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc720bf8640 1 -- 172.21.15.79:0/60269405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588ae45c3f0 con 0x5588ae7e56d0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc720bf8640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7ee510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc70c006a50 tx=0x7fc70c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f963ef0e3c8407 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc6fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc6fbfff640 1 -- 172.21.15.79:0/60269405 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c017050 con 0x5588ae7e56d0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc70c013470 con 0x5588ae7e56d0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc6fbfff640 1 -- 172.21.15.79:0/60269405 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc70c013ab0 con 0x5588ae7e56d0 2024-04-22T06:48:26.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.674+0000 7fc6fbfff640 1 -- 172.21.15.79:0/60269405 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc70c02d480 con 0x5588ae7e56d0 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.675+0000 7fc725a50740 10 client.25787 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.675+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5588ae5267d0 con 0x5588ae7e56d0 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc6fbfff640 1 -- 172.21.15.79:0/60269405 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc70c013470 con 0x5588ae7e56d0 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc6fbfff640 1 -- 172.21.15.79:0/60269405 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fc70c026610 con 0x5588ae7e56d0 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc725a50740 10 client.25787 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc725a50740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:26.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc725a50740 20 client.25787 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:26.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc725a50740 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae7ff1e0 0x5588ae801660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc725a50740 4 client.25787 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:26.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.676+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588ae6e9930 con 0x5588ae7ff1e0 2024-04-22T06:48:26.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.678+0000 7fc71bfff640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae7ff1e0 0x5588ae801660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.678+0000 7fc6fbfff640 10 client.25787 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:26.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.678+0000 7fc71bfff640 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae7ff1e0 0x5588ae801660 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bdb9ca0cea2bc4d server_cookie=73052bcf64eb96b5 in_seq=0 out_seq=0 2024-04-22T06:48:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.679+0000 7fc6fbfff640 1 -- 172.21.15.79:0/60269405 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588ae6e9930 con 0x5588ae7ff1e0 2024-04-22T06:48:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.679+0000 7fc6fbfff640 10 client.25787 handle_command_reply: tid=0 2024-04-22T06:48:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.679+0000 7fc725a50740 1 client.25787 shutdown 2024-04-22T06:48:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 msgr2=0x5588ae7ee510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7ee510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc70c006a50 tx=0x7fc70c00ef10 comp rx=0 tx=0).stop 2024-04-22T06:48:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 shutdown_connections 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae7ff1e0 0x5588ae801660 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae7eb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 --2- 172.21.15.79:0/60269405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7ee510 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 >> 172.21.15.79:0/60269405 conn(0x5588ae7dfda0 msgr2=0x5588ae7e1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc6fbfff640 0 client.25787 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 shutdown_connections 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 1 -- 172.21.15.79:0/60269405 wait complete. 2024-04-22T06:48:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.680+0000 7fc725a50740 20 client.25787 trim_cache size 0 max 16384 2024-04-22T06:48:26.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:26 smithi193 ceph-mon[31815]: pgmap v127: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 63 MiB/s wr, 140 op/s 2024-04-22T06:48:26.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.705+0000 7fc725a50740 1 Processor -- start 2024-04-22T06:48:26.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.706+0000 7fc725a50740 1 -- start start 2024-04-22T06:48:26.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.706+0000 7fc725a50740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7faac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.706+0000 7fc725a50740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae8011f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc720bf8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7faac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc720bf8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7faac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57310/0 (socket says 172.21.15.79:57310) 2024-04-22T06:48:26.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc725a50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7deca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae8011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae8011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47496/0 (socket says 172.21.15.79:47496) 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc725a50740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc708011260 con 0x5588ae7e6080 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800eb30 con 0x5588ae7e56d0 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc720bf8640 1 -- 172.21.15.79:0/3446826009 learned_addr learned my addr 172.21.15.79:0/3446826009 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800c980 con 0x5588ae7eec00 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc7213f9640 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7deca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.706 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 -- 172.21.15.79:0/3446826009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 msgr2=0x5588ae7deca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7deca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 -- 172.21.15.79:0/3446826009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 msgr2=0x5588ae7faac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7faac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.707+0000 7fc71bfff640 1 -- 172.21.15.79:0/3446826009 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588ae7db670 con 0x5588ae7e6080 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.708+0000 7fc71bfff640 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae8011f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc7100f0010 tx=0x7fc7100be040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495fda2942595cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.708+0000 7fc7027fc640 1 -- 172.21.15.79:0/3446826009 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100c30d0 con 0x5588ae7e6080 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.708+0000 7fc7027fc640 1 -- 172.21.15.79:0/3446826009 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7100c74d0 con 0x5588ae7e6080 2024-04-22T06:48:26.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.708+0000 7fc7027fc640 1 -- 172.21.15.79:0/3446826009 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710005040 con 0x5588ae7e6080 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.709+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 msgr2=0x5588ae8011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.709+0000 7fc725a50740 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae8011f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc7100f0010 tx=0x7fc7100be040 comp rx=0 tx=0).stop 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.709+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 shutdown_connections 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7eec00 0x5588ae7deca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7e6080 0x5588ae8011f0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 --2- 172.21.15.79:0/3446826009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e56d0 0x5588ae7faac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 >> 172.21.15.79:0/3446826009 conn(0x5588ae7dfda0 msgr2=0x5588ae7de590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 shutdown_connections 2024-04-22T06:48:26.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 -- 172.21.15.79:0/3446826009 wait complete. 2024-04-22T06:48:26.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.710+0000 7fc725a50740 1 -- start start 2024-04-22T06:48:26.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 Processor -- start 2024-04-22T06:48:26.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e56d0 0x5588ae7f1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 0x5588ae7f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7eec00 0x5588ae7f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800cff0 con 0x5588ae7eec00 2024-04-22T06:48:26.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800e460 con 0x5588ae7e6080 2024-04-22T06:48:26.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.711+0000 7fc725a50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800d6c0 con 0x5588ae7e56d0 2024-04-22T06:48:26.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc725a50740 10 client.0 _ll_register_callbacks cb 0x7fc70c00eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:26.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 0x5588ae7f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 0x5588ae7f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57312/0 (socket says 172.21.15.79:57312) 2024-04-22T06:48:26.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 -- 172.21.15.79:0/1747728571 learned_addr learned my addr 172.21.15.79:0/1747728571 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:26.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc720bf8640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e56d0 0x5588ae7f1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc7213f9640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7eec00 0x5588ae7f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 -- 172.21.15.79:0/1747728571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e56d0 msgr2=0x5588ae7f1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e56d0 0x5588ae7f1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 -- 172.21.15.79:0/1747728571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7eec00 msgr2=0x5588ae7f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7eec00 0x5588ae7f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.712+0000 7fc71bfff640 1 -- 172.21.15.79:0/1747728571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588ae863890 con 0x5588ae7e6080 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.713+0000 7fc71bfff640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 0x5588ae7f24d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7100bef60 tx=0x7fc710000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93bdf534e9656369 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.713+0000 7fc6fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.713+0000 7fc6fbfff640 1 -- 172.21.15.79:0/1747728571 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100a3060 con 0x5588ae7e6080 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.713+0000 7fc6fbfff640 1 -- 172.21.15.79:0/1747728571 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fc7100bbf20 con 0x5588ae7e6080 2024-04-22T06:48:26.712 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.713+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7100c74d0 con 0x5588ae7e6080 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.713+0000 7fc6fbfff640 1 -- 172.21.15.79:0/1747728571 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc710005040 con 0x5588ae7e6080 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 10 client.25793 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5588ae864ba0 con 0x5588ae7e6080 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc6fbfff640 1 -- 172.21.15.79:0/1747728571 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7100c74d0 con 0x5588ae7e6080 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc6fbfff640 1 -- 172.21.15.79:0/1747728571 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fc7100c7040 con 0x5588ae7e6080 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 10 client.25793 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 20 client.25793 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae865160 0x5588ae867580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.714 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 4 client.25793 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588ae757c10 con 0x5588ae865160 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.714+0000 7fc720bf8640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae865160 0x5588ae867580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.715+0000 7fc720bf8640 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae865160 0x5588ae867580 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbc1459cac80442a server_cookie=22da2a95b6db76c3 in_seq=0 out_seq=0 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.715+0000 7fc6fbfff640 10 client.25793 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.715+0000 7fc6fbfff640 1 -- 172.21.15.79:0/1747728571 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588ae757c10 con 0x5588ae865160 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.715+0000 7fc6fbfff640 10 client.25793 handle_command_reply: tid=0 2024-04-22T06:48:26.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.715+0000 7fc725a50740 1 client.25793 shutdown 2024-04-22T06:48:26.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 msgr2=0x5588ae7f24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 0x5588ae7f24d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7100bef60 tx=0x7fc710000bc0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 shutdown_connections 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5588ae865160 0x5588ae867580 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5588ae7e6080 0x5588ae7f24d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5588ae7eec00 0x5588ae7f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.716+0000 7fc725a50740 1 --2- 172.21.15.79:0/1747728571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ae7e56d0 0x5588ae7f1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.717+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 >> 172.21.15.79:0/1747728571 conn(0x5588ae7dfda0 msgr2=0x5588ae7dee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.717+0000 7fc6fbfff640 0 client.25793 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.717+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 shutdown_connections 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.717+0000 7fc725a50740 1 -- 172.21.15.79:0/1747728571 wait complete. 2024-04-22T06:48:26.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.718+0000 7fc725a50740 20 client.25793 trim_cache size 0 max 16384 2024-04-22T06:48:26.718 INFO:teuthology.orchestra.run.smithi079.stdout:[] 2024-04-22T06:48:26.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.722+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7040a3e20 msgr2=0x7fc7040a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.722+0000 7fc721bfa640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7040a3e20 0x7fc7040a62e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc71c109f30 tx=0x7fc714004b90 comp rx=0 tx=0).stop 2024-04-22T06:48:26.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.722+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 msgr2=0x7fc71c10d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.722+0000 7fc721bfa640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c10d110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc71002c290 tx=0x7fc71002c8b0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.726+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 shutdown_connections 2024-04-22T06:48:26.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7040a3e20 0x7fc7040a62e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc71c10c500 0x7fc71c10d110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c116f50 0x7fc71c10d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 --2- 172.21.15.79:0/1670665746 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc71c109490 0x7fc71c10cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 >> 172.21.15.79:0/1670665746 conn(0x7fc71c06ced0 msgr2=0x7fc71c117330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 shutdown_connections 2024-04-22T06:48:26.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:26.727+0000 7fc721bfa640 1 -- 172.21.15.79:0/1670665746 wait complete. 2024-04-22T06:48:26.756 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.054+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/588890095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb41107a0 msgr2=0x7fcdb410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.054+0000 7fcdb9ce0640 1 --2- 172.21.15.79:0/588890095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb41107a0 0x7fcdb410e290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcda801a940 tx=0x7fcda803cca0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.056+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/588890095 shutdown_connections 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.056+0000 7fcdb9ce0640 1 --2- 172.21.15.79:0/588890095 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdb411b730 0x7fcdb411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.056+0000 7fcdb9ce0640 1 --2- 172.21.15.79:0/588890095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdb410e7d0 0x7fcdb410ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.056+0000 7fcdb9ce0640 1 --2- 172.21.15.79:0/588890095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb41107a0 0x7fcdb410e290 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.056+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/588890095 >> 172.21.15.79:0/588890095 conn(0x7fcdb406d060 msgr2=0x7fcdb406d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.056+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/588890095 shutdown_connections 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.058+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/588890095 wait complete. 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.059+0000 7fcdb9ce0640 1 Processor -- start 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.059+0000 7fcdb9ce0640 1 -- start start 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb9ce0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb410e7d0 0x7fcdb411adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb9ce0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdb41107a0 0x7fcdb4116e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb9ce0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdb411b730 0x7fcdb4117390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb9ce0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4113790 con 0x7fcdb411b730 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb9ce0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4113610 con 0x7fcdb410e7d0 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb9ce0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4113910 con 0x7fcdb41107a0 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdb411b730 0x7fcdb4117390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdb411b730 0x7fcdb4117390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47510/0 (socket says 172.21.15.79:47510) 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2463313084 learned_addr learned my addr 172.21.15.79:0/2463313084 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.060+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdb41107a0 0x7fcdb4116e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.061+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2463313084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdb41107a0 msgr2=0x7fcdb4116e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.061+0000 7fcdb3fff640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdb41107a0 0x7fcdb4116e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.061+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2463313084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb410e7d0 msgr2=0x7fcdb411adc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.061+0000 7fcdb3fff640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcdb410e7d0 0x7fcdb411adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.062 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.061+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2463313084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb4117ac0 con 0x7fcdb411b730 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.062+0000 7fcdb3fff640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdb411b730 0x7fcdb4117390 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00eaa0 tx=0x7fcdac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.063+0000 7fcdb0ff9640 1 -- 172.21.15.79:0/2463313084 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac00cdb0 con 0x7fcdb411b730 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.063+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/2463313084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb41ab4c0 con 0x7fcdb411b730 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.063+0000 7fcdb9ce0640 1 -- 172.21.15.79:0/2463313084 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb41ab8f0 con 0x7fcdb411b730 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.063+0000 7fcdb0ff9640 1 -- 172.21.15.79:0/2463313084 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdac005020 con 0x7fcdb411b730 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.064+0000 7fcdb0ff9640 1 -- 172.21.15.79:0/2463313084 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0116d0 con 0x7fcdb411b730 2024-04-22T06:48:27.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.067+0000 7fcdbc9ae740 1 Processor -- start 2024-04-22T06:48:27.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.071+0000 7fcdb0ff9640 1 -- 172.21.15.79:0/2463313084 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcdac0051d0 con 0x7fcdb411b730 2024-04-22T06:48:27.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.072+0000 7fcdb0ff9640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd9c0a3d50 0x7fcd9c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.073+0000 7fcdb0ff9640 1 -- 172.21.15.79:0/2463313084 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fcdac0d0b00 con 0x7fcdb411b730 2024-04-22T06:48:27.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.075+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd9c0a3d50 0x7fcd9c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 -- start start 2024-04-22T06:48:27.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e774d0 0x555864e778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e77e80 0x555864e80430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 0x555864e82e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x555864ddc1f0 con 0x555864e80a00 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555864d794b0 con 0x555864e77e80 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.076+0000 7fcdbc9ae740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555864dd81b0 con 0x555864e774d0 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 0x555864e82e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 0x555864e82e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47516/0 (socket says 172.21.15.79:47516) 2024-04-22T06:48:27.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2778104108 learned_addr learned my addr 172.21.15.79:0/2778104108 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2778104108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e774d0 msgr2=0x555864e778b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e774d0 0x555864e778b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2778104108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e77e80 msgr2=0x555864e80430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e77e80 0x555864e80430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.077+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2778104108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555864d6b1b0 con 0x555864e80a00 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.078+0000 7fcdb3fff640 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 0x555864e82e50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009270 tx=0x7fcdac0a0cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237b69a368bb99cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.078+0000 7fcd99ffb640 1 -- 172.21.15.79:0/2778104108 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac01f040 con 0x555864e80a00 2024-04-22T06:48:27.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.078+0000 7fcd99ffb640 1 -- 172.21.15.79:0/2778104108 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdac0054c0 con 0x555864e80a00 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.079+0000 7fcd99ffb640 1 -- 172.21.15.79:0/2778104108 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0bbd40 con 0x555864e80a00 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.079+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2778104108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 msgr2=0x555864e82e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.079+0000 7fcdbc9ae740 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 0x555864e82e50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009270 tx=0x7fcdac0a0cd0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.079+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2463313084 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcd9c0a3d50 0x7fcd9c0a6210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4118610 tx=0x7fcda4008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.080+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2778104108 shutdown_connections 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.080+0000 7fcdbc9ae740 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e77e80 0x555864e80430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.080+0000 7fcdbc9ae740 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e80a00 0x555864e82e50 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.080+0000 7fcdbc9ae740 1 --2- 172.21.15.79:0/2778104108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e774d0 0x555864e778b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.080+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2778104108 >> 172.21.15.79:0/2778104108 conn(0x555864e71ba0 msgr2=0x555864e71f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.081+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2778104108 shutdown_connections 2024-04-22T06:48:27.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.082+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2778104108 wait complete. 2024-04-22T06:48:27.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.082+0000 7fcdbc9ae740 1 -- start start 2024-04-22T06:48:27.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.082+0000 7fcdbc9ae740 1 Processor -- start 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e774d0 0x555864e801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e77e80 0x555864e7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e80a00 0x555864e7daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x555864e839c0 con 0x555864e774d0 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555864d95500 con 0x555864e80a00 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555864e83b40 con 0x555864e77e80 2024-04-22T06:48:27.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.083+0000 7fcdbc9ae740 10 client.0 _ll_register_callbacks cb 0x555864d63b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:27.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.084+0000 7fcdb3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e80a00 0x555864e7daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.085+0000 7fcdb3fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e80a00 0x555864e7daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57370/0 (socket says 172.21.15.79:57370) 2024-04-22T06:48:27.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.085+0000 7fcdb3fff640 1 -- 172.21.15.79:0/2325839592 learned_addr learned my addr 172.21.15.79:0/2325839592 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.085+0000 7fcdb37fe640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e774d0 0x555864e801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.086+0000 7fcdb37fe640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e77e80 msgr2=0x555864e7d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.086+0000 7fcdb37fe640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555864e77e80 0x555864e7d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.086+0000 7fcdb37fe640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e80a00 msgr2=0x555864e7daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.086+0000 7fcdb37fe640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555864e80a00 0x555864e7daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.086+0000 7fcdb37fe640 1 -- 172.21.15.79:0/2325839592 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555864aee3f0 con 0x555864e774d0 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.086+0000 7fcdb37fe640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x555864e774d0 0x555864e801d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fcda8000c00 tx=0x7fcda801a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8a2f1db60eba12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.087+0000 7fcd92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:27.086 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.088+0000 7fe0a863a640 1 -- 172.21.15.79:0/2612476723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 msgr2=0x7fe0a0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.086 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.088+0000 7fe0a863a640 1 --2- 172.21.15.79:0/2612476723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a0108f50 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe09001a940 tx=0x7fe09003cd00 comp rx=0 tx=0).stop 2024-04-22T06:48:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.089+0000 7fcd92ffd640 1 -- 172.21.15.79:0/2325839592 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8053020 con 0x555864e774d0 2024-04-22T06:48:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2325839592 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdac0054c0 con 0x555864e774d0 2024-04-22T06:48:27.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fcd92ffd640 1 -- 172.21.15.79:0/2325839592 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fcda8004250 con 0x555864e774d0 2024-04-22T06:48:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fe0a863a640 1 -- 172.21.15.79:0/2612476723 shutdown_connections 2024-04-22T06:48:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fe0a863a640 1 --2- 172.21.15.79:0/2612476723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a0110b20 0x7fe0a0109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fe0a863a640 1 --2- 172.21.15.79:0/2612476723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0a011b730 0x7fe0a011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fe0a863a640 1 --2- 172.21.15.79:0/2612476723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a0108f50 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.090+0000 7fe0a863a640 1 -- 172.21.15.79:0/2612476723 >> 172.21.15.79:0/2612476723 conn(0x7fe0a006d020 msgr2=0x7fe0a006d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.091+0000 7fcd92ffd640 1 -- 172.21.15.79:0/2325839592 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fcda805e050 con 0x555864e774d0 2024-04-22T06:48:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.091+0000 7fcdbc9ae740 10 client.16794 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:27.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.091+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2325839592 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x555864bb87d0 con 0x555864e774d0 2024-04-22T06:48:27.090 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.091+0000 7fe0a863a640 1 -- 172.21.15.79:0/2612476723 shutdown_connections 2024-04-22T06:48:27.090 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fe0a863a640 1 -- 172.21.15.79:0/2612476723 wait complete. 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.091+0000 7fcd92ffd640 1 -- 172.21.15.79:0/2325839592 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdac0054c0 con 0x555864e774d0 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcd92ffd640 1 -- 172.21.15.79:0/2325839592 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fcda805a4a0 con 0x555864e774d0 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcdbc9ae740 10 client.16794 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcdbc9ae740 10 client.16794 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcdbc9ae740 20 client.16794 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcdbc9ae740 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcdbc9ae740 4 client.16794 mds_command: new command op to 25261 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:27.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fcdbc9ae740 1 -- 172.21.15.79:0/2325839592 --> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555864d7b930 con 0x555864e91090 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.092+0000 7fe0a863a640 1 Processor -- start 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 -- start start 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a011aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 0x7fe0a0116f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0a01174a0 0x7fe0a0117950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01138a0 con 0x7fe0a0110740 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0113720 con 0x7fe0a011b730 2024-04-22T06:48:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a863a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0113a20 con 0x7fe0a01174a0 2024-04-22T06:48:27.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a5bd5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 0x7fe0a0116f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a5bd5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 0x7fe0a0116f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57382/0 (socket says 172.21.15.79:57382) 2024-04-22T06:48:27.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1594772056 learned_addr learned my addr 172.21.15.79:0/1594772056 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.093+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:27.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.094+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:48:27.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.094+0000 7fcd92ffd640 1 client.16794 ms_handle_refused on v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0a01174a0 0x7fe0a0117950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a011aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1594772056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0a01174a0 msgr2=0x7fe0a0117950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0a01174a0 0x7fe0a0117950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 msgr2=0x7fe0a011aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a011aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.095+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1594772056 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a01ab570 con 0x7fe0a011b730 2024-04-22T06:48:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.096+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a011aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:48:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.096+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 0x7fe0a0116f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00b4a0 tx=0x7fe09c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.097+0000 7fe0977fe640 1 -- 172.21.15.79:0/1594772056 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c014070 con 0x7fe0a011b730 2024-04-22T06:48:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.097+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a01ab860 con 0x7fe0a011b730 2024-04-22T06:48:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.097+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a01abdb0 con 0x7fe0a011b730 2024-04-22T06:48:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.101+0000 7fe0977fe640 1 -- 172.21.15.79:0/1594772056 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09c010ac0 con 0x7fe0a011b730 2024-04-22T06:48:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.101+0000 7fe0977fe640 1 -- 172.21.15.79:0/1594772056 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c017c40 con 0x7fe0a011b730 2024-04-22T06:48:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.104+0000 7fe0ab21f740 1 Processor -- start 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.105+0000 7fe0ab21f740 1 -- start start 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.106+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3eba620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.106+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 0x55c9c3ebafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.106+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.106+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c9c3e1f8d0 con 0x55c9c3ec40e0 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.106+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c9c3dbce80 con 0x55c9c3ebab60 2024-04-22T06:48:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.106+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9c3e1b890 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 0x55c9c3ebafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 0x55c9c3ebafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:57388/0 (socket says 172.21.15.79:57388) 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 -- 172.21.15.79:0/1609462900 learned_addr learned my addr 172.21.15.79:0/1609462900 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0977fe640 1 -- 172.21.15.79:0/1594772056 <== mon.2 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe09c017070 con 0x7fe0a011b730 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3eba620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 -- 172.21.15.79:0/1609462900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebb8d0 msgr2=0x55c9c3eba620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3eba620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 -- 172.21.15.79:0/1609462900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 msgr2=0x55c9c3ec64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.107+0000 7fe0a63d6640 1 -- 172.21.15.79:0/1609462900 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9c3dde320 con 0x55c9c3ebab60 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.108+0000 7fe0977fe640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe08c0a3ff0 0x7fe08c0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.109+0000 7fe0977fe640 1 -- 172.21.15.79:0/1594772056 <== mon.2 v2:172.21.15.190:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe09c0cfa40 con 0x7fe0a011b730 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.110+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 0x55c9c3ebafe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe090006010 tx=0x7fe090004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fccb26dc079fa508 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.111+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe08c0a3ff0 0x7fe08c0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.111+0000 7fe094bfe640 1 -- 172.21.15.79:0/1609462900 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090052020 con 0x55c9c3ebab60 2024-04-22T06:48:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.111+0000 7fe094bfe640 1 -- 172.21.15.79:0/1609462900 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09003dd60 con 0x55c9c3ebab60 2024-04-22T06:48:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.111+0000 7fe094bfe640 1 -- 172.21.15.79:0/1609462900 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090003a20 con 0x55c9c3ebab60 2024-04-22T06:48:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.112+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe08c0a3ff0 0x7fe08c0a64b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0a010e1f0 tx=0x7fe0980075c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.117+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1609462900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 msgr2=0x55c9c3ebafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.117+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 0x55c9c3ebafe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe090006010 tx=0x7fe090004040 comp rx=0 tx=0).stop 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.118+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1609462900 shutdown_connections 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.118+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebab60 0x55c9c3ebafe0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.118+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.118+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1609462900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3eba620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.118+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1609462900 >> 172.21.15.79:0/1609462900 conn(0x55c9c3e81c90 msgr2=0x55c9c3eb5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.118+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1609462900 shutdown_connections 2024-04-22T06:48:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.119+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1609462900 wait complete. 2024-04-22T06:48:27.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.119+0000 7fe0ab21f740 1 -- start start 2024-04-22T06:48:27.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.120+0000 7fe0ab21f740 1 Processor -- start 2024-04-22T06:48:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.120+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ec3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.120+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.120+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.120+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55c9c3ec6db0 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.121+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55c9c3da7eb0 con 0x55c9c3ec40e0 2024-04-22T06:48:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.121+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9c3ec6f30 con 0x55c9c3ebab60 2024-04-22T06:48:27.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.121+0000 7fe0ab21f740 10 client.0 _ll_register_callbacks cb 0x55c9c3dd4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.121+0000 7fe0a5bd5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.121+0000 7fe0a5bd5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47564/0 (socket says 172.21.15.79:47564) 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.121+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/2199891452 learned_addr learned my addr 172.21.15.79:0/2199891452 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.122+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/2199891452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 msgr2=0x55c9c3ec3a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.122+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ec3a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.122+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 msgr2=0x55c9c3ec13b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.122+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec13b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.122+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/2199891452 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9c3b32210 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.123+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec0e10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe09c000c00 tx=0x7fe09c09b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fddb30f68d2214 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.123+0000 7fe089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.124+0000 7fe089ffb640 1 -- 172.21.15.79:0/2199891452 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c01e070 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.124+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe09003dd60 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.124+0000 7fe089ffb640 1 -- 172.21.15.79:0/2199891452 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe09c097fe0 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.126+0000 7fe089ffb640 1 -- 172.21.15.79:0/2199891452 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe09c09b970 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.126+0000 7fe0ab21f740 10 client.16800 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.126+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c9c3bfe620 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe089ffb640 1 -- 172.21.15.79:0/2199891452 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09003dd60 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe089ffb640 1 -- 172.21.15.79:0/2199891452 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fe09c0bab30 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe0ab21f740 10 client.16800 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe0ab21f740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe0ab21f740 20 client.16800 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3ed45e0 0x55c9c3ed6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe0ab21f740 4 client.16800 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.127+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9c3e33880 con 0x55c9c3ed45e0 2024-04-22T06:48:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.129+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3ed45e0 0x55c9c3ed6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.132+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3ed45e0 0x55c9c3ed6a80 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35cc8dda39b88b25 server_cookie=1b765a7c0083f9aa in_seq=0 out_seq=0 2024-04-22T06:48:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.132+0000 7fe089ffb640 10 client.16800 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe089ffb640 1 -- 172.21.15.79:0/2199891452 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9c3e33880 con 0x55c9c3ed45e0 2024-04-22T06:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe089ffb640 10 client.16800 handle_command_reply: tid=0 2024-04-22T06:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe0ab21f740 1 client.16800 shutdown 2024-04-22T06:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 msgr2=0x55c9c3ec0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec0e10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe09c000c00 tx=0x7fe09c09b570 comp rx=0 tx=0).stop 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 shutdown_connections 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3ed45e0 0x55c9c3ed6a80 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.134+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec0e10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2199891452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ec3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 >> 172.21.15.79:0/2199891452 conn(0x55c9c3e81c90 msgr2=0x55c9c3eb75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe089ffb640 0 client.16800 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 shutdown_connections 2024-04-22T06:48:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2199891452 wait complete. 2024-04-22T06:48:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.135+0000 7fe0ab21f740 20 client.16800 trim_cache size 0 max 16384 2024-04-22T06:48:27.165 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.167+0000 7fe0ab21f740 1 Processor -- start 2024-04-22T06:48:27.165 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.167+0000 7fe0ab21f740 1 -- start start 2024-04-22T06:48:27.165 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.167+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ecfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.167+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe0840112d0 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe08400eba0 con 0x55c9c3ec40e0 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe08400c9f0 con 0x55c9c3ebab60 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47568/0 (socket says 172.21.15.79:47568) 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1630616571 learned_addr learned my addr 172.21.15.79:0/1630616571 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1630616571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 msgr2=0x55c9c3ecfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ecfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1630616571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 msgr2=0x55c9c3ec3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.168+0000 7fe0a5bd5640 1 -- 172.21.15.79:0/1630616571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9c3ed0880 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.169+0000 7fe0a5bd5640 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec3540 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe09c09bb30 tx=0x7fe09c099e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f734d0037e0a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.169+0000 7fe08bfff640 1 -- 172.21.15.79:0/1630616571 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c01e070 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.169+0000 7fe08bfff640 1 -- 172.21.15.79:0/1630616571 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe09c09fd20 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.169+0000 7fe08bfff640 1 -- 172.21.15.79:0/1630616571 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0bab30 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.170+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1630616571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 msgr2=0x55c9c3ec3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec3540 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe09c09bb30 tx=0x7fe09c099e80 comp rx=0 tx=0).stop 2024-04-22T06:48:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1630616571 shutdown_connections 2024-04-22T06:48:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ec3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ec3540 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/1630616571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ecfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1630616571 >> 172.21.15.79:0/1630616571 conn(0x55c9c3e81c90 msgr2=0x55c9c3eb5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1630616571 shutdown_connections 2024-04-22T06:48:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.171+0000 7fe0ab21f740 1 -- 172.21.15.79:0/1630616571 wait complete. 2024-04-22T06:48:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 -- start start 2024-04-22T06:48:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 Processor -- start 2024-04-22T06:48:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ecc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ebd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ebd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fe08400d060 con 0x55c9c3ec40e0 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe08400e4d0 con 0x55c9c3ebb8d0 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe08400d730 con 0x55c9c3ebab60 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.172+0000 7fe0ab21f740 10 client.0 _ll_register_callbacks cb 0x7fe09c09b740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ebd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ebd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47574/0 (socket says 172.21.15.79:47574) 2024-04-22T06:48:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 -- 172.21.15.79:0/2042175470 learned_addr learned my addr 172.21.15.79:0/2042175470 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 -- 172.21.15.79:0/2042175470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 msgr2=0x55c9c3ecc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ecc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 -- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebb8d0 msgr2=0x55c9c3ebd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ebd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.173+0000 7fe0a6bd7640 1 -- 172.21.15.79:0/2042175470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9c3ebdf10 con 0x55c9c3ec40e0 2024-04-22T06:48:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.174+0000 7fe0a6bd7640 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ebd9d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe098012e00 tx=0x7fe098004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650d228c6e7b9203 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.174+0000 7fe089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.174+0000 7fe089ffb640 1 -- 172.21.15.79:0/2042175470 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098024070 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.174+0000 7fe089ffb640 1 -- 172.21.15.79:0/2042175470 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8813+0+0 (secure 0 0 0) 0x7fe098020620 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.175+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe09c09fd20 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.175+0000 7fe089ffb640 1 -- 172.21.15.79:0/2042175470 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe098030050 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.175+0000 7fe0ab21f740 10 client.16806 fetch_fsmap learned FSMap version 40 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.175+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c9c3f3a8b0 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe089ffb640 1 -- 172.21.15.79:0/2042175470 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe098018040 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe089ffb640 1 -- 172.21.15.79:0/2042175470 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 40) ==== 8338+0+0 (secure 0 0 0) 0x7fe098011040 con 0x55c9c3ec40e0 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0ab21f740 10 client.16806 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T06:48:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0ab21f740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0ab21f740 20 client.16806 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3f3b060 0x55c9c3f3d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0ab21f740 4 client.16806 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe09c09fd20 con 0x55c9c3f3b060 2024-04-22T06:48:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.176+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3f3b060 0x55c9c3f3d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.177+0000 7fe089ffb640 10 client.16806 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:27.177+0000 7fe0a63d6640 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3f3b060 0x55c9c3f3d4c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a42d689eb1c5722 server_cookie=bfde05ab13173813 in_seq=0 out_seq=0 2024-04-22T06:48:27.205 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:27 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3445861895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:27.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.298+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:27.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.298+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:48:27.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.301+0000 7fcd92ffd640 1 client.16794 ms_handle_refused on v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:27.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:27 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3445861895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:27.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.699+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.699+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:48:27.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:27.699+0000 7fcd92ffd640 1 client.16794 ms_handle_refused on v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:27.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:27 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3445861895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:28.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:28 smithi079 ceph-mon[27946]: pgmap v128: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 322 MiB/s wr, 1.39k op/s 2024-04-22T06:48:28.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:28.501+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:28.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:28.501+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:48:28.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:28.501+0000 7fcd92ffd640 1 client.16794 ms_handle_refused on v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:28.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:28 smithi193 ceph-mon[31815]: pgmap v128: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 322 MiB/s wr, 1.39k op/s 2024-04-22T06:48:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:28 smithi190 ceph-mon[31421]: pgmap v128: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 322 MiB/s wr, 1.39k op/s 2024-04-22T06:48:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.935+0000 7fe089ffb640 1 -- 172.21.15.79:0/2042175470 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe09c09fd20 con 0x55c9c3f3b060 2024-04-22T06:48:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.935+0000 7fe089ffb640 10 client.16806 handle_command_reply: tid=0 2024-04-22T06:48:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.936+0000 7fe0ab21f740 1 client.16806 shutdown 2024-04-22T06:48:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.936+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 msgr2=0x55c9c3ebd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.936+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ebd9d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe098012e00 tx=0x7fe098004a40 comp rx=0 tx=0).stop 2024-04-22T06:48:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 shutdown_connections 2024-04-22T06:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55c9c3f3b060 0x55c9c3f3d4c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe089ffb640 0 client.16806 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c9c3ebb8d0 0x55c9c3ebd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55c9c3ec40e0 0x55c9c3ebd9d0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 --2- 172.21.15.79:0/2042175470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9c3ebab60 0x55c9c3ecc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 >> 172.21.15.79:0/2042175470 conn(0x55c9c3e81c90 msgr2=0x55c9c3ebf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 shutdown_connections 2024-04-22T06:48:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.937+0000 7fe0ab21f740 1 -- 172.21.15.79:0/2042175470 wait complete. 2024-04-22T06:48:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.938+0000 7fe0ab21f740 20 client.16806 trim_cache size 0 max 16384 2024-04-22T06:48:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:48:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:48:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.939+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe08c0a3ff0 msgr2=0x7fe08c0a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.939+0000 7fe0a863a640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe08c0a3ff0 0x7fe08c0a64b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0a010e1f0 tx=0x7fe0980075c0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.939+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 msgr2=0x7fe0a0116f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.939+0000 7fe0a863a640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 0x7fe0a0116f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00b4a0 tx=0x7fe09c00b960 comp rx=0 tx=0).stop 2024-04-22T06:48:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 shutdown_connections 2024-04-22T06:48:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fe08c0a3ff0 0x7fe08c0a64b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0a011b730 0x7fe0a0116f60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0a01174a0 0x7fe0a0117950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 --2- 172.21.15.79:0/1594772056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe0a0110740 0x7fe0a011aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 >> 172.21.15.79:0/1594772056 conn(0x7fe0a006d020 msgr2=0x7fe0a010d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 shutdown_connections 2024-04-22T06:48:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:28.940+0000 7fe0a863a640 1 -- 172.21.15.79:0/1594772056 wait complete. 2024-04-22T06:48:30.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:30.104+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:30.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:30.104+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:48:30.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:30.104+0000 7fcd92ffd640 1 client.16794 ms_handle_refused on v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:30.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:30 smithi079 ceph-mon[27946]: pgmap v129: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 291 MiB/s wr, 1.33k op/s 2024-04-22T06:48:30.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:30 smithi193 ceph-mon[31815]: pgmap v129: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 291 MiB/s wr, 1.33k op/s 2024-04-22T06:48:30.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:30 smithi190 ceph-mon[31421]: pgmap v129: 97 pgs: 97 active+clean; 21 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 291 MiB/s wr, 1.33k op/s 2024-04-22T06:48:32.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:32 smithi079 ceph-mon[27946]: pgmap v130: 97 pgs: 97 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 491 MiB/s wr, 2.12k op/s 2024-04-22T06:48:32.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:32 smithi193 ceph-mon[31815]: pgmap v130: 97 pgs: 97 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 491 MiB/s wr, 2.12k op/s 2024-04-22T06:48:32.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:32 smithi190 ceph-mon[31421]: pgmap v130: 97 pgs: 97 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 491 MiB/s wr, 2.12k op/s 2024-04-22T06:48:33.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:33.310+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:33.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:33.310+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-22T06:48:33.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:33.310+0000 7fcd92ffd640 1 client.16794 ms_handle_refused on v2:172.21.15.193:6838/1588799940 2024-04-22T06:48:33.540 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:33 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:48:33.541 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:33 smithi193 systemd[1]: Stopped Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:48:33.541 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:33 smithi193 systemd[1]: Starting Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:48:33.588 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:33 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:48:33.589 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:33 smithi190 systemd[1]: Stopped Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:48:33.589 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:33 smithi190 systemd[1]: Starting Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:48:33.932 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:33 smithi193 podman[67106]: 2024-04-22 06:48:33.449696843 +0000 UTC m=+0.034108343 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:48:33.952 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:33 smithi190 podman[66072]: 2024-04-22 06:48:33.49495099 +0000 UTC m=+0.039513316 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:48:34.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:34 smithi190 ceph-mon[31421]: pgmap v131: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 MiB/s wr, 2.59k op/s 2024-04-22T06:48:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:34 smithi193 ceph-mon[31815]: pgmap v131: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 MiB/s wr, 2.59k op/s 2024-04-22T06:48:34.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:34 smithi079 ceph-mon[27946]: pgmap v131: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 MiB/s wr, 2.59k op/s 2024-04-22T06:48:35.192 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:34 smithi193 podman[67106]: 2024-04-22 06:48:34.834473143 +0000 UTC m=+1.418884681 container create 60088dbe46cb57c31b342a2d18a2502038138bab5cf0b55ab7d906d7d630d1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T06:48:35.452 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:35 smithi190 podman[66072]: 2024-04-22 06:48:35.200959028 +0000 UTC m=+1.745521336 container create 19d9ccf2235e3d19a446d9778c2312222b34721b5b375d3448eab0042d05d07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:48:36.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:36 smithi193 ceph-mon[31815]: pgmap v132: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 600 MiB/s wr, 2.54k op/s 2024-04-22T06:48:36.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:36 smithi190 ceph-mon[31421]: pgmap v132: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 600 MiB/s wr, 2.54k op/s 2024-04-22T06:48:36.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:36 smithi079 ceph-mon[27946]: pgmap v132: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 600 MiB/s wr, 2.54k op/s 2024-04-22T06:48:37.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:37 smithi193 podman[67106]: 2024-04-22 06:48:37.072014405 +0000 UTC m=+3.656425893 container init 60088dbe46cb57c31b342a2d18a2502038138bab5cf0b55ab7d906d7d630d1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T06:48:37.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:37 smithi193 podman[67106]: 2024-04-22 06:48:37.077157168 +0000 UTC m=+3.661568650 container start 60088dbe46cb57c31b342a2d18a2502038138bab5cf0b55ab7d906d7d630d1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T06:48:37.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mds[67203]: -- 172.21.15.193:0/335393716 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b622cfa40 con 0x560b622a2800 2024-04-22T06:48:37.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: starting mds.i at 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: daemon mds.i restarted 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:boot 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: Replacing daemon mds.h as rank 1 with standby daemon mds.e 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:48:37.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:48:37.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:48:37.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:37 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=f=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: daemon mds.i restarted 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:boot 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: Replacing daemon mds.h as rank 1 with standby daemon mds.e 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-22T06:48:37.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:48:37.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:48:37.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:48:37.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:37 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=f=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:48:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:48:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: daemon mds.i restarted 2024-04-22T06:48:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:48:37.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:boot 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: Replacing daemon mds.h as rank 1 with standby daemon mds.e 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:48:37.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:37 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:replay,2=f=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:48:37.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:37 smithi193 bash[67106]: 60088dbe46cb57c31b342a2d18a2502038138bab5cf0b55ab7d906d7d630d1f9 2024-04-22T06:48:37.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:48:37 smithi193 systemd[1]: Started Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:48:38.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:38 smithi190 ceph-mon[31421]: pgmap v133: 97 pgs: 97 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 910 MiB/s wr, 3.93k op/s 2024-04-22T06:48:38.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:38 smithi190 podman[66072]: 2024-04-22 06:48:38.039508359 +0000 UTC m=+4.584070678 container init 19d9ccf2235e3d19a446d9778c2312222b34721b5b375d3448eab0042d05d07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T06:48:38.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:38 smithi190 podman[66072]: 2024-04-22 06:48:38.045850555 +0000 UTC m=+4.590412870 container start 19d9ccf2235e3d19a446d9778c2312222b34721b5b375d3448eab0042d05d07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:48:38.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:38 smithi190 ceph-mds[66177]: -- 172.21.15.190:0/107604217 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558db12c3a40 con 0x558db1296800 2024-04-22T06:48:38.453 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:38 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: starting mds.h at 2024-04-22T06:48:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:38 smithi193 ceph-mon[31815]: pgmap v133: 97 pgs: 97 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 910 MiB/s wr, 3.93k op/s 2024-04-22T06:48:38.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:38 smithi079 ceph-mon[27946]: pgmap v133: 97 pgs: 97 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 910 MiB/s wr, 3.93k op/s 2024-04-22T06:48:39.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:39 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:resolve 2024-04-22T06:48:39.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:39 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:resolve 2024-04-22T06:48:39.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:39 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:boot 2024-04-22T06:48:39.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:39 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=f=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:39.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:39 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:48:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:39 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:resolve 2024-04-22T06:48:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:39 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:resolve 2024-04-22T06:48:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:39 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:boot 2024-04-22T06:48:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:39 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=f=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:39.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:39 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:48:39.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:39.720+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:39.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:39.734+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:48:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:39.734+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:48:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:39.734+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:48:39.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:39.734+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-22T06:48:39.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:39 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:resolve 2024-04-22T06:48:39.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:39 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:resolve 2024-04-22T06:48:39.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:39 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:boot 2024-04-22T06:48:39.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:39 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:resolve,2=f=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:39.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:39 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:48:39.953 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:39 smithi190 bash[66072]: 19d9ccf2235e3d19a446d9778c2312222b34721b5b375d3448eab0042d05d07c 2024-04-22T06:48:39.953 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:48:39 smithi190 systemd[1]: Started Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:48:40.642 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: pgmap v136: 97 pgs: 97 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 677 MiB/s wr, 2.84k op/s 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00299992 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00399994 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00999972 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:reconnect 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:reconnect 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0129998 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=f=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:rejoin 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:rejoin 2024-04-22T06:48:40.643 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:40 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=f=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:40.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: pgmap v136: 97 pgs: 97 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 677 MiB/s wr, 2.84k op/s 2024-04-22T06:48:40.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00299992 2024-04-22T06:48:40.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00399994 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00999972 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:reconnect 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:reconnect 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0129998 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=f=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:rejoin 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:rejoin 2024-04-22T06:48:40.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:40 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=f=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: pgmap v136: 97 pgs: 97 active+clean; 26 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 677 MiB/s wr, 2.84k op/s 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00299992 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00399994 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00999972 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:reconnect 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:reconnect 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0129998 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:reconnect,2=f=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:48:40.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:48:40.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:rejoin 2024-04-22T06:48:40.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:rejoin 2024-04-22T06:48:40.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:40 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=e=up:rejoin,2=f=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:41 smithi193 ceph-mon[31815]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:48:41.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:41 smithi193 ceph-mon[31815]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-22T06:48:41.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:41 smithi190 ceph-mon[31421]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:48:41.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:41 smithi190 ceph-mon[31421]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-22T06:48:41.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:41 smithi079 ceph-mon[27946]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:48:41.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:41 smithi079 ceph-mon[27946]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.113+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2674773872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81017d0 msgr2=0x7f04c8101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.113+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2674773872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81017d0 0x7f04c8101bb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009900 tx=0x7f04b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2674773872 shutdown_connections 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2674773872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04c8102180 0x7f04c8105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2674773872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04c8106380 0x7f04c810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2674773872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81017d0 0x7f04c8101bb0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2674773872 >> 172.21.15.79:0/2674773872 conn(0x7f04c80773e0 msgr2=0x7f04c80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2674773872 shutdown_connections 2024-04-22T06:48:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.114+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2674773872 wait complete. 2024-04-22T06:48:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.115+0000 7f04ccdbd640 1 Processor -- start 2024-04-22T06:48:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.115+0000 7f04ccdbd640 1 -- start start 2024-04-22T06:48:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04ccdbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04c81017d0 0x7f04c819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04ccdbd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04c8106380 0x7f04c819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04ccdbd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 0x7f04c81a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04ccdbd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8111390 con 0x7f04c81a3750 2024-04-22T06:48:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04ccdbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8111210 con 0x7f04c81017d0 2024-04-22T06:48:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04ccdbd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04c8111510 con 0x7f04c8106380 2024-04-22T06:48:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 0x7f04c81a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 0x7f04c81a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36746/0 (socket says 172.21.15.79:36746) 2024-04-22T06:48:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 -- 172.21.15.79:0/2329162050 learned_addr learned my addr 172.21.15.79:0/2329162050 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 -- 172.21.15.79:0/2329162050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04c8106380 msgr2=0x7f04c819f790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04c8106380 0x7f04c819f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 -- 172.21.15.79:0/2329162050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04c81017d0 msgr2=0x7f04c819f250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04c81017d0 0x7f04c819f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.116+0000 7f04c6d9d640 1 -- 172.21.15.79:0/2329162050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c81a4240 con 0x7f04c81a3750 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.117+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 0x7f04c81a3c00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f04c000b970 tx=0x7f04c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.117+0000 7f04bf7fe640 1 -- 172.21.15.79:0/2329162050 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0019070 con 0x7f04c81a3750 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.118+0000 7f04bf7fe640 1 -- 172.21.15.79:0/2329162050 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c0014440 con 0x7f04c81a3750 2024-04-22T06:48:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.118+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c81a4530 con 0x7f04c81a3750 2024-04-22T06:48:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.118+0000 7f04bf7fe640 1 -- 172.21.15.79:0/2329162050 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c00075d0 con 0x7f04c81a3750 2024-04-22T06:48:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.118+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c81abd20 con 0x7f04c81a3750 2024-04-22T06:48:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.122+0000 7f04cf9b1740 1 Processor -- start 2024-04-22T06:48:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.123+0000 7f04bf7fe640 1 -- 172.21.15.79:0/2329162050 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f04c001c070 con 0x7f04c81a3750 2024-04-22T06:48:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.125+0000 7f04cf9b1740 1 -- start start 2024-04-22T06:48:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.128+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0ee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.128+0000 7f04bf7fe640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04b40a3e90 0x7f04b40a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.128+0000 7f04c659c640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04b40a3e90 0x7f04b40a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.128+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04bf7fe640 1 -- 172.21.15.79:0/2329162050 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f04c00cfeb0 con 0x7f04c81a3750 2024-04-22T06:48:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0fa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04cf9b1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f2ae0538d0 con 0x55f2ae0eeb70 2024-04-22T06:48:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04cf9b1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2adff0e80 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04cf9b1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f2ae04f890 con 0x55f2ae0f80f0 2024-04-22T06:48:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0eeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36760/0 (socket says 172.21.15.79:36760) 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04bffff640 1 -- 172.21.15.79:0/3306992654 learned_addr learned my addr 172.21.15.79:0/3306992654 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04c659c640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04b40a3e90 0x7f04b40a6350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f04b8005c20 tx=0x7f04b8005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.129+0000 7f04bffff640 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0fa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bffff640 1 -- 172.21.15.79:0/3306992654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0f80f0 msgr2=0x55f2ae0fa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bffff640 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0fa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bffff640 1 -- 172.21.15.79:0/3306992654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0ef8e0 msgr2=0x55f2ae0ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bffff640 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0ee630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bffff640 1 -- 172.21.15.79:0/3306992654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ae012320 con 0x55f2ae0eeb70 2024-04-22T06:48:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bffff640 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0eeff0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f04b000eca0 tx=0x7f04b0005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1db9ad629b8f713 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.130+0000 7f04bcff9640 1 -- 172.21.15.79:0/3306992654 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0004220 con 0x55f2ae0eeb70 2024-04-22T06:48:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.131+0000 7f04bcff9640 1 -- 172.21.15.79:0/3306992654 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b00043c0 con 0x55f2ae0eeb70 2024-04-22T06:48:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.131+0000 7f04bcff9640 1 -- 172.21.15.79:0/3306992654 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00127c0 con 0x55f2ae0eeb70 2024-04-22T06:48:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.132+0000 7f04cf9b1740 1 -- 172.21.15.79:0/3306992654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 msgr2=0x55f2ae0eeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.132+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0eeff0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f04b000eca0 tx=0x7f04b0005f50 comp rx=0 tx=0).stop 2024-04-22T06:48:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.133+0000 7f04cf9b1740 1 -- 172.21.15.79:0/3306992654 shutdown_connections 2024-04-22T06:48:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.133+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0fa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.133+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0eeff0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.133+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/3306992654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.133+0000 7f04cf9b1740 1 -- 172.21.15.79:0/3306992654 >> 172.21.15.79:0/3306992654 conn(0x55f2ae0b5c90 msgr2=0x55f2ae0e9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.133+0000 7f04cf9b1740 1 -- 172.21.15.79:0/3306992654 shutdown_connections 2024-04-22T06:48:42.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.139+0000 7f04cf9b1740 1 -- 172.21.15.79:0/3306992654 wait complete. 2024-04-22T06:48:42.138 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.139+0000 7f04cf9b1740 1 -- start start 2024-04-22T06:48:42.138 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.140+0000 7f04cf9b1740 1 Processor -- start 2024-04-22T06:48:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.141+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.141+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.141+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04cf9b1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55f2ae0fae20 con 0x55f2ae0f80f0 2024-04-22T06:48:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04cf9b1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55f2adfdbeb0 con 0x55f2ae0eeb70 2024-04-22T06:48:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04cf9b1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f2ae0fafa0 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04cf9b1740 10 client.0 _ll_register_callbacks cb 0x55f2ae008b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04c659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04c659c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:43112/0 (socket says 172.21.15.79:43112) 2024-04-22T06:48:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04c659c640 1 -- 172.21.15.79:0/4192725545 learned_addr learned my addr 172.21.15.79:0/4192725545 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.142+0000 7f04bffff640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.143+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04c6d9d640 1 -- 172.21.15.79:0/4192725545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0ef8e0 msgr2=0x55f2ae0f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04c6d9d640 1 -- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0eeb70 msgr2=0x55f2ae0f7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04c6d9d640 1 -- 172.21.15.79:0/4192725545 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2add66210 con 0x55f2ae0f80f0 2024-04-22T06:48:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f54b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f04c00cd540 tx=0x7f04c0004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfea20f64892559 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.144+0000 7f04adffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.145+0000 7f04adffb640 1 -- 172.21.15.79:0/4192725545 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c00b5260 con 0x55f2ae0f80f0 2024-04-22T06:48:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.145+0000 7f04adffb640 1 -- 172.21.15.79:0/4192725545 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f04c00145f0 con 0x55f2ae0f80f0 2024-04-22T06:48:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.145+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b00043c0 con 0x55f2ae0f80f0 2024-04-22T06:48:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.146+0000 7f04adffb640 1 -- 172.21.15.79:0/4192725545 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f04c0098400 con 0x55f2ae0f80f0 2024-04-22T06:48:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.146+0000 7f04cf9b1740 10 client.16818 fetch_fsmap learned FSMap version 47 2024-04-22T06:48:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.146+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f2ade32620 con 0x55f2ae0f80f0 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.146+0000 7f04adffb640 1 -- 172.21.15.79:0/4192725545 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c00a4570 con 0x55f2ae0f80f0 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04adffb640 1 -- 172.21.15.79:0/4192725545 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7f04c00cdb80 con 0x55f2ae0f80f0 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04cf9b1740 10 client.16818 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04cf9b1740 10 client.16818 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04cf9b1740 20 client.16818 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae108230 0x55f2ae10a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04cf9b1740 4 client.16818 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f04b00043c0 con 0x55f2ae108230 2024-04-22T06:48:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.147+0000 7f04c659c640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae108230 0x55f2ae10a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.149+0000 7f04c659c640 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae108230 0x55f2ae10a620 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56b75efee69f5242 server_cookie=684b5827b3e56524 in_seq=0 out_seq=0 2024-04-22T06:48:42.148 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.149+0000 7f04adffb640 10 client.16818 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.155+0000 7f04adffb640 1 -- 172.21.15.79:0/4192725545 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f04b00043c0 con 0x55f2ae108230 2024-04-22T06:48:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.155+0000 7f04adffb640 10 client.16818 handle_command_reply: tid=0 2024-04-22T06:48:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.155+0000 7f04cf9b1740 1 client.16818 shutdown 2024-04-22T06:48:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.155+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0f80f0 msgr2=0x55f2ae0f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.155+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f54b0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f04c00cd540 tx=0x7f04c0004b50 comp rx=0 tx=0).stop 2024-04-22T06:48:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 shutdown_connections 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae108230 0x55f2ae10a620 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f54b0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/4192725545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 >> 172.21.15.79:0/4192725545 conn(0x55f2ae0b5c90 msgr2=0x55f2ae0eb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04adffb640 0 client.16818 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 shutdown_connections 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 1 -- 172.21.15.79:0/4192725545 wait complete. 2024-04-22T06:48:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.156+0000 7f04cf9b1740 20 client.16818 trim_cache size 0 max 16384 2024-04-22T06:48:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.187+0000 7f04cf9b1740 1 Processor -- start 2024-04-22T06:48:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.187+0000 7f04cf9b1740 1 -- start start 2024-04-22T06:48:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae103d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04c659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04c659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae103d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:53864/0 (socket says 172.21.15.79:53864) 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04c659c640 1 -- 172.21.15.79:0/510687546 learned_addr learned my addr 172.21.15.79:0/510687546 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81b6cb0 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04bffff640 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81b4580 con 0x55f2ae0f80f0 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81b23d0 con 0x55f2ae0eeb70 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.188+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04c6d9d640 1 -- 172.21.15.79:0/510687546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 msgr2=0x55f2ae103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04c6d9d640 1 -- 172.21.15.79:0/510687546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 msgr2=0x55f2ae0f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04c6d9d640 1 -- 172.21.15.79:0/510687546 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ae0ede40 con 0x55f2ae0f80f0 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04bffff640 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04c6d9d640 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f75d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f04c00981f0 tx=0x7f04c009a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47716fba32f7c9a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04affff640 1 -- 172.21.15.79:0/510687546 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c00985a0 con 0x55f2ae0f80f0 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04affff640 1 -- 172.21.15.79:0/510687546 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04c009c490 con 0x55f2ae0f80f0 2024-04-22T06:48:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.189+0000 7f04affff640 1 -- 172.21.15.79:0/510687546 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0009040 con 0x55f2ae0f80f0 2024-04-22T06:48:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 msgr2=0x55f2ae0f75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f75d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f04c00981f0 tx=0x7f04c009a2a0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 shutdown_connections 2024-04-22T06:48:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f75d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/510687546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 >> 172.21.15.79:0/510687546 conn(0x55f2ae0b5c90 msgr2=0x55f2ae0f4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 shutdown_connections 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 -- 172.21.15.79:0/510687546 wait complete. 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.191+0000 7f04cf9b1740 1 -- start start 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.192+0000 7f04cf9b1740 1 Processor -- start 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.192+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.192+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.192+0000 7f04bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36782/0 (socket says 172.21.15.79:36782) 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04cf9b1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 -- 172.21.15.79:0/435271153 learned_addr learned my addr 172.21.15.79:0/435271153 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:48:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81b2a40 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81b3eb0 con 0x55f2ae0f80f0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04c81b3110 con 0x55f2ae0eeb70 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 -- 172.21.15.79:0/435271153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 msgr2=0x55f2ae0f8ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f8ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 -- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 msgr2=0x55f2ae0f9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2ae0fe800 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04cf9b1740 10 client.0 _ll_register_callbacks cb 0x7f04c00baf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04c009c490 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04bffff640 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f9530 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f04b000eca0 tx=0x7f04b0012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a87058302da6033 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.193+0000 7f04adffb640 10 client.16824 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04adffb640 1 -- 172.21.15.79:0/435271153 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0029020 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04adffb640 1 -- 172.21.15.79:0/435271153 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f04b0025460 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04adffb640 1 -- 172.21.15.79:0/435271153 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f04b00256a0 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 10 client.16824 fetch_fsmap learned FSMap version 47 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({fsmap=47}) -- 0x55f2ae16eac0 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04adffb640 1 -- 172.21.15.79:0/435271153 <== mon.0 v2:172.21.15.79:3300/0 4 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7f04b000f040 con 0x55f2ae0ef8e0 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 10 client.16824 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 10 client.16824 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 20 client.16824 populate_metadata read hostname 'smithi079' 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae16edf0 0x55f2ae171220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 4 client.16824 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.194+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2ae067880 con 0x55f2ae16edf0 2024-04-22T06:48:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.195+0000 7f04c659c640 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae16edf0 0x55f2ae171220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.196+0000 7f04adffb640 10 client.16824 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:42.196+0000 7f04c659c640 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae16edf0 0x55f2ae171220 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf6c0e79a0a76ecf server_cookie=9bee1e436c53b580 in_seq=0 out_seq=0 2024-04-22T06:48:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: pgmap v137: 97 pgs: 97 active+clean; 27 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 792 MiB/s wr, 2.75k op/s 2024-04-22T06:48:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:48:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:48:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:active 2024-04-22T06:48:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:active 2024-04-22T06:48:42.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:42.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:42 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:48:42.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: pgmap v137: 97 pgs: 97 active+clean; 27 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 792 MiB/s wr, 2.75k op/s 2024-04-22T06:48:42.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:48:42.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:48:42.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:active 2024-04-22T06:48:42.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:active 2024-04-22T06:48:42.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:42.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:42 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: pgmap v137: 97 pgs: 97 active+clean; 27 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 792 MiB/s wr, 2.75k op/s 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/833753836,v1:172.21.15.190:6835/833753836] up:active 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/3147972524,v1:172.21.15.193:6835/3147972524] up:active 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:48:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:42 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=e=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:48:44.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:44 smithi193 ceph-mon[31815]: pgmap v138: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 977 MiB/s wr, 3.13k op/s 2024-04-22T06:48:44.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:44 smithi190 ceph-mon[31421]: pgmap v138: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 977 MiB/s wr, 3.13k op/s 2024-04-22T06:48:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:44 smithi079 ceph-mon[27946]: pgmap v138: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 977 MiB/s wr, 3.13k op/s 2024-04-22T06:48:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:46 smithi193 ceph-mon[31815]: pgmap v139: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 512 MiB/s wr, 1.04k op/s 2024-04-22T06:48:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:46 smithi190 ceph-mon[31421]: pgmap v139: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 512 MiB/s wr, 1.04k op/s 2024-04-22T06:48:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:46 smithi079 ceph-mon[27946]: pgmap v139: 97 pgs: 97 active+clean; 28 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 512 MiB/s wr, 1.04k op/s 2024-04-22T06:48:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.569+0000 7f04adffb640 1 -- 172.21.15.79:0/435271153 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2ae067880 con 0x55f2ae16edf0 2024-04-22T06:48:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.569+0000 7f04adffb640 10 client.16824 handle_command_reply: tid=0 2024-04-22T06:48:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.569+0000 7f04cf9b1740 1 client.16824 shutdown 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.570+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 msgr2=0x55f2ae0f9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.570+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f9530 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f04b000eca0 tx=0x7f04b0012c40 comp rx=0 tx=0).stop 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.570+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 shutdown_connections 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.570+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55f2ae16edf0 0x55f2ae171220 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04adffb640 0 client.16824 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.570+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2ae0f80f0 0x55f2ae0f9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55f2ae0ef8e0 0x55f2ae0f9530 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04cf9b1740 1 --2- 172.21.15.79:0/435271153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2ae0eeb70 0x55f2ae0f8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 >> 172.21.15.79:0/435271153 conn(0x55f2ae0b5c90 msgr2=0x55f2ae0e80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:47.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 shutdown_connections 2024-04-22T06:48:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04cf9b1740 1 -- 172.21.15.79:0/435271153 wait complete. 2024-04-22T06:48:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.571+0000 7f04cf9b1740 20 client.16824 trim_cache size 0 max 16384 2024-04-22T06:48:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:48:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:48:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:48:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:48:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.574+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04b40a3e90 msgr2=0x7f04b40a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.574+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04b40a3e90 0x7f04b40a6350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f04b8005c20 tx=0x7f04b8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.574+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 msgr2=0x7f04c81a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.574+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 0x7f04c81a3c00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f04c000b970 tx=0x7f04c000be30 comp rx=0 tx=0).stop 2024-04-22T06:48:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.575+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 shutdown_connections 2024-04-22T06:48:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.575+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04c81a3750 0x7f04c81a3c00 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.575+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04c8106380 0x7f04c819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.575+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04b40a3e90 0x7f04b40a6350 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.575+0000 7f04ccdbd640 1 --2- 172.21.15.79:0/2329162050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04c81017d0 0x7f04c819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.575+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 >> 172.21.15.79:0/2329162050 conn(0x7f04c80773e0 msgr2=0x7f04c8100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.577+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 shutdown_connections 2024-04-22T06:48:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:48:47.577+0000 7f04ccdbd640 1 -- 172.21.15.79:0/2329162050 wait complete. 2024-04-22T06:48:48.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:48 smithi190 ceph-mon[31421]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 834 MiB/s wr, 1.67k op/s 2024-04-22T06:48:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:48 smithi079 ceph-mon[27946]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 834 MiB/s wr, 1.67k op/s 2024-04-22T06:48:48.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:48 smithi193 ceph-mon[31815]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 834 MiB/s wr, 1.67k op/s 2024-04-22T06:48:50.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:50 smithi079 ceph-mon[27946]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 696 MiB/s wr, 1.40k op/s 2024-04-22T06:48:50.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:50 smithi193 ceph-mon[31815]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 696 MiB/s wr, 1.40k op/s 2024-04-22T06:48:50.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:50 smithi190 ceph-mon[31421]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 696 MiB/s wr, 1.40k op/s 2024-04-22T06:48:52.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:52.551+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:52.568+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:48:52.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:52.568+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:48:52.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:52.568+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:48:52.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:48:52.569+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:48:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:52 smithi193 ceph-mon[31815]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 898 MiB/s wr, 1.92k op/s 2024-04-22T06:48:52.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:52 smithi190 ceph-mon[31421]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 898 MiB/s wr, 1.92k op/s 2024-04-22T06:48:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:52 smithi079 ceph-mon[27946]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 898 MiB/s wr, 1.92k op/s 2024-04-22T06:48:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:54 smithi193 ceph-mon[31815]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 816 MiB/s wr, 1.83k op/s 2024-04-22T06:48:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:54 smithi190 ceph-mon[31421]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 816 MiB/s wr, 1.83k op/s 2024-04-22T06:48:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:54 smithi079 ceph-mon[27946]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 816 MiB/s wr, 1.83k op/s 2024-04-22T06:48:56.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:56 smithi193 ceph-mon[31815]: pgmap v144: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 693 MiB/s wr, 1.58k op/s 2024-04-22T06:48:56.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:56 smithi190 ceph-mon[31421]: pgmap v144: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 693 MiB/s wr, 1.58k op/s 2024-04-22T06:48:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:56 smithi079 ceph-mon[27946]: pgmap v144: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 693 MiB/s wr, 1.58k op/s 2024-04-22T06:48:58.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:48:58 smithi193 ceph-mon[31815]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 958 MiB/s wr, 1.82k op/s 2024-04-22T06:48:58.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:48:58 smithi190 ceph-mon[31421]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 958 MiB/s wr, 1.82k op/s 2024-04-22T06:48:58.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:48:58 smithi079 ceph-mon[27946]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 958 MiB/s wr, 1.82k op/s 2024-04-22T06:49:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 -- 172.21.15.79:0/3445853855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc1031c0 msgr2=0x7fcfbc1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/3445853855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc1031c0 0x7fcfbc1035a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009900 tx=0x7fcfb002bce0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 -- 172.21.15.79:0/3445853855 shutdown_connections 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/3445853855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfbc10c590 0x7fcfbc10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/3445853855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/3445853855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc1031c0 0x7fcfbc1035a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 -- 172.21.15.79:0/3445853855 >> 172.21.15.79:0/3445853855 conn(0x7fcfbc078060 msgr2=0x7fcfbc078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 -- 172.21.15.79:0/3445853855 shutdown_connections 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.718+0000 7fcfc2c31640 1 -- 172.21.15.79:0/3445853855 wait complete. 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 Processor -- start 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 -- start start 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfbc1031c0 0x7fcfbc19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfbc10c590 0x7fcfbc198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1122d0 con 0x7fcfbc103ae0 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc112150 con 0x7fcfbc1031c0 2024-04-22T06:49:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc2c31640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc112450 con 0x7fcfbc10c590 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfbbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfbbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51306/0 (socket says 172.21.15.79:51306) 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2217521178 learned_addr learned my addr 172.21.15.79:0/2217521178 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.719+0000 7fcfc09cd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfbc1031c0 0x7fcfbc19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2217521178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfbc10c590 msgr2=0x7fcfbc198690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfbc10c590 0x7fcfbc198690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2217521178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfbc1031c0 msgr2=0x7fcfbc19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfbc1031c0 0x7fcfbc19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2217521178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfbc198f50 con 0x7fcfbc103ae0 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc19eac0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800b4a0 tx=0x7fcfa800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfb9ffb640 1 -- 172.21.15.79:0/2217521178 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8014070 con 0x7fcfbc103ae0 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfb9ffb640 1 -- 172.21.15.79:0/2217521178 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfa8010ac0 con 0x7fcfbc103ae0 2024-04-22T06:49:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfbc199240 con 0x7fcfbc103ae0 2024-04-22T06:49:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.720+0000 7fcfb9ffb640 1 -- 172.21.15.79:0/2217521178 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8017c30 con 0x7fcfbc103ae0 2024-04-22T06:49:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.721+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfbc1a5500 con 0x7fcfbc103ae0 2024-04-22T06:49:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.724+0000 7fcfb9ffb640 1 -- 172.21.15.79:0/2217521178 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcfa8017070 con 0x7fcfbc103ae0 2024-04-22T06:49:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.724+0000 7fcfc5823740 1 Processor -- start 2024-04-22T06:49:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.724+0000 7fcfc5823740 1 -- start start 2024-04-22T06:49:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.724+0000 7fcfb9ffb640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcfac0a3dc0 0x7fcfac0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a3e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 0x5646d0a3efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 0x5646d0a4e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc5823740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5646d09a38d0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc5823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5646d0940e80 con 0x5646d0a4c4f0 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc5823740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5646d099f890 con 0x5646d0a3eb60 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc11ce640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcfac0a3dc0 0x7fcfac0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfb9ffb640 1 -- 172.21.15.79:0/2217521178 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fcfa80cfc50 con 0x7fcfbc103ae0 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 0x5646d0a3efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 0x5646d0a3efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:33070/0 (socket says 172.21.15.79:33070) 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2389492008 learned_addr learned my addr 172.21.15.79:0/2389492008 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a3e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfc11ce640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcfac0a3dc0 0x7fcfac0a6280 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400ab70 tx=0x7fcfb4009290 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2389492008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 msgr2=0x5646d0a4e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 0x5646d0a4e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2389492008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 msgr2=0x5646d0a3e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a3e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.725+0000 7fcfbbfff640 1 -- 172.21.15.79:0/2389492008 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646d0962320 con 0x5646d0a3eb60 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.726+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 0x5646d0a3efe0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcfa809bac0 tx=0x7fcfa80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c3b3d66d98dd029 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.726+0000 7fcfa6ffd640 1 -- 172.21.15.79:0/2389492008 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa809ce60 con 0x5646d0a3eb60 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.726+0000 7fcfa6ffd640 1 -- 172.21.15.79:0/2389492008 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfa809fee0 con 0x5646d0a3eb60 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.726+0000 7fcfa6ffd640 1 -- 172.21.15.79:0/2389492008 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa801e050 con 0x5646d0a3eb60 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 -- 172.21.15.79:0/2389492008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 msgr2=0x5646d0a3efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 0x5646d0a3efe0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcfa809bac0 tx=0x7fcfa80046c0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 -- 172.21.15.79:0/2389492008 shutdown_connections 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 0x5646d0a4e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3eb60 0x5646d0a3efe0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2389492008 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a3e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 -- 172.21.15.79:0/2389492008 >> 172.21.15.79:0/2389492008 conn(0x5646d0a05c90 msgr2=0x5646d0a39850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 -- 172.21.15.79:0/2389492008 shutdown_connections 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 -- 172.21.15.79:0/2389492008 wait complete. 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 -- start start 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.727+0000 7fcfc5823740 1 Processor -- start 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a4bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3f8d0 0x5646d0a49260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 0x5646d0a49870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5646d0a4f220 con 0x5646d0a3eb60 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5646d092beb0 con 0x5646d0a4c4f0 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5646d0a4f3a0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc5823740 10 client.0 _ll_register_callbacks cb 0x5646d0958b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a4bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a4bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51324/0 (socket says 172.21.15.79:51324) 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 -- 172.21.15.79:0/3216096764 learned_addr learned my addr 172.21.15.79:0/3216096764 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 -- 172.21.15.79:0/3216096764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3f8d0 msgr2=0x5646d0a49260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3f8d0 0x5646d0a49260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 -- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 msgr2=0x5646d0a49870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 0x5646d0a49870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 -- 172.21.15.79:0/3216096764 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646d06b6210 con 0x5646d0a3eb60 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a4bed0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009900 tx=0x7fcfb002dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ed47b10049aac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.728+0000 7fcf9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcf9bfff640 1 -- 172.21.15.79:0/3216096764 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0044070 con 0x5646d0a3eb60 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcf9bfff640 1 -- 172.21.15.79:0/3216096764 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fcfb002ddb0 con 0x5646d0a3eb60 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa809fee0 con 0x5646d0a3eb60 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcf9bfff640 1 -- 172.21.15.79:0/3216096764 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcfb0050050 con 0x5646d0a3eb60 2024-04-22T06:49:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 10 client.16836 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5646d0782620 con 0x5646d0a3eb60 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcf9bfff640 1 -- 172.21.15.79:0/3216096764 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfa809fee0 con 0x5646d0a3eb60 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcf9bfff640 1 -- 172.21.15.79:0/3216096764 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fcfb00035f0 con 0x5646d0a3eb60 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 10 client.16836 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 20 client.16836 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0a5c5e0 0x5646d0a5ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 4 client.16836 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.729+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646d09b7880 con 0x5646d0a5c5e0 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.731+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0a5c5e0 0x5646d0a5ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.732+0000 7fcf9bfff640 10 client.16836 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.732+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0a5c5e0 0x5646d0a5ea30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c91c564fa266f36 server_cookie=63c2b5e1795ca642 in_seq=0 out_seq=0 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.735+0000 7fcf9bfff640 1 -- 172.21.15.79:0/3216096764 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646d09b7880 con 0x5646d0a5c5e0 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.735+0000 7fcf9bfff640 10 client.16836 handle_command_reply: tid=0 2024-04-22T06:49:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.735+0000 7fcfc5823740 1 client.16836 shutdown 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.735+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 msgr2=0x5646d0a4bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.735+0000 7fcfc5823740 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a4bed0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009900 tx=0x7fcfb002dab0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 shutdown_connections 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0a5c5e0 0x5646d0a5ea30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a4c4f0 0x5646d0a49870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a3f8d0 0x5646d0a49260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 --2- 172.21.15.79:0/3216096764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a4bed0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 >> 172.21.15.79:0/3216096764 conn(0x5646d0a05c90 msgr2=0x5646d0a3b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcf9bfff640 0 client.16836 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 shutdown_connections 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 1 -- 172.21.15.79:0/3216096764 wait complete. 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.736+0000 7fcfc5823740 20 client.16836 trim_cache size 0 max 16384 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 Processor -- start 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 -- start start 2024-04-22T06:49:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a48b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3f8d0 0x5646d0a38750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 0x5646d0a4b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1ae5b0 con 0x5646d0a3eb60 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1abe80 con 0x5646d0a3f8d0 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc5823740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1a9cd0 con 0x5646d0a4c4f0 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a48b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a48b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51336/0 (socket says 172.21.15.79:51336) 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 -- 172.21.15.79:0/2804200853 learned_addr learned my addr 172.21.15.79:0/2804200853 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 -- 172.21.15.79:0/2804200853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 msgr2=0x5646d0a4b990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 0x5646d0a4b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 -- 172.21.15.79:0/2804200853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3f8d0 msgr2=0x5646d0a38750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3f8d0 0x5646d0a38750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.759+0000 7fcfc09cd640 1 -- 172.21.15.79:0/2804200853 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646d0a3e990 con 0x5646d0a3eb60 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.760+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a48b60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0006010 tx=0x7fcfb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab3ac69092beb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.760+0000 7fcfa67fc640 1 -- 172.21.15.79:0/2804200853 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0044040 con 0x5646d0a3eb60 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.760+0000 7fcfa67fc640 1 -- 172.21.15.79:0/2804200853 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfb00035f0 con 0x5646d0a3eb60 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.760+0000 7fcfa67fc640 1 -- 172.21.15.79:0/2804200853 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0049440 con 0x5646d0a3eb60 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 -- 172.21.15.79:0/2804200853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 msgr2=0x5646d0a48b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a48b60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0006010 tx=0x7fcfb0009250 comp rx=0 tx=0).stop 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 -- 172.21.15.79:0/2804200853 shutdown_connections 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3f8d0 0x5646d0a38750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 0x5646d0a4b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 --2- 172.21.15.79:0/2804200853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3eb60 0x5646d0a48b60 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 -- 172.21.15.79:0/2804200853 >> 172.21.15.79:0/2804200853 conn(0x5646d0a05c90 msgr2=0x5646d0a38040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 -- 172.21.15.79:0/2804200853 shutdown_connections 2024-04-22T06:49:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 -- 172.21.15.79:0/2804200853 wait complete. 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 -- start start 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.761+0000 7fcfc5823740 1 Processor -- start 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3eb60 0x5646d0a54850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a45af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 0x5646d0a46030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1aa340 con 0x5646d0a3f8d0 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1ab7b0 con 0x5646d0a3eb60 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfbc1aaa10 con 0x5646d0a4c4f0 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc5823740 10 client.0 _ll_register_callbacks cb 0x7fcfb0005e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a45af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a45af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51350/0 (socket says 172.21.15.79:51350) 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 -- 172.21.15.79:0/4168194583 learned_addr learned my addr 172.21.15.79:0/4168194583 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3eb60 0x5646d0a54850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 -- 172.21.15.79:0/4168194583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 msgr2=0x5646d0a46030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 0x5646d0a46030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 -- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3eb60 msgr2=0x5646d0a54850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3eb60 0x5646d0a54850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfbbfff640 1 -- 172.21.15.79:0/4168194583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646d0a49be0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.762+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3eb60 0x5646d0a54850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcfbbfff640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a45af0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80cd7d0 tx=0x7fcfa80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462159df56a265fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcf9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcf9bfff640 1 -- 172.21.15.79:0/4168194583 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa801e070 con 0x5646d0a3f8d0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcf9bfff640 1 -- 172.21.15.79:0/4168194583 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fcfa8010c70 con 0x5646d0a3f8d0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb00035f0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcf9bfff640 1 -- 172.21.15.79:0/4168194583 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcfa809ce60 con 0x5646d0a3f8d0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcfc5823740 10 client.16842 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5646d0ac27a0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.763+0000 7fcf9bfff640 1 -- 172.21.15.79:0/4168194583 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfb00035f0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcf9bfff640 1 -- 172.21.15.79:0/4168194583 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fcfa80981a0 con 0x5646d0a3f8d0 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc5823740 10 client.16842 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc5823740 10 client.16842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc5823740 20 client.16842 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc5823740 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0ac2ba0 0x5646d0ac4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc5823740 4 client.16842 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5646d09433d0 con 0x5646d0ac2ba0 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0ac2ba0 0x5646d0ac4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.764+0000 7fcf9bfff640 10 client.16842 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:00.765+0000 7fcfc09cd640 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0ac2ba0 0x5646d0ac4f90 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8c4c4115965f0e server_cookie=b39e8dc10e618239 in_seq=0 out_seq=0 2024-04-22T06:49:00.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:00 smithi193 ceph-mon[31815]: pgmap v146: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 645 MiB/s wr, 1.15k op/s 2024-04-22T06:49:00.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:00 smithi190 ceph-mon[31421]: pgmap v146: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 645 MiB/s wr, 1.15k op/s 2024-04-22T06:49:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:00 smithi079 ceph-mon[27946]: pgmap v146: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 645 MiB/s wr, 1.15k op/s 2024-04-22T06:49:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:02 smithi193 ceph-mon[31815]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 807 MiB/s wr, 1.30k op/s 2024-04-22T06:49:02.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:02 smithi190 ceph-mon[31421]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 807 MiB/s wr, 1.30k op/s 2024-04-22T06:49:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:02 smithi079 ceph-mon[27946]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 807 MiB/s wr, 1.30k op/s 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcf9bfff640 1 -- 172.21.15.79:0/4168194583 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5646d09433d0 con 0x5646d0ac2ba0 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcf9bfff640 10 client.16842 handle_command_reply: tid=0 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 client.16842 shutdown 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 msgr2=0x5646d0a45af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a45af0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80cd7d0 tx=0x7fcfa80047d0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 shutdown_connections 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5646d0ac2ba0 0x5646d0ac4f90 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646d0a4c4f0 0x5646d0a46030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5646d0a3f8d0 0x5646d0a45af0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 --2- 172.21.15.79:0/4168194583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5646d0a3eb60 0x5646d0a54850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 >> 172.21.15.79:0/4168194583 conn(0x5646d0a05c90 msgr2=0x5646d0a388d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcf9bfff640 0 client.16842 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 shutdown_connections 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 1 -- 172.21.15.79:0/4168194583 wait complete. 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.566+0000 7fcfc5823740 20 client.16842 trim_cache size 0 max 16384 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:49:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:49:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.567+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcfac0a3dc0 msgr2=0x7fcfac0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.567+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcfac0a3dc0 0x7fcfac0a6280 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400ab70 tx=0x7fcfb4009290 comp rx=0 tx=0).stop 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.567+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 msgr2=0x7fcfbc19eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.567+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc19eac0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800b4a0 tx=0x7fcfa800b960 comp rx=0 tx=0).stop 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.567+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 shutdown_connections 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.567+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfbc10c590 0x7fcfbc198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.568+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcfbc103ae0 0x7fcfbc19eac0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.568+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcfac0a3dc0 0x7fcfac0a6280 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.568+0000 7fcfc2c31640 1 --2- 172.21.15.79:0/2217521178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfbc1031c0 0x7fcfbc19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.568+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 >> 172.21.15.79:0/2217521178 conn(0x7fcfbc078060 msgr2=0x7fcfbc0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.568+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 shutdown_connections 2024-04-22T06:49:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:03.568+0000 7fcfc2c31640 1 -- 172.21.15.79:0/2217521178 wait complete. 2024-04-22T06:49:04.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:04 smithi193 ceph-mon[31815]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 699 MiB/s wr, 865 op/s 2024-04-22T06:49:04.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:04 smithi190 ceph-mon[31421]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 699 MiB/s wr, 865 op/s 2024-04-22T06:49:04.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:04 smithi079 ceph-mon[27946]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 699 MiB/s wr, 865 op/s 2024-04-22T06:49:06.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:06 smithi193 ceph-mon[31815]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 563 MiB/s wr, 509 op/s 2024-04-22T06:49:06.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:06 smithi190 ceph-mon[31421]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 563 MiB/s wr, 509 op/s 2024-04-22T06:49:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:06 smithi079 ceph-mon[27946]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 563 MiB/s wr, 509 op/s 2024-04-22T06:49:07.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:07.584+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:07.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:07.585+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:49:07.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:07.585+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:49:07.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:07.585+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:49:07.586 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:07.585+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:49:08.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:08 smithi193 ceph-mon[31815]: pgmap v150: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 820 MiB/s wr, 715 op/s 2024-04-22T06:49:08.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:08 smithi190 ceph-mon[31421]: pgmap v150: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 820 MiB/s wr, 715 op/s 2024-04-22T06:49:08.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:08 smithi079 ceph-mon[27946]: pgmap v150: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 820 MiB/s wr, 715 op/s 2024-04-22T06:49:10.554 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.3 2024-04-22T06:49:10.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:49:10.807 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.808+0000 7f4ade7fc640 1 -- 172.21.15.79:0/3179526240 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad402cc90 con 0x7f4ae0109490 2024-04-22T06:49:10.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.809+0000 7f4ae5bef640 1 -- 172.21.15.79:0/3179526240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae0109490 msgr2=0x7f4ae0109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:10.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.809+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/3179526240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae0109490 0x7f4ae0109910 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4009900 tx=0x7f4ad402bce0 comp rx=0 tx=0).stop 2024-04-22T06:49:10.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.814+0000 7f4ae5bef640 1 -- 172.21.15.79:0/3179526240 shutdown_connections 2024-04-22T06:49:10.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.814+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/3179526240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0116fe0 0x7f4ae01193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:10.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.814+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/3179526240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae0109490 0x7f4ae0109910 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:10.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.814+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/3179526240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ae010c680 0x7f4ae0108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:10.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.814+0000 7f4ae5bef640 1 -- 172.21.15.79:0/3179526240 >> 172.21.15.79:0/3179526240 conn(0x7f4ae006e410 msgr2=0x7f4ae00719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:10.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.817+0000 7f4ae5bef640 1 -- 172.21.15.79:0/3179526240 shutdown_connections 2024-04-22T06:49:10.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.817+0000 7f4ae5bef640 1 -- 172.21.15.79:0/3179526240 wait complete. 2024-04-22T06:49:10.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.818+0000 7f4ae5bef640 1 Processor -- start 2024-04-22T06:49:10.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.818+0000 7f4ae5bef640 1 -- start start 2024-04-22T06:49:10.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4ae5bef640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ae0109490 0x7f4ae01155e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:10.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4ae5bef640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 0x7f4ae0115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:10.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4ae5bef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0116fe0 0x7f4ae01a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4ae5bef640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010db60 con 0x7f4ae010c680 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4ae5bef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010d9e0 con 0x7f4ae0116fe0 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4ae5bef640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010dce0 con 0x7f4ae0109490 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4adeffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 0x7f4ae0115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4adeffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 0x7f4ae0115b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46368/0 (socket says 172.21.15.79:46368) 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.819+0000 7f4adeffd640 1 -- 172.21.15.79:0/1484134539 learned_addr learned my addr 172.21.15.79:0/1484134539 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.820+0000 7f4adeffd640 1 -- 172.21.15.79:0/1484134539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ae0109490 msgr2=0x7f4ae01155e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.820+0000 7f4adeffd640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ae0109490 0x7f4ae01155e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.820+0000 7f4adeffd640 1 -- 172.21.15.79:0/1484134539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0116fe0 msgr2=0x7f4ae01a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.820+0000 7f4adeffd640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0116fe0 0x7f4ae01a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.820+0000 7f4adeffd640 1 -- 172.21.15.79:0/1484134539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae01a50d0 con 0x7f4ae010c680 2024-04-22T06:49:10.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.821+0000 7f4adeffd640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 0x7f4ae0115b20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4ad402c220 tx=0x7f4ad40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:10.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.827+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4065020 con 0x7f4ae010c680 2024-04-22T06:49:10.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.827+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae01a5360 con 0x7f4ae010c680 2024-04-22T06:49:10.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.827+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae0074180 con 0x7f4ae010c680 2024-04-22T06:49:10.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.827+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ad4004e00 con 0x7f4ae010c680 2024-04-22T06:49:10.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.827+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40038e0 con 0x7f4ae010c680 2024-04-22T06:49:10.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.830+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4acc005290 con 0x7f4ae010c680 2024-04-22T06:49:10.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.836+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4ad4068020 con 0x7f4ae010c680 2024-04-22T06:49:10.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.838+0000 7f4adcff9640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4ac80ac680 0x7f4ac80aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:10.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.838+0000 7f4adf7fe640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4ac80ac680 0x7f4ac80aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:10.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.840+0000 7f4adf7fe640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4ac80ac680 0x7f4ac80aeb40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00096b0 tx=0x7f4ad0006c20 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:10.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.857+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f4ad40ded20 con 0x7f4ae010c680 2024-04-22T06:49:10.855 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:10.857+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ad40733e0 con 0x7f4ae010c680 2024-04-22T06:49:10.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:10 smithi193 ceph-mon[31815]: pgmap v151: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 593 MiB/s wr, 512 op/s 2024-04-22T06:49:10.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:10 smithi190 ceph-mon[31421]: pgmap v151: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 593 MiB/s wr, 512 op/s 2024-04-22T06:49:10.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:10 smithi079 ceph-mon[27946]: pgmap v151: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 593 MiB/s wr, 512 op/s 2024-04-22T06:49:11.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.391+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4acc005580 con 0x7f4ae010c680 2024-04-22T06:49:11.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.398+0000 7f4adcff9640 1 -- 172.21.15.79:0/1484134539 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11881 (secure 0 0 0) 0x7f4ad40e1380 con 0x7f4ae010c680 2024-04-22T06:49:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:49:11.397 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:48:41.413672+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25559,"mds_2":25552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25810":{"gid":25810,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/1413649941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1413649941},{"type":"v1","addr":"172.21.15.193:6839","nonce":1413649941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25817":{"gid":25817,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1169559599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1169559599},{"type":"v1","addr":"172.21.15.190:6837","nonce":1169559599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,25552,25559]},"id":1}]} 2024-04-22T06:49:11.397 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 47 2024-04-22T06:49:11.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4ac80ac680 msgr2=0x7f4ac80aeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4ac80ac680 0x7f4ac80aeb40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00096b0 tx=0x7f4ad0006c20 comp rx=0 tx=0).stop 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 msgr2=0x7f4ae0115b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 0x7f4ae0115b20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4ad402c220 tx=0x7f4ad40044c0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 shutdown_connections 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4ac80ac680 0x7f4ac80aeb40 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00096b0 tx=0x7f4ad0006c20 comp rx=0 tx=0).stop 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ae0116fe0 0x7f4ae01a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4ae010c680 0x7f4ae0115b20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 --2- 172.21.15.79:0/1484134539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ae0109490 0x7f4ae01155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.403+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 >> 172.21.15.79:0/1484134539 conn(0x7f4ae006e410 msgr2=0x7f4ae01177a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:11.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.404+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 shutdown_connections 2024-04-22T06:49:11.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.409+0000 7f4ae5bef640 1 -- 172.21.15.79:0/1484134539 wait complete. 2024-04-22T06:49:11.437 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 87.3 --await' 2024-04-22T06:49:11.437 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 87.3 --await 2024-04-22T06:49:11.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.681+0000 7f3ec3bca640 1 -- 172.21.15.79:0/1312091313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc110b20 msgr2=0x7f3ebc10e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.685 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.681+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/1312091313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc110b20 0x7f3ebc10e7d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009900 tx=0x7f3eac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:49:11.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.682+0000 7f3ec3bca640 1 -- 172.21.15.79:0/1312091313 shutdown_connections 2024-04-22T06:49:11.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.682+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/1312091313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ebc10eeb0 0x7f3ebc11d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.682+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/1312091313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc110b20 0x7f3ebc10e7d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.682+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/1312091313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.682+0000 7f3ec3bca640 1 -- 172.21.15.79:0/1312091313 >> 172.21.15.79:0/1312091313 conn(0x7f3ebc06d070 msgr2=0x7f3ebc06d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:11.686 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.686+0000 7f3ec3bca640 1 -- 172.21.15.79:0/1312091313 shutdown_connections 2024-04-22T06:49:11.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.692+0000 7f3ec3bca640 1 -- 172.21.15.79:0/1312091313 wait complete. 2024-04-22T06:49:11.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.693+0000 7f3ec3bca640 1 Processor -- start 2024-04-22T06:49:11.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.693+0000 7f3ec3bca640 1 -- start start 2024-04-22T06:49:11.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.694+0000 7f3ec3bca640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc1a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.694+0000 7f3ec3bca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ebc1a7b90 0x7f3ebc1abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.694+0000 7f3ec3bca640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ebc1ac450 0x7f3ebc1ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.694+0000 7f3ec3bca640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc120fd0 con 0x7f3ebc1106b0 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.694+0000 7f3ec3bca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc120e50 con 0x7f3ebc1a7b90 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.694+0000 7f3ec3bca640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc121150 con 0x7f3ebc1ac450 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.696+0000 7f3ec1966640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc1a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.696+0000 7f3ec1966640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc1a7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46380/0 (socket says 172.21.15.79:46380) 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.696+0000 7f3ec1966640 1 -- 172.21.15.79:0/3278667649 learned_addr learned my addr 172.21.15.79:0/3278667649 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:11.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.697+0000 7f3ec2167640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ebc1ac450 0x7f3ebc1ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.699+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ebc1a7b90 0x7f3ebc1abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.699+0000 7f3ec1966640 1 -- 172.21.15.79:0/3278667649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ebc1ac450 msgr2=0x7f3ebc1ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.699+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ebc1ac450 0x7f3ebc1ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.700+0000 7f3ec1966640 1 -- 172.21.15.79:0/3278667649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ebc1a7b90 msgr2=0x7f3ebc1abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.700+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ebc1a7b90 0x7f3ebc1abf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.700+0000 7f3ec1966640 1 -- 172.21.15.79:0/3278667649 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ebc110f10 con 0x7f3ebc1106b0 2024-04-22T06:49:11.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.701+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc1a7650 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800e9f0 tx=0x7f3eb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:11.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.705+0000 7f3eb2ffd640 1 -- 172.21.15.79:0/3278667649 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb800cd80 con 0x7f3ebc1106b0 2024-04-22T06:49:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.705+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ebc111200 con 0x7f3ebc1106b0 2024-04-22T06:49:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.705+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ebc111750 con 0x7f3ebc1106b0 2024-04-22T06:49:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.706+0000 7f3eb2ffd640 1 -- 172.21.15.79:0/3278667649 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eb80108e0 con 0x7f3ebc1106b0 2024-04-22T06:49:11.704 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.706+0000 7f3eb2ffd640 1 -- 172.21.15.79:0/3278667649 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80188b0 con 0x7f3ebc1106b0 2024-04-22T06:49:11.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.718+0000 7f3eb2ffd640 1 -- 172.21.15.79:0/3278667649 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3eb8010430 con 0x7f3ebc1106b0 2024-04-22T06:49:11.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.732+0000 7f3eb2ffd640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3ea80a3f20 0x7f3ea80a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.730 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.732+0000 7f3eb2ffd640 1 -- 172.21.15.79:0/3278667649 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f3eb80d0ae0 con 0x7f3ebc1106b0 2024-04-22T06:49:11.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.733+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3ea80a3f20 0x7f3ea80a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.734+0000 7f3ec67b7740 1 Processor -- start 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 -- start start 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b44da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b502b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55adb8aa50b0 con 0x55adb8b452e0 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55adb8a56680 con 0x55adb8b449c0 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.735+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55adb8a62c60 con 0x55adb8b4dec0 2024-04-22T06:49:11.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.736+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.736+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46388/0 (socket says 172.21.15.79:46388) 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.736+0000 7f3ec1165640 1 -- 172.21.15.79:0/4086803004 learned_addr learned my addr 172.21.15.79:0/4086803004 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.736+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3ea80a3f20 0x7f3ea80a63e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3eac005c40 tx=0x7f3eac005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.737+0000 7f3ec1165640 1 -- 172.21.15.79:0/4086803004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 msgr2=0x55adb8b502b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.737+0000 7f3ec1165640 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b502b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.737+0000 7f3ec1165640 1 -- 172.21.15.79:0/4086803004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 msgr2=0x55adb8b44da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.737+0000 7f3ec1165640 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b44da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.737+0000 7f3ec1165640 1 -- 172.21.15.79:0/4086803004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adb8887c90 con 0x55adb8b452e0 2024-04-22T06:49:11.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.741+0000 7f3ec1165640 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4d980 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009900 tx=0x7f3eac044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d6d863d4d61303 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:11.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:11 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1484134539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.742+0000 7f3ea7fff640 1 -- 172.21.15.79:0/4086803004 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac0042e0 con 0x55adb8b452e0 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.742+0000 7f3ea7fff640 1 -- 172.21.15.79:0/4086803004 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eac04f070 con 0x55adb8b452e0 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.743+0000 7f3ec67b7740 1 -- 172.21.15.79:0/4086803004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 msgr2=0x55adb8b4d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.743+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4d980 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009900 tx=0x7f3eac044000 comp rx=0 tx=0).stop 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.745+0000 7f3ec67b7740 1 -- 172.21.15.79:0/4086803004 shutdown_connections 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.745+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.745+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4d980 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.745+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/4086803004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b44da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.744 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.745+0000 7f3ec67b7740 1 -- 172.21.15.79:0/4086803004 >> 172.21.15.79:0/4086803004 conn(0x55adb8b3f0f0 msgr2=0x55adb8b3f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:11.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.751+0000 7f3ec67b7740 1 -- 172.21.15.79:0/4086803004 shutdown_connections 2024-04-22T06:49:11.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.751+0000 7f3ec67b7740 1 -- 172.21.15.79:0/4086803004 wait complete. 2024-04-22T06:49:11.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.752+0000 7f3ec67b7740 1 -- start start 2024-04-22T06:49:11.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.752+0000 7f3ec67b7740 1 Processor -- start 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.753+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b4afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.753+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.753+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b4ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.753+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55adb8aa8400 con 0x55adb8b452e0 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.754+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55adb8a64840 con 0x55adb8b449c0 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.754+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55adb8aad720 con 0x55adb8b4dec0 2024-04-22T06:49:11.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.754+0000 7f3ec67b7740 10 client.0 _ll_register_callbacks cb 0x55adb8ab5ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:11.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.755+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.755+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46390/0 (socket says 172.21.15.79:46390) 2024-04-22T06:49:11.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.755+0000 7f3ec1165640 1 -- 172.21.15.79:0/3348922785 learned_addr learned my addr 172.21.15.79:0/3348922785 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:11.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.756+0000 7f3ec1165640 1 -- 172.21.15.79:0/3348922785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 msgr2=0x55adb8b4ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:11.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.756+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b4ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.756+0000 7f3ec1165640 1 -- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 msgr2=0x55adb8b4afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:11.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.756+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b4afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.756+0000 7f3ec1165640 1 -- 172.21.15.79:0/3348922785 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adb88e8420 con 0x55adb8b452e0 2024-04-22T06:49:11.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.757+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4b520 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0096c0 tx=0x7f3eac004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b1e9a61800373d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:11.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.758+0000 7f3ea57fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:11.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.758+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3348922785 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac0053d0 con 0x55adb8b452e0 2024-04-22T06:49:11.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.758+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3348922785 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f3eac005570 con 0x55adb8b452e0 2024-04-22T06:49:11.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.759+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55adb8ab6b10 con 0x55adb8b452e0 2024-04-22T06:49:11.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.760+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3348922785 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3eac0035f0 con 0x55adb8b452e0 2024-04-22T06:49:11.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.760+0000 7f3ec67b7740 10 client.16860 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:11.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.760+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55adb877b580 con 0x55adb8b452e0 2024-04-22T06:49:11.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3348922785 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55adb8ab6b10 con 0x55adb8b452e0 2024-04-22T06:49:11.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3348922785 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7f3eac0408d0 con 0x55adb8b452e0 2024-04-22T06:49:11.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ec67b7740 10 client.16860 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:11.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ec67b7740 10 client.16860 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:49:11.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ec67b7740 20 client.16860 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:11.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8b5ddf0 0x55adb8b60270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ec67b7740 4 client.16860 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:11.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.765+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adb8ab0980 con 0x55adb8b5ddf0 2024-04-22T06:49:11.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.767+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8b5ddf0 0x55adb8b60270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.771+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8b5ddf0 0x55adb8b60270 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43a85208b0a35bf4 server_cookie=c83231878dde0582 in_seq=0 out_seq=0 2024-04-22T06:49:11.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.774+0000 7f3ea57fa640 10 client.16860 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:11.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.779+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3348922785 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adb8ab0980 con 0x55adb8b5ddf0 2024-04-22T06:49:11.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.779+0000 7f3ea57fa640 10 client.16860 handle_command_reply: tid=0 2024-04-22T06:49:11.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.779+0000 7f3ec67b7740 1 client.16860 shutdown 2024-04-22T06:49:11.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.783+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 msgr2=0x55adb8b4b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.783+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4b520 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0096c0 tx=0x7f3eac004590 comp rx=0 tx=0).stop 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.787+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 shutdown_connections 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.787+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8b5ddf0 0x55adb8b60270 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.787+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b4ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.787+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b4b520 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.787+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3348922785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b4afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.787+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 >> 172.21.15.79:0/3348922785 conn(0x55adb8b3f0f0 msgr2=0x55adb8b40e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:11.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.788+0000 7f3ea57fa640 0 client.16860 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:11.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.789+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 shutdown_connections 2024-04-22T06:49:11.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.790+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3348922785 wait complete. 2024-04-22T06:49:11.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.790+0000 7f3ec67b7740 20 client.16860 trim_cache size 0 max 16384 2024-04-22T06:49:11.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.838+0000 7f3ec67b7740 1 Processor -- start 2024-04-22T06:49:11.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 -- start start 2024-04-22T06:49:11.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b4d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b59990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b4b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000ed90 con 0x55adb8b452e0 2024-04-22T06:49:11.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000cbe0 con 0x55adb8b449c0 2024-04-22T06:49:11.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.839+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000e6c0 con 0x55adb8b4dec0 2024-04-22T06:49:11.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b59990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b59990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46406/0 (socket says 172.21.15.79:46406) 2024-04-22T06:49:11.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 -- 172.21.15.79:0/977110098 learned_addr learned my addr 172.21.15.79:0/977110098 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 -- 172.21.15.79:0/977110098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 msgr2=0x55adb8b4b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b4b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 -- 172.21.15.79:0/977110098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 msgr2=0x55adb8b4d710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b4d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 -- 172.21.15.79:0/977110098 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adb8b50620 con 0x55adb8b452e0 2024-04-22T06:49:11.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.841+0000 7f3ec1165640 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b59990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3eac036040 tx=0x7f3eac02c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1d524e37990831 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.842+0000 7f3ea77fe640 1 -- 172.21.15.79:0/977110098 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac065070 con 0x55adb8b452e0 2024-04-22T06:49:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.842+0000 7f3ea77fe640 1 -- 172.21.15.79:0/977110098 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eac058070 con 0x55adb8b452e0 2024-04-22T06:49:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.842+0000 7f3ea77fe640 1 -- 172.21.15.79:0/977110098 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac005610 con 0x55adb8b452e0 2024-04-22T06:49:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 -- 172.21.15.79:0/977110098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 msgr2=0x55adb8b59990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b59990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3eac036040 tx=0x7f3eac02c200 comp rx=0 tx=0).stop 2024-04-22T06:49:11.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 -- 172.21.15.79:0/977110098 shutdown_connections 2024-04-22T06:49:11.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b4b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b59990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/977110098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b4d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.843+0000 7f3ec67b7740 1 -- 172.21.15.79:0/977110098 >> 172.21.15.79:0/977110098 conn(0x55adb8b3f0f0 msgr2=0x55adb8b4afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:11.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.844+0000 7f3ec67b7740 1 -- 172.21.15.79:0/977110098 shutdown_connections 2024-04-22T06:49:11.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.844+0000 7f3ec67b7740 1 -- 172.21.15.79:0/977110098 wait complete. 2024-04-22T06:49:11.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.844+0000 7f3ec67b7740 1 -- start start 2024-04-22T06:49:11.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.844+0000 7f3ec67b7740 1 Processor -- start 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b56100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b47850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b47d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000d920 con 0x55adb8b452e0 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000d250 con 0x55adb8b449c0 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea000dff0 con 0x55adb8b4dec0 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec67b7740 10 client.0 _ll_register_callbacks cb 0x7f3eac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b47850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b47850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46416/0 (socket says 172.21.15.79:46416) 2024-04-22T06:49:11.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 -- 172.21.15.79:0/3524630300 learned_addr learned my addr 172.21.15.79:0/3524630300 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:11.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 -- 172.21.15.79:0/3524630300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 msgr2=0x55adb8b47d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:11.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b47d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 -- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 msgr2=0x55adb8b56100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b56100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.845+0000 7f3ec1165640 1 -- 172.21.15.79:0/3524630300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adb8b48400 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ec1165640 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b47850 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0041f0 tx=0x7f3eac040a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838a96fc7154eb50 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ea57fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3524630300 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac065040 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3524630300 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f3eac0039a0 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eac058070 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3524630300 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3eac074050 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ec67b7740 10 client.16866 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.846+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55adb8bc4200 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3524630300 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eac058070 con 0x55adb8b452e0 2024-04-22T06:49:11.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3524630300 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7f3eac04a9b0 con 0x55adb8b452e0 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ec67b7740 10 client.16866 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ec67b7740 10 client.16866 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ec67b7740 20 client.16866 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8bc4970 0x55adb8bc6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ec67b7740 4 client.16866 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.3, "await": true}] 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.847+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.3, "await": true}) -- 0x55adb8a2cd30 con 0x55adb8bc4970 2024-04-22T06:49:11.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.849+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8bc4970 0x55adb8bc6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:11.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.853+0000 7f3ea57fa640 10 client.16866 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:11.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:11.853+0000 7f3ec1966640 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8bc4970 0x55adb8bc6d90 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96d6c6b981f24f7 server_cookie=9e8ad157dd42f99b in_seq=0 out_seq=0 2024-04-22T06:49:11.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:11 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1484134539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:49:11.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:11 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1484134539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:49:12.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ea57fa640 1 -- 172.21.15.79:0/3524630300 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55adb8a2cd30 con 0x55adb8bc4970 2024-04-22T06:49:12.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ea57fa640 10 client.16866 handle_command_reply: tid=0 2024-04-22T06:49:12.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 client.16866 shutdown 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 msgr2=0x55adb8b47850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b47850 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0041f0 tx=0x7f3eac040a30 comp rx=0 tx=0).stop 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 shutdown_connections 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55adb8bc4970 0x55adb8bc6d90 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55adb8b4dec0 0x55adb8b47d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55adb8b452e0 0x55adb8b47850 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ea57fa640 0 client.16866 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 --2- 172.21.15.79:0/3524630300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55adb8b449c0 0x55adb8b56100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 >> 172.21.15.79:0/3524630300 conn(0x55adb8b3f0f0 msgr2=0x55adb8b3e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 shutdown_connections 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 1 -- 172.21.15.79:0/3524630300 wait complete. 2024-04-22T06:49:12.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.585+0000 7f3ec67b7740 20 client.16866 trim_cache size 0 max 16384 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 47, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 16, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "fdf32dfd": { 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 16, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 87.3, 2024-04-22T06:49:12.586 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:12.587 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3ea80a3f20 msgr2=0x7f3ea80a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3ea80a3f20 0x7f3ea80a63e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3eac005c40 tx=0x7f3eac005fd0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 msgr2=0x7f3ebc1a7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc1a7650 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800e9f0 tx=0x7f3eb800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 shutdown_connections 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f3ea80a3f20 0x7f3ea80a63e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ebc1a7b90 0x7f3ebc1abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ebc1ac450 0x7f3ebc1ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 --2- 172.21.15.79:0/3278667649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3ebc1106b0 0x7f3ebc1a7650 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 >> 172.21.15.79:0/3278667649 conn(0x7f3ebc06d070 msgr2=0x7f3ebc11a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:12.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 shutdown_connections 2024-04-22T06:49:12.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:12.586+0000 7f3ec3bca640 1 -- 172.21.15.79:0/3278667649 wait complete. 2024-04-22T06:49:12.595 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: fdf32dfd, quiesce took 2.0 seconds. Will release after: 27.3 2024-04-22T06:49:12.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:12 smithi193 ceph-mon[31815]: pgmap v152: 97 pgs: 97 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 720 MiB/s wr, 1.43k op/s 2024-04-22T06:49:12.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:12 smithi190 ceph-mon[31421]: pgmap v152: 97 pgs: 97 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 720 MiB/s wr, 1.43k op/s 2024-04-22T06:49:12.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:12 smithi079 ceph-mon[27946]: pgmap v152: 97 pgs: 97 active+clean; 31 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 720 MiB/s wr, 1.43k op/s 2024-04-22T06:49:14.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:14 smithi190 ceph-mon[31421]: pgmap v153: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 667 MiB/s wr, 2.06k op/s 2024-04-22T06:49:14.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:14 smithi079 ceph-mon[27946]: pgmap v153: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 667 MiB/s wr, 2.06k op/s 2024-04-22T06:49:15.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:14 smithi193 ceph-mon[31815]: pgmap v153: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 667 MiB/s wr, 2.06k op/s 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 -- 172.21.15.79:0/1651040644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601017f0 msgr2=0x7fa560101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 --2- 172.21.15.79:0/1651040644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601017f0 0x7fa560101bd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa550009900 tx=0x7fa55002bc30 comp rx=0 tx=0).stop 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 -- 172.21.15.79:0/1651040644 shutdown_connections 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 --2- 172.21.15.79:0/1651040644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5601063a0 0x7fa56010df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 --2- 172.21.15.79:0/1651040644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601021a0 0x7fa560105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 --2- 172.21.15.79:0/1651040644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601017f0 0x7fa560101bd0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 -- 172.21.15.79:0/1651040644 >> 172.21.15.79:0/1651040644 conn(0x7fa560077370 msgr2=0x7fa5600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 -- 172.21.15.79:0/1651040644 shutdown_connections 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.667+0000 7fa565b15640 1 -- 172.21.15.79:0/1651040644 wait complete. 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 Processor -- start 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 -- start start 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601017f0 0x7fa56019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 0x7fa56019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5601977a0 0x7fa560197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601113e0 con 0x7fa5601063a0 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa560111260 con 0x7fa5601977a0 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa565b15640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa560111560 con 0x7fa5601017f0 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa55effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 0x7fa56019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa55effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 0x7fa56019dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39612/0 (socket says 172.21.15.79:39612) 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.668+0000 7fa55effd640 1 -- 172.21.15.79:0/2599876702 learned_addr learned my addr 172.21.15.79:0/2599876702 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601017f0 0x7fa56019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55effd640 1 -- 172.21.15.79:0/2599876702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601017f0 msgr2=0x7fa56019d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55effd640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601017f0 0x7fa56019d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55effd640 1 -- 172.21.15.79:0/2599876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5601977a0 msgr2=0x7fa560197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5601977a0 0x7fa560197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55effd640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5601977a0 0x7fa560197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55effd640 1 -- 172.21.15.79:0/2599876702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5601983a0 con 0x7fa5601063a0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601017f0 0x7fa56019d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55effd640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 0x7fa56019dbd0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa55400b450 tx=0x7fa55400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55cff9640 1 -- 172.21.15.79:0/2599876702 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554015020 con 0x7fa5601063a0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5601a4430 con 0x7fa5601063a0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55cff9640 1 -- 172.21.15.79:0/2599876702 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa554011d40 con 0x7fa5601063a0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa55cff9640 1 -- 172.21.15.79:0/2599876702 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55400f340 con 0x7fa5601063a0 2024-04-22T06:49:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.669+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5601a4830 con 0x7fa5601063a0 2024-04-22T06:49:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.671+0000 7fa55cff9640 1 -- 172.21.15.79:0/2599876702 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa554018020 con 0x7fa5601063a0 2024-04-22T06:49:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 Processor -- start 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 -- start start 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b775830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b7761f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b7816e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa55cff9640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa5480a3dc0 0x7fa5480a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56328b6d88d0 con 0x56328b77f2f0 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56328b675e80 con 0x56328b776ae0 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa5687dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56328b6d4890 con 0x56328b775d70 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa55ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b7816e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa55effd640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa5480a3dc0 0x7fa5480a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa55ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b7816e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39624/0 (socket says 172.21.15.79:39624) 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa55ffff640 1 -- 172.21.15.79:0/2020563880 learned_addr learned my addr 172.21.15.79:0/2020563880 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.672+0000 7fa55cff9640 1 -- 172.21.15.79:0/2599876702 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fa5540d0410 con 0x7fa5601063a0 2024-04-22T06:49:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55ffff640 1 -- 172.21.15.79:0/2020563880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 msgr2=0x56328b7761f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b7761f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55ffff640 1 -- 172.21.15.79:0/2020563880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 msgr2=0x56328b775830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b775830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55ffff640 1 -- 172.21.15.79:0/2020563880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56328b697320 con 0x56328b77f2f0 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b7816e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa558008ea0 tx=0x7fa55800f590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7476a8e7e2bd4ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa55effd640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa5480a3dc0 0x7fa5480a6280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5540cdb50 tx=0x7fa55401c040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa545ffb640 1 -- 172.21.15.79:0/2020563880 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558013410 con 0x56328b77f2f0 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa545ffb640 1 -- 172.21.15.79:0/2020563880 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa558013a60 con 0x56328b77f2f0 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.673+0000 7fa545ffb640 1 -- 172.21.15.79:0/2020563880 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558012a80 con 0x56328b77f2f0 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 -- 172.21.15.79:0/2020563880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 msgr2=0x56328b7816e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b7816e0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa558008ea0 tx=0x7fa55800f590 comp rx=0 tx=0).stop 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 -- 172.21.15.79:0/2020563880 shutdown_connections 2024-04-22T06:49:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b7816e0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b7761f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2020563880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b775830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 -- 172.21.15.79:0/2020563880 >> 172.21.15.79:0/2020563880 conn(0x56328b73ac90 msgr2=0x56328b770a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 -- 172.21.15.79:0/2020563880 shutdown_connections 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 -- 172.21.15.79:0/2020563880 wait complete. 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 -- start start 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.674+0000 7fa5687dd740 1 Processor -- start 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b77c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b77c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56328b782020 con 0x56328b77f2f0 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56328b660eb0 con 0x56328b776ae0 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56328b7821a0 con 0x56328b775d70 2024-04-22T06:49:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 10 client.0 _ll_register_callbacks cb 0x56328b68db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b77c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b77c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39636/0 (socket says 172.21.15.79:39636) 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 -- 172.21.15.79:0/2162501501 learned_addr learned my addr 172.21.15.79:0/2162501501 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b77c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 -- 172.21.15.79:0/2162501501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 msgr2=0x56328b77ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 -- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 msgr2=0x56328b77c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b77c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 -- 172.21.15.79:0/2162501501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56328b3eb210 con 0x56328b77f2f0 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b77c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa55ffff640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b77c680 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa558008620 tx=0x7fa5580126c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c5c09b4abd088f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa53effd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa53effd640 1 -- 172.21.15.79:0/2162501501 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558024020 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa53effd640 1 -- 172.21.15.79:0/2162501501 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fa558013410 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.675+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa558013a60 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa53effd640 1 -- 172.21.15.79:0/2162501501 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa558013820 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 10 client.16878 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56328b4b7620 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa53effd640 1 -- 172.21.15.79:0/2162501501 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa558013a60 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa53effd640 1 -- 172.21.15.79:0/2162501501 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fa55801f440 con 0x56328b77f2f0 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 10 client.16878 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 10 client.16878 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 20 client.16878 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b78f310 0x56328b791700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 4 client.16878 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56328b6ec880 con 0x56328b78f310 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.676+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b78f310 0x56328b791700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.677+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b78f310 0x56328b791700 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e5028aec7feb6ed server_cookie=18890e22b9da59f0 in_seq=0 out_seq=0 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.677+0000 7fa53effd640 10 client.16878 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa53effd640 1 -- 172.21.15.79:0/2162501501 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56328b6ec880 con 0x56328b78f310 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa53effd640 10 client.16878 handle_command_reply: tid=0 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 client.16878 shutdown 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 msgr2=0x56328b77c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b77c680 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa558008620 tx=0x7fa5580126c0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 shutdown_connections 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b78f310 0x56328b791700 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b77c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b77c680 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 --2- 172.21.15.79:0/2162501501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa53effd640 0 client.16878 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 >> 172.21.15.79:0/2162501501 conn(0x56328b73ac90 msgr2=0x56328b7727b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 shutdown_connections 2024-04-22T06:49:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 1 -- 172.21.15.79:0/2162501501 wait complete. 2024-04-22T06:49:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.678+0000 7fa5687dd740 20 client.16878 trim_cache size 0 max 16384 2024-04-22T06:49:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 Processor -- start 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 -- start start 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b78af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 0x56328b77e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b77f2f0 0x56328b77ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601ae730 con 0x56328b776ae0 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601ac000 con 0x56328b77f2f0 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa5687dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601a9e50 con 0x56328b775d70 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 0x56328b77e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 0x56328b77e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39642/0 (socket says 172.21.15.79:39642) 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 -- 172.21.15.79:0/4078366363 learned_addr learned my addr 172.21.15.79:0/4078366363 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55ffff640 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b77f2f0 0x56328b77ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55ffff640 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b78af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 -- 172.21.15.79:0/4078366363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 msgr2=0x56328b78af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b78af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 -- 172.21.15.79:0/4078366363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b77f2f0 msgr2=0x56328b77ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b77f2f0 0x56328b77ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.693+0000 7fa55f7fe640 1 -- 172.21.15.79:0/4078366363 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56328b76a2a0 con 0x56328b776ae0 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 0x56328b77e7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa550005ab0 tx=0x7fa550000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6131a1da7e652c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5457fa640 1 -- 172.21.15.79:0/4078366363 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500042c0 con 0x56328b776ae0 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5457fa640 1 -- 172.21.15.79:0/4078366363 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa550048070 con 0x56328b776ae0 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5457fa640 1 -- 172.21.15.79:0/4078366363 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550043250 con 0x56328b776ae0 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 -- 172.21.15.79:0/4078366363 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 msgr2=0x56328b77e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 0x56328b77e7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa550005ab0 tx=0x7fa550000bc0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 -- 172.21.15.79:0/4078366363 shutdown_connections 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b77f2f0 0x56328b77ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b776ae0 0x56328b77e7a0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 --2- 172.21.15.79:0/4078366363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b78af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 -- 172.21.15.79:0/4078366363 >> 172.21.15.79:0/4078366363 conn(0x56328b73ac90 msgr2=0x56328b770590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 -- 172.21.15.79:0/4078366363 shutdown_connections 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.694+0000 7fa5687dd740 1 -- 172.21.15.79:0/4078366363 wait complete. 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 -- start start 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 Processor -- start 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b780230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b786460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601aa4c0 con 0x56328b77f2f0 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601ab930 con 0x56328b776ae0 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5601aab90 con 0x56328b775d70 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa5687dd740 10 client.0 _ll_register_callbacks cb 0x7fa558008020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa55ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b786460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa55ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b786460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39654/0 (socket says 172.21.15.79:39654) 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.695+0000 7fa55ffff640 1 -- 172.21.15.79:0/1405132235 learned_addr learned my addr 172.21.15.79:0/1405132235 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b780230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55ffff640 1 -- 172.21.15.79:0/1405132235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 msgr2=0x56328b77fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55ffff640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55ffff640 1 -- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 msgr2=0x56328b780230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55ffff640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b780230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55ffff640 1 -- 172.21.15.79:0/1405132235 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56328b786a50 con 0x56328b77f2f0 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b780230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa55ffff640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b786460 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa558008ea0 tx=0x7fa55800fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1505beb9044223b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa53effd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa53effd640 1 -- 172.21.15.79:0/1405132235 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558024070 con 0x56328b77f2f0 2024-04-22T06:49:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa53effd640 1 -- 172.21.15.79:0/1405132235 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fa558018040 con 0x56328b77f2f0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa550048070 con 0x56328b77f2f0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa53effd640 1 -- 172.21.15.79:0/1405132235 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa558008750 con 0x56328b77f2f0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa5687dd740 10 client.16884 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56328b787d40 con 0x56328b77f2f0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa53effd640 1 -- 172.21.15.79:0/1405132235 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa550048070 con 0x56328b77f2f0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.696+0000 7fa53effd640 1 -- 172.21.15.79:0/1405132235 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fa55801fb30 con 0x56328b77f2f0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa5687dd740 10 client.16884 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa5687dd740 10 client.16884 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa5687dd740 20 client.16884 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa5687dd740 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b7f5ac0 0x56328b7f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa5687dd740 4 client.16884 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56328b6783d0 con 0x56328b7f5ac0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b7f5ac0 0x56328b7f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa55f7fe640 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b7f5ac0 0x56328b7f7ee0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c872fa8ac78033d server_cookie=df9755bf12a33db2 in_seq=0 out_seq=0 2024-04-22T06:49:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.697+0000 7fa53effd640 10 client.16884 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa53effd640 1 -- 172.21.15.79:0/1405132235 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56328b6783d0 con 0x56328b7f5ac0 2024-04-22T06:49:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa53effd640 10 client.16884 handle_command_reply: tid=0 2024-04-22T06:49:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 client.16884 shutdown 2024-04-22T06:49:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 msgr2=0x56328b786460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b786460 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa558008ea0 tx=0x7fa55800fbd0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 shutdown_connections 2024-04-22T06:49:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x56328b7f5ac0 0x56328b7f7ee0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56328b776ae0 0x56328b780230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56328b77f2f0 0x56328b786460 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 --2- 172.21.15.79:0/1405132235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56328b775d70 0x56328b77fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 >> 172.21.15.79:0/1405132235 conn(0x56328b73ac90 msgr2=0x56328b76d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa53effd640 0 client.16884 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.714+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 shutdown_connections 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.715+0000 7fa5687dd740 1 -- 172.21.15.79:0/1405132235 wait complete. 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.715+0000 7fa5687dd740 20 client.16884 trim_cache size 0 max 16384 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:49:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.715+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa5480a3dc0 msgr2=0x7fa5480a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.715+0000 7fa565b15640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa5480a3dc0 0x7fa5480a6280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5540cdb50 tx=0x7fa55401c040 comp rx=0 tx=0).stop 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.715+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 msgr2=0x7fa56019dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.715+0000 7fa565b15640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 0x7fa56019dbd0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa55400b450 tx=0x7fa55400b910 comp rx=0 tx=0).stop 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 shutdown_connections 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa5480a3dc0 0x7fa5480a6280 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5601977a0 0x7fa560197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5601063a0 0x7fa56019dbd0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 --2- 172.21.15.79:0/2599876702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5601017f0 0x7fa56019d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 >> 172.21.15.79:0/2599876702 conn(0x7fa560077370 msgr2=0x7fa5600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 shutdown_connections 2024-04-22T06:49:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:16.716+0000 7fa565b15640 1 -- 172.21.15.79:0/2599876702 wait complete. 2024-04-22T06:49:16.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:16 smithi190 ceph-mon[31421]: pgmap v154: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 570 MiB/s wr, 1.97k op/s 2024-04-22T06:49:16.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:16 smithi079 ceph-mon[27946]: pgmap v154: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 570 MiB/s wr, 1.97k op/s 2024-04-22T06:49:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:16 smithi193 ceph-mon[31815]: pgmap v154: 97 pgs: 97 active+clean; 32 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 570 MiB/s wr, 1.97k op/s 2024-04-22T06:49:18.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:18 smithi079 ceph-mon[27946]: pgmap v155: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 MiB/s wr, 3.11k op/s 2024-04-22T06:49:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:18 smithi193 ceph-mon[31815]: pgmap v155: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 MiB/s wr, 3.11k op/s 2024-04-22T06:49:19.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:18 smithi190 ceph-mon[31421]: pgmap v155: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 MiB/s wr, 3.11k op/s 2024-04-22T06:49:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:20 smithi079 ceph-mon[27946]: pgmap v156: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 479 MiB/s wr, 3.00k op/s 2024-04-22T06:49:21.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:20 smithi193 ceph-mon[31815]: pgmap v156: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 479 MiB/s wr, 3.00k op/s 2024-04-22T06:49:21.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:20 smithi190 ceph-mon[31421]: pgmap v156: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 479 MiB/s wr, 3.00k op/s 2024-04-22T06:49:22.593 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:22.595+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:22.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:22.596+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:49:22.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:22.596+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:49:22.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:22.596+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:49:22.594 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:22.596+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:49:22.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:22 smithi079 ceph-mon[27946]: pgmap v157: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 441 MiB/s wr, 2.97k op/s 2024-04-22T06:49:23.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:22 smithi193 ceph-mon[31815]: pgmap v157: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 441 MiB/s wr, 2.97k op/s 2024-04-22T06:49:23.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:22 smithi190 ceph-mon[31421]: pgmap v157: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 441 MiB/s wr, 2.97k op/s 2024-04-22T06:49:24.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:24 smithi079 ceph-mon[27946]: pgmap v158: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 277 MiB/s wr, 2.03k op/s 2024-04-22T06:49:25.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:24 smithi193 ceph-mon[31815]: pgmap v158: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 277 MiB/s wr, 2.03k op/s 2024-04-22T06:49:25.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:24 smithi190 ceph-mon[31421]: pgmap v158: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 277 MiB/s wr, 2.03k op/s 2024-04-22T06:49:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:26 smithi079 ceph-mon[27946]: pgmap v159: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 167 MiB/s wr, 1.25k op/s 2024-04-22T06:49:27.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:26 smithi193 ceph-mon[31815]: pgmap v159: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 167 MiB/s wr, 1.25k op/s 2024-04-22T06:49:27.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:26 smithi190 ceph-mon[31421]: pgmap v159: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 167 MiB/s wr, 1.25k op/s 2024-04-22T06:49:28.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:28 smithi079 ceph-mon[27946]: pgmap v160: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 167 MiB/s wr, 1.25k op/s 2024-04-22T06:49:29.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:28 smithi193 ceph-mon[31815]: pgmap v160: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 167 MiB/s wr, 1.25k op/s 2024-04-22T06:49:29.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:28 smithi190 ceph-mon[31421]: pgmap v160: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 167 MiB/s wr, 1.25k op/s 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1054025042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c406a9f0 msgr2=0x7fd8c406add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1054025042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c406a9f0 0x7fd8c406add0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009900 tx=0x7fd8b402bc80 comp rx=0 tx=0).stop 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1054025042 shutdown_connections 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1054025042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406b3a0 0x7fd8c4106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1054025042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 0x7fd8c410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1054025042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c406a9f0 0x7fd8c406add0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1054025042 >> 172.21.15.79:0/1054025042 conn(0x7fd8c4076c50 msgr2=0x7fd8c4077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1054025042 shutdown_connections 2024-04-22T06:49:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.811+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1054025042 wait complete. 2024-04-22T06:49:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 Processor -- start 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 -- start start 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406a9f0 0x7fd8c4101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 0x7fd8c41017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 0x7fd8c41a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41122e0 con 0x7fd8c4101cf0 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c4112160 con 0x7fd8c406a9f0 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.812+0000 7fd8c9f22640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c4112460 con 0x7fd8c4107260 2024-04-22T06:49:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 0x7fd8c41a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 0x7fd8c41a6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43662/0 (socket says 172.21.15.79:43662) 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 -- 172.21.15.79:0/1897449811 learned_addr learned my addr 172.21.15.79:0/1897449811 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406a9f0 0x7fd8c4101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 -- 172.21.15.79:0/1897449811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 msgr2=0x7fd8c41017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 0x7fd8c41017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 0x7fd8c41017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 -- 172.21.15.79:0/1897449811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406a9f0 msgr2=0x7fd8c4101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406a9f0 0x7fd8c4101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 -- 172.21.15.79:0/1897449811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c41a7110 con 0x7fd8c4101cf0 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406a9f0 0x7fd8c4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 0x7fd8c41017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 0x7fd8c41a6a10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00b990 tx=0x7fd8bc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c0ff9640 1 -- 172.21.15.79:0/1897449811 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc019070 con 0x7fd8c4101cf0 2024-04-22T06:49:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c0ff9640 1 -- 172.21.15.79:0/1897449811 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8bc00cba0 con 0x7fd8c4101cf0 2024-04-22T06:49:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c0ff9640 1 -- 172.21.15.79:0/1897449811 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc0075d0 con 0x7fd8c4101cf0 2024-04-22T06:49:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c41a7400 con 0x7fd8c4101cf0 2024-04-22T06:49:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.813+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c41a7950 con 0x7fd8c4101cf0 2024-04-22T06:49:29.820 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.815+0000 7fd8c0ff9640 1 -- 172.21.15.79:0/1897449811 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd8bc01c070 con 0x7fd8c4101cf0 2024-04-22T06:49:29.820 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 Processor -- start 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 -- start start 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8c0ff9640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd8ac0a3e70 0x7fd8ac0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d24620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d24fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5648d7c898d0 con 0x5648d7d258d0 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5648d7c26e80 con 0x5648d7d2e0e0 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648d7c85890 con 0x5648d7d24b60 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd8ac0a3e70 0x7fd8ac0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8c0ff9640 1 -- 172.21.15.79:0/1897449811 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fd8bc0cfea0 con 0x7fd8c4101cf0 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8c37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d24620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8c37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d24620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43670/0 (socket says 172.21.15.79:43670) 2024-04-22T06:49:29.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.816+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2374989615 learned_addr learned my addr 172.21.15.79:0/2374989615 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2374989615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 msgr2=0x5648d7d24fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d24fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2374989615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 msgr2=0x5648d7d304d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d304d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2374989615 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648d7c48320 con 0x5648d7d258d0 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd8ac0a3e70 0x7fd8ac0a6330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4102d00 tx=0x7fd8b8008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d24620 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009900 tx=0x7fd8b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208652a3231df0c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8a9ffb640 1 -- 172.21.15.79:0/2374989615 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4044020 con 0x5648d7d258d0 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8a9ffb640 1 -- 172.21.15.79:0/2374989615 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b402cce0 con 0x5648d7d258d0 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.817+0000 7fd8a9ffb640 1 -- 172.21.15.79:0/2374989615 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40046c0 con 0x5648d7d258d0 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2374989615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 msgr2=0x5648d7d24620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d24620 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009900 tx=0x7fd8b4004040 comp rx=0 tx=0).stop 2024-04-22T06:49:29.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2374989615 shutdown_connections 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d304d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d24fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2374989615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d24620 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2374989615 >> 172.21.15.79:0/2374989615 conn(0x5648d7cebc90 msgr2=0x5648d7d1f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2374989615 shutdown_connections 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2374989615 wait complete. 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 -- start start 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.818+0000 7fd8ccbea740 1 Processor -- start 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d2daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d2ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5648d7d30db0 con 0x5648d7d258d0 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5648d7c11eb0 con 0x5648d7d2e0e0 2024-04-22T06:49:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648d7d30f30 con 0x5648d7d24b60 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 10 client.0 _ll_register_callbacks cb 0x5648d7c3eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d2ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d2ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43678/0 (socket says 172.21.15.79:43678) 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 -- 172.21.15.79:0/3690023278 learned_addr learned my addr 172.21.15.79:0/3690023278 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d2daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 -- 172.21.15.79:0/3690023278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 msgr2=0x5648d7d2daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d2daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 -- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 msgr2=0x5648d7d2b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 -- 172.21.15.79:0/3690023278 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648d799c210 con 0x5648d7d258d0 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:29.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8c2ffd640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d2ae30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800ce30 tx=0x7fd8b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4f02c2b14409f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/3690023278 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8023060 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/3690023278 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fd8b80117e0 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.819+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b402cce0 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/3690023278 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd8b802a050 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 10 client.16896 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5648d7a68620 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/3690023278 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b402cce0 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/3690023278 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fd8b801d970 con 0x5648d7d258d0 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 10 client.16896 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 10 client.16896 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 20 client.16896 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7d3e160 0x5648d7d40570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 4 client.16896 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:29.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.820+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648d7c9d880 con 0x5648d7d3e160 2024-04-22T06:49:29.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.821+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7d3e160 0x5648d7d40570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.823+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7d3e160 0x5648d7d40570 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8f635113456ba99 server_cookie=9d51922b3762effd in_seq=0 out_seq=0 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.823+0000 7fd8a2ffd640 10 client.16896 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/3690023278 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648d7c9d880 con 0x5648d7d3e160 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8a2ffd640 10 client.16896 handle_command_reply: tid=0 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 client.16896 shutdown 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 msgr2=0x5648d7d2ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d2ae30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800ce30 tx=0x7fd8b8004750 comp rx=0 tx=0).stop 2024-04-22T06:49:29.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 shutdown_connections 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7d3e160 0x5648d7d40570 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d258d0 0x5648d7d2ae30 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8a2ffd640 0 client.16896 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/3690023278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d24b60 0x5648d7d2daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.824+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 >> 172.21.15.79:0/3690023278 conn(0x5648d7cebc90 msgr2=0x5648d7d215a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.825+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 shutdown_connections 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.825+0000 7fd8ccbea740 1 -- 172.21.15.79:0/3690023278 wait complete. 2024-04-22T06:49:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.825+0000 7fd8ccbea740 20 client.16896 trim_cache size 0 max 16384 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.839+0000 7fd8ccbea740 1 Processor -- start 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.839+0000 7fd8ccbea740 1 -- start start 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 0x5648d7d266b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d258d0 0x5648d7d39cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41b2980 con 0x5648d7d24b60 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41b0250 con 0x5648d7d258d0 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41ae0c0 con 0x5648d7d2e0e0 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 0x5648d7d266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 0x5648d7d266b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43686/0 (socket says 172.21.15.79:43686) 2024-04-22T06:49:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2843932878 learned_addr learned my addr 172.21.15.79:0/2843932878 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2843932878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d2e0e0 msgr2=0x5648d7d2d560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2843932878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d258d0 msgr2=0x5648d7d39cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d258d0 0x5648d7d39cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 -- 172.21.15.79:0/2843932878 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648d7d23e30 con 0x5648d7d24b60 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 0x5648d7d266b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005ec0 tx=0x7fd8b4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d971c91c14d0626 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8a97fa640 1 -- 172.21.15.79:0/2843932878 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4044040 con 0x5648d7d24b60 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8a97fa640 1 -- 172.21.15.79:0/2843932878 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b40044b0 con 0x5648d7d24b60 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.840+0000 7fd8a97fa640 1 -- 172.21.15.79:0/2843932878 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4003e20 con 0x5648d7d24b60 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2843932878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 msgr2=0x5648d7d266b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 0x5648d7d266b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005ec0 tx=0x7fd8b4005fd0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2843932878 shutdown_connections 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d258d0 0x5648d7d39cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d2e0e0 0x5648d7d2d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/2843932878 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d24b60 0x5648d7d266b0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2843932878 >> 172.21.15.79:0/2843932878 conn(0x5648d7cebc90 msgr2=0x5648d7d25fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2843932878 shutdown_connections 2024-04-22T06:49:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 -- 172.21.15.79:0/2843932878 wait complete. 2024-04-22T06:49:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.841+0000 7fd8ccbea740 1 -- start start 2024-04-22T06:49:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 Processor -- start 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d24b60 0x5648d7d363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d258d0 0x5648d7d276a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 0x5648d7d27be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41ae710 con 0x5648d7d2e0e0 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41afb80 con 0x5648d7d24b60 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c41aede0 con 0x5648d7d258d0 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 10 client.0 _ll_register_callbacks cb 0x7fd8b800ae40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:29.845 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 0x5648d7d27be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 0x5648d7d27be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43692/0 (socket says 172.21.15.79:43692) 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 -- 172.21.15.79:0/146735459 learned_addr learned my addr 172.21.15.79:0/146735459 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 -- 172.21.15.79:0/146735459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d258d0 msgr2=0x5648d7d276a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d258d0 0x5648d7d276a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 -- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d24b60 msgr2=0x5648d7d363b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d24b60 0x5648d7d363b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 -- 172.21.15.79:0/146735459 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648d7d28120 con 0x5648d7d2e0e0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8c3fff640 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 0x5648d7d27be0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00b960 tx=0x7fd8bc09c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc230bc2d12fea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/146735459 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc0a49f0 con 0x5648d7d2e0e0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/146735459 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fd8bc0983f0 con 0x5648d7d2e0e0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.842+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b40044b0 con 0x5648d7d2e0e0 2024-04-22T06:49:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/146735459 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd8bc0a4560 con 0x5648d7d2e0e0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 10 client.16902 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5648d7da4440 con 0x5648d7d2e0e0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/146735459 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b40044b0 con 0x5648d7d2e0e0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/146735459 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fd8bc0cdb70 con 0x5648d7d2e0e0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 10 client.16902 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 10 client.16902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 20 client.16902 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7da4bc0 0x5648d7da6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 4 client.16902 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.843+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648d7c293d0 con 0x5648d7da4bc0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.844+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7da4bc0 0x5648d7da6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.846+0000 7fd8c37fe640 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7da4bc0 0x5648d7da6fe0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ec25942f7ddd6e server_cookie=7ed46a86dbebaa5d in_seq=0 out_seq=0 2024-04-22T06:49:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.846+0000 7fd8a2ffd640 10 client.16902 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8a2ffd640 1 -- 172.21.15.79:0/146735459 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648d7c293d0 con 0x5648d7da4bc0 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8a2ffd640 10 client.16902 handle_command_reply: tid=0 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 client.16902 shutdown 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 msgr2=0x5648d7d27be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 0x5648d7d27be0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00b960 tx=0x7fd8bc09c500 comp rx=0 tx=0).stop 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 shutdown_connections 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648d7da4bc0 0x5648d7da6fe0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648d7d2e0e0 0x5648d7d27be0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648d7d258d0 0x5648d7d276a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 --2- 172.21.15.79:0/146735459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648d7d24b60 0x5648d7d363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8a2ffd640 0 client.16902 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 >> 172.21.15.79:0/146735459 conn(0x5648d7cebc90 msgr2=0x5648d7d29700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 shutdown_connections 2024-04-22T06:49:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 1 -- 172.21.15.79:0/146735459 wait complete. 2024-04-22T06:49:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.858+0000 7fd8ccbea740 20 client.16902 trim_cache size 0 max 16384 2024-04-22T06:49:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:49:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:49:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:49:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd8ac0a3e70 msgr2=0x7fd8ac0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd8ac0a3e70 0x7fd8ac0a6330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4102d00 tx=0x7fd8b8008040 comp rx=0 tx=0).stop 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 msgr2=0x7fd8c41a6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 0x7fd8c41a6a10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00b990 tx=0x7fd8bc00be50 comp rx=0 tx=0).stop 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 shutdown_connections 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8c4101cf0 0x7fd8c41a6a10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8c4107260 0x7fd8c41017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd8ac0a3e70 0x7fd8ac0a6330 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 --2- 172.21.15.79:0/1897449811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c406a9f0 0x7fd8c4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 >> 172.21.15.79:0/1897449811 conn(0x7fd8c4076c50 msgr2=0x7fd8c40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 shutdown_connections 2024-04-22T06:49:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:29.859+0000 7fd8c9f22640 1 -- 172.21.15.79:0/1897449811 wait complete. 2024-04-22T06:49:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:30 smithi079 ceph-mon[27946]: pgmap v161: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 12 MiB/s wr, 103 op/s 2024-04-22T06:49:31.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:30 smithi193 ceph-mon[31815]: pgmap v161: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 12 MiB/s wr, 103 op/s 2024-04-22T06:49:31.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:30 smithi190 ceph-mon[31421]: pgmap v161: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 12 MiB/s wr, 103 op/s 2024-04-22T06:49:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:32 smithi079 ceph-mon[27946]: pgmap v162: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.9 KiB/s wr, 10 op/s 2024-04-22T06:49:32.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:32 smithi193 ceph-mon[31815]: pgmap v162: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.9 KiB/s wr, 10 op/s 2024-04-22T06:49:33.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:32 smithi190 ceph-mon[31421]: pgmap v162: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.9 KiB/s wr, 10 op/s 2024-04-22T06:49:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:34 smithi079 ceph-mon[27946]: pgmap v163: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.6 KiB/s wr, 8 op/s 2024-04-22T06:49:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:34 smithi193 ceph-mon[31815]: pgmap v163: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.6 KiB/s wr, 8 op/s 2024-04-22T06:49:35.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:34 smithi190 ceph-mon[31421]: pgmap v163: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.6 KiB/s wr, 8 op/s 2024-04-22T06:49:36.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:36 smithi079 ceph-mon[27946]: pgmap v164: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2024-04-22T06:49:37.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:36 smithi193 ceph-mon[31815]: pgmap v164: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2024-04-22T06:49:37.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:36 smithi190 ceph-mon[31421]: pgmap v164: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2024-04-22T06:49:37.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:37.597+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:37.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:37.598+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:49:37.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:37.598+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:49:37.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:37.598+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:49:37.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:37.598+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:49:38.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:38 smithi079 ceph-mon[27946]: pgmap v165: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-04-22T06:49:39.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:38 smithi193 ceph-mon[31815]: pgmap v165: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-04-22T06:49:39.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:38 smithi190 ceph-mon[31421]: pgmap v165: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-04-22T06:49:39.897 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 -- 172.21.15.79:0/3784419164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed8106f50 msgr2=0x7faed8107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 --2- 172.21.15.79:0/3784419164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed8106f50 0x7faed8107330 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7faec8009900 tx=0x7faec802bc50 comp rx=0 tx=0).stop 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 -- 172.21.15.79:0/3784419164 shutdown_connections 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 --2- 172.21.15.79:0/3784419164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806b040 0x7faed810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 --2- 172.21.15.79:0/3784419164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed806a680 0x7faed806ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 --2- 172.21.15.79:0/3784419164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed8106f50 0x7faed8107330 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 -- 172.21.15.79:0/3784419164 >> 172.21.15.79:0/3784419164 conn(0x7faed8077e30 msgr2=0x7faed8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 -- 172.21.15.79:0/3784419164 shutdown_connections 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.009+0000 7faede3bd640 1 -- 172.21.15.79:0/3784419164 wait complete. 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 Processor -- start 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 -- start start 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806a680 0x7faed81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 0x7faed8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed8106f50 0x7faed819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7faed8112150 con 0x7faed806b040 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faed8111fd0 con 0x7faed806a680 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faede3bd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faed81122d0 con 0x7faed8106f50 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faed77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 0x7faed8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faed77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 0x7faed8198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48506/0 (socket says 172.21.15.79:48506) 2024-04-22T06:49:40.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.010+0000 7faed77fe640 1 -- 172.21.15.79:0/3582033710 learned_addr learned my addr 172.21.15.79:0/3582033710 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed7fff640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806a680 0x7faed81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed77fe640 1 -- 172.21.15.79:0/3582033710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed8106f50 msgr2=0x7faed819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faedc95a640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed8106f50 0x7faed819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed77fe640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed8106f50 0x7faed819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed77fe640 1 -- 172.21.15.79:0/3582033710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806a680 msgr2=0x7faed81985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed77fe640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806a680 0x7faed81985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed77fe640 1 -- 172.21.15.79:0/3582033710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed819d570 con 0x7faed806b040 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed7fff640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806a680 0x7faed81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faedc95a640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed8106f50 0x7faed819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed77fe640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 0x7faed8198b10 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7faecc00c920 tx=0x7faecc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc007c70 con 0x7faed806b040 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed819d860 con 0x7faed806b040 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faecc007e10 con 0x7faed806b040 2024-04-22T06:49:40.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc018820 con 0x7faed806b040 2024-04-22T06:49:40.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.011+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed81a50f0 con 0x7faed806b040 2024-04-22T06:49:40.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.012+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed8069230 con 0x7faed806b040 2024-04-22T06:49:40.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.013+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faecc00f450 con 0x7faed806b040 2024-04-22T06:49:40.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.014+0000 7faed57fa640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faec00a3da0 0x7faec00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.014+0000 7faed7fff640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faec00a3da0 0x7faec00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.014+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7faecc0d0c00 con 0x7faed806b040 2024-04-22T06:49:40.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.014+0000 7faed7fff640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faec00a3da0 0x7faec00a6260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faec8000c00 tx=0x7faec802c940 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.018+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faecc014030 con 0x7faed806b040 2024-04-22T06:49:40.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.235+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faed802d5c0 con 0x7faed806b040 2024-04-22T06:49:40.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.236+0000 7faed57fa640 1 -- 172.21.15.79:0/3582033710 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11881 (secure 0 0 0) 0x7faecc00de20 con 0x7faed806b040 2024-04-22T06:49:40.234 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:49:40.234 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:48:41.413672+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25559,"mds_2":25552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25552":{"gid":25552,"name":"f","rank":2,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.193:6835/3147972524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3147972524},{"type":"v1","addr":"172.21.15.193:6835","nonce":3147972524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25559":{"gid":25559,"name":"e","rank":1,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.190:6835/833753836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":833753836},{"type":"v1","addr":"172.21.15.190:6835","nonce":833753836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25810":{"gid":25810,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/1413649941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1413649941},{"type":"v1","addr":"172.21.15.193:6839","nonce":1413649941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25817":{"gid":25817,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1169559599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1169559599},{"type":"v1","addr":"172.21.15.190:6837","nonce":1169559599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,25552,25559]},"id":1}]} 2024-04-22T06:49:40.235 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 47 2024-04-22T06:49:40.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faec00a3da0 msgr2=0x7faec00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faec00a3da0 0x7faec00a6260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faec8000c00 tx=0x7faec802c940 comp rx=0 tx=0).stop 2024-04-22T06:49:40.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 msgr2=0x7faed8198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 0x7faed8198b10 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7faecc00c920 tx=0x7faecc00cde0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 shutdown_connections 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faed8106f50 0x7faed819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faed806b040 0x7faed8198b10 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7faec00a3da0 0x7faec00a6260 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 --2- 172.21.15.79:0/3582033710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faed806a680 0x7faed81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 >> 172.21.15.79:0/3582033710 conn(0x7faed8077e30 msgr2=0x7faed80fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 shutdown_connections 2024-04-22T06:49:40.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.238+0000 7faede3bd640 1 -- 172.21.15.79:0/3582033710 wait complete. 2024-04-22T06:49:40.247 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id fdf32dfd --release --await' 2024-04-22T06:49:40.248 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id fdf32dfd --release --await 2024-04-22T06:49:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 -- 172.21.15.79:0/60100459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a1806b1d0 msgr2=0x7f8a18106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 --2- 172.21.15.79:0/60100459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a1806b1d0 0x7f8a18106b90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009900 tx=0x7f8a0802bc80 comp rx=0 tx=0).stop 2024-04-22T06:49:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 -- 172.21.15.79:0/60100459 shutdown_connections 2024-04-22T06:49:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 --2- 172.21.15.79:0/60100459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a18107270 0x7f8a1810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 --2- 172.21.15.79:0/60100459 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a1806b1d0 0x7f8a18106b90 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 --2- 172.21.15.79:0/60100459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a1806a820 0x7f8a1806ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 -- 172.21.15.79:0/60100459 >> 172.21.15.79:0/60100459 conn(0x7f8a18077eb0 msgr2=0x7f8a180782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 -- 172.21.15.79:0/60100459 shutdown_connections 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 -- 172.21.15.79:0/60100459 wait complete. 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 Processor -- start 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.355+0000 7f8a1f643640 1 -- start start 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a1806a820 0x7f8a1819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a18107270 0x7f8a1819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 0x7f8a18198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181122d0 con 0x7f8a18198790 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18112150 con 0x7f8a18107270 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a18112450 con 0x7f8a1806a820 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 0x7f8a18198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 0x7f8a18198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48522/0 (socket says 172.21.15.79:48522) 2024-04-22T06:49:40.356 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 -- 172.21.15.79:0/376998788 learned_addr learned my addr 172.21.15.79:0/376998788 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a1806a820 0x7f8a1819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 -- 172.21.15.79:0/376998788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a1806a820 msgr2=0x7f8a1819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a1806a820 0x7f8a1819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 -- 172.21.15.79:0/376998788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a18107270 msgr2=0x7f8a1819eb60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a18107270 0x7f8a1819eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 -- 172.21.15.79:0/376998788 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a18199320 con 0x7f8a18198790 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1dbe0640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 0x7f8a18198c40 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000b990 tx=0x7f8a1000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a0e7fc640 1 -- 172.21.15.79:0/376998788 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10019070 con 0x7f8a18198790 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a18199610 con 0x7f8a18198790 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a0e7fc640 1 -- 172.21.15.79:0/376998788 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a1000cba0 con 0x7f8a18198790 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a0e7fc640 1 -- 172.21.15.79:0/376998788 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a100075d0 con 0x7f8a18198790 2024-04-22T06:49:40.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.356+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a181a57a0 con 0x7f8a18198790 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.359+0000 7f8a0e7fc640 1 -- 172.21.15.79:0/376998788 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a1001c070 con 0x7f8a18198790 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.359+0000 7f8a2223c740 1 Processor -- start 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 -- start start 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 0x558a69ec3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 0x558a69ed06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ed3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558a69e28500 con 0x558a69ec3040 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558a69dce6a0 con 0x558a69ed0c20 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a2223c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558a69e244c0 con 0x558a69ec3960 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 0x558a69ec3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 0x558a69ec3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48524/0 (socket says 172.21.15.79:48524) 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2448743670 learned_addr learned my addr 172.21.15.79:0/2448743670 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a0e7fc640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8a040a3d20 0x7f8a040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1dbe0640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8a040a3d20 0x7f8a040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2448743670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 msgr2=0x558a69ed06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 0x558a69ed06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2448743670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 msgr2=0x558a69ed3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ed3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2448743670 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a69ded5e0 con 0x558a69ec3040 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a0e7fc640 1 -- 172.21.15.79:0/376998788 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f8a100cfeb0 con 0x7f8a18198790 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.360+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 0x558a69ec3420 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8a08036040 tx=0x7f8a0802c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe56b3c1e31575c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a037fe640 1 -- 172.21.15.79:0/2448743670 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08042020 con 0x558a69ec3040 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a1dbe0640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8a040a3d20 0x7f8a040a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a100982a0 tx=0x7f8a100c6ea0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a037fe640 1 -- 172.21.15.79:0/2448743670 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a0802cce0 con 0x558a69ec3040 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a037fe640 1 -- 172.21.15.79:0/2448743670 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08003c60 con 0x558a69ec3040 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 -- 172.21.15.79:0/2448743670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 msgr2=0x558a69ec3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 0x558a69ec3420 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8a08036040 tx=0x7f8a0802c6f0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 -- 172.21.15.79:0/2448743670 shutdown_connections 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ed3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 0x558a69ed06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2448743670 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3040 0x558a69ec3420 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 -- 172.21.15.79:0/2448743670 >> 172.21.15.79:0/2448743670 conn(0x558a69ebda40 msgr2=0x558a69ebde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 -- 172.21.15.79:0/2448743670 shutdown_connections 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.361+0000 7f8a2223c740 1 -- 172.21.15.79:0/2448743670 wait complete. 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 -- start start 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 Processor -- start 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 0x558a69ecdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3960 0x558a69ece080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ece5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558a69ed3980 con 0x558a69ed0c20 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558a69de1a60 con 0x558a69ec3960 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558a69ed3b00 con 0x558a69ec3040 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a2223c740 10 client.0 _ll_register_callbacks cb 0x558a69b3a110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a1cbde640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ece5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a1cbde640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ece5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48534/0 (socket says 172.21.15.79:48534) 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a1cbde640 1 -- 172.21.15.79:0/498618814 learned_addr learned my addr 172.21.15.79:0/498618814 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3960 0x558a69ece080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.362+0000 7f8a1cbde640 1 -- 172.21.15.79:0/498618814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 msgr2=0x558a69ecdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 0x558a69ecdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a1cbde640 1 -- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3960 msgr2=0x558a69ece080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3960 0x558a69ece080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a1cbde640 1 -- 172.21.15.79:0/498618814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a69b3a920 con 0x558a69ed0c20 2024-04-22T06:49:40.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3960 0x558a69ece080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ece5c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8a08003940 tx=0x7f8a08003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79dc27ace22f8af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a00ff9640 1 -- 172.21.15.79:0/498618814 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08042030 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a00ff9640 1 -- 172.21.15.79:0/498618814 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f8a08004240 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a0802cce0 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a00ff9640 1 -- 172.21.15.79:0/498618814 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8a08004440 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 10 client.16920 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558a69c06d30 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a00ff9640 1 -- 172.21.15.79:0/498618814 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a0802cce0 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a00ff9640 1 -- 172.21.15.79:0/498618814 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7f8a080484c0 con 0x558a69ed0c20 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 10 client.16920 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 10 client.16920 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 20 client.16920 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69ee0cd0 0x558a69ee30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 4 client.16920 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:40.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.363+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a69dfba10 con 0x558a69ee0cd0 2024-04-22T06:49:40.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.364+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69ee0cd0 0x558a69ee30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.365+0000 7f8a00ff9640 10 client.16920 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:40.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.365+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69ee0cd0 0x558a69ee30c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42bc1f5d8ea6da5 server_cookie=a9f91fd31d4ee788 in_seq=0 out_seq=0 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a00ff9640 1 -- 172.21.15.79:0/498618814 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a69dfba10 con 0x558a69ee0cd0 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a00ff9640 10 client.16920 handle_command_reply: tid=0 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 client.16920 shutdown 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 msgr2=0x558a69ece5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ece5c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8a08003940 tx=0x7f8a08003820 comp rx=0 tx=0).stop 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 shutdown_connections 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69ee0cd0 0x558a69ee30c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3960 0x558a69ece080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ece5c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a00ff9640 0 client.16920 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:40.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 --2- 172.21.15.79:0/498618814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 0x558a69ecdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 >> 172.21.15.79:0/498618814 conn(0x558a69ebda40 msgr2=0x558a69ebf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 shutdown_connections 2024-04-22T06:49:40.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 1 -- 172.21.15.79:0/498618814 wait complete. 2024-04-22T06:49:40.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.366+0000 7f8a2223c740 20 client.16920 trim_cache size 0 max 16384 2024-04-22T06:49:40.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 Processor -- start 2024-04-22T06:49:40.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 -- start start 2024-04-22T06:49:40.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3040 0x558a69ed0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 0x558a69ebc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ecade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181ae570 con 0x558a69ed0c20 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181abe40 con 0x558a69ec3040 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a2223c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181a9cb0 con 0x558a69ec3960 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ecade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ecade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48546/0 (socket says 172.21.15.79:48546) 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2300692466 learned_addr learned my addr 172.21.15.79:0/2300692466 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2300692466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 msgr2=0x558a69ebc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 0x558a69ebc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2300692466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3040 msgr2=0x558a69ed0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3040 0x558a69ed0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.383+0000 7f8a1cbde640 1 -- 172.21.15.79:0/2300692466 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a69ee8f00 con 0x558a69ed0c20 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ecade0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009580 tx=0x7f8a08005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1c268c1bdf77d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a02ffd640 1 -- 172.21.15.79:0/2300692466 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08042040 con 0x558a69ed0c20 2024-04-22T06:49:40.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a02ffd640 1 -- 172.21.15.79:0/2300692466 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a08003cf0 con 0x558a69ed0c20 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a02ffd640 1 -- 172.21.15.79:0/2300692466 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08004040 con 0x558a69ed0c20 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 -- 172.21.15.79:0/2300692466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 msgr2=0x558a69ecade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ecade0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009580 tx=0x7f8a08005890 comp rx=0 tx=0).stop 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 -- 172.21.15.79:0/2300692466 shutdown_connections 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ed0c20 0x558a69ecade0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3960 0x558a69ebc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 --2- 172.21.15.79:0/2300692466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ec3040 0x558a69ed0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 -- 172.21.15.79:0/2300692466 >> 172.21.15.79:0/2300692466 conn(0x558a69ebda40 msgr2=0x558a69ebc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 -- 172.21.15.79:0/2300692466 shutdown_connections 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.384+0000 7f8a2223c740 1 -- 172.21.15.79:0/2300692466 wait complete. 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 -- start start 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 Processor -- start 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 0x558a69ee5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 0x558a69ee63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ee42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181aa300 con 0x558a69ec3960 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181ab770 con 0x558a69ed0c20 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a181aa9d0 con 0x558a69ec3040 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a2223c740 10 client.0 _ll_register_callbacks cb 0x7f8a0802c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 0x558a69ee63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 0x558a69ee63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48552/0 (socket says 172.21.15.79:48552) 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 -- 172.21.15.79:0/4119171477 learned_addr learned my addr 172.21.15.79:0/4119171477 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ee42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 -- 172.21.15.79:0/4119171477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 msgr2=0x558a69ee5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 0x558a69ee5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 -- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 msgr2=0x558a69ee42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ee42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1d3df640 1 -- 172.21.15.79:0/4119171477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a69ee4ad0 con 0x558a69ec3960 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.385+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ee42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a1d3df640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 0x558a69ee63e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8a140099a0 tx=0x7f8a14016800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f737148ba5736430 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:40.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a00ff9640 1 -- 172.21.15.79:0/4119171477 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14005970 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a00ff9640 1 -- 172.21.15.79:0/4119171477 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7f8a14005b10 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a08003cf0 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a00ff9640 1 -- 172.21.15.79:0/4119171477 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8a14005d10 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 10 client.16926 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558a69f47a10 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a00ff9640 1 -- 172.21.15.79:0/4119171477 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a08003cf0 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a00ff9640 1 -- 172.21.15.79:0/4119171477 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7f8a140246c0 con 0x558a69ec3960 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 10 client.16926 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 10 client.16926 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 20 client.16926 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69f481c0 0x558a69f4a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 4 client.16926 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fdf32dfd", "release": true, "await": true}] 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.386+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fdf32dfd", "release": true, "await": true}) -- 0x558a69e35f20 con 0x558a69f481c0 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.387+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69f481c0 0x558a69f4a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.387+0000 7f8a00ff9640 10 client.16926 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:40.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.387+0000 7f8a1cbde640 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69f481c0 0x558a69f4a5e0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=382d6ebc7ccd34e9 server_cookie=58b030d0cbea70fc in_seq=0 out_seq=0 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a00ff9640 1 -- 172.21.15.79:0/4119171477 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x558a69e35f20 con 0x558a69f481c0 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a00ff9640 10 client.16926 handle_command_reply: tid=0 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 client.16926 shutdown 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 msgr2=0x558a69ee63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 0x558a69ee63e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8a140099a0 tx=0x7f8a14016800 comp rx=0 tx=0).stop 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 shutdown_connections 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558a69f481c0 0x558a69f4a5e0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558a69ed0c20 0x558a69ee42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558a69ec3960 0x558a69ee63e0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 --2- 172.21.15.79:0/4119171477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a69ec3040 0x558a69ee5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 >> 172.21.15.79:0/4119171477 conn(0x558a69ebda40 msgr2=0x558a69ece390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a00ff9640 0 client.16926 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 shutdown_connections 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 1 -- 172.21.15.79:0/4119171477 wait complete. 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.392+0000 7f8a2223c740 20 client.16926 trim_cache size 0 max 16384 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 47, 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 18, 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "fdf32dfd": { 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 18, 2024-04-22T06:49:40.392 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 87.3, 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:40.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:49:40.394 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:49:40.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8a040a3d20 msgr2=0x7f8a040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8a040a3d20 0x7f8a040a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a100982a0 tx=0x7f8a100c6ea0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 msgr2=0x7f8a18198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:40.394 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 0x7f8a18198c40 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000b990 tx=0x7f8a1000be50 comp rx=0 tx=0).stop 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 shutdown_connections 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f8a040a3d20 0x7f8a040a61e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a18107270 0x7f8a1819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8a18198790 0x7f8a18198c40 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 --2- 172.21.15.79:0/376998788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a1806a820 0x7f8a1819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 >> 172.21.15.79:0/376998788 conn(0x7f8a18077eb0 msgr2=0x7f8a180ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 shutdown_connections 2024-04-22T06:49:40.395 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:40.393+0000 7f8a1f643640 1 -- 172.21.15.79:0/376998788 wait complete. 2024-04-22T06:49:40.417 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: fdf32dfd, seconds elapsed: 29.9 2024-04-22T06:49:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f046309c700 time 2024-04-22T06:49:40.389226+0000 2024-04-22T06:49:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04766d73dd] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: clone() 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.390+0000 7f046309c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f046309c700 time 2024-04-22T06:49:40.389226+0000 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04766d73dd] 2024-04-22T06:49:40.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: clone() 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: *** Caught signal (Aborted) ** 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: in thread 7f046309c700 thread_name:quiesce.agt 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f047545fd20] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: gsignal() 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: abort() 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04766d7437] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 13: clone() 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.391+0000 7f046309c700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: in thread 7f046309c700 thread_name:quiesce.agt 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f047545fd20] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: gsignal() 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: abort() 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04766d7437] 2024-04-22T06:49:40.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 13: clone() 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: -1> 2024-04-22T06:49:40.390+0000 7f046309c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f046309c700 time 2024-04-22T06:49:40.389226+0000 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04766d73dd] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: clone() 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 0> 2024-04-22T06:49:40.391+0000 7f046309c700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: in thread 7f046309c700 thread_name:quiesce.agt 2024-04-22T06:49:40.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f047545fd20] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: gsignal() 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: abort() 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04766d7437] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 13: clone() 2024-04-22T06:49:40.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: -2> 2024-04-22T06:49:40.390+0000 7f046309c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f046309c700 time 2024-04-22T06:49:40.389226+0000 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04766d73dd] 2024-04-22T06:49:40.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: clone() 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: -1> 2024-04-22T06:49:40.391+0000 7f046309c700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: in thread 7f046309c700 thread_name:quiesce.agt 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f047545fd20] 2024-04-22T06:49:40.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2: gsignal() 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 3: abort() 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04766d7437] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04766d75a3] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5654526c6e9c] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5654526c6f9f] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5654525f50d3] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5654525f503c] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x565452874d3f] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5654528dd560] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04754551ca] 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 13: clone() 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[66765]: 2024-04-22T06:49:40.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:40 smithi193 podman[67333]: 2024-04-22 06:49:40.449380905 +0000 UTC m=+0.022186302 container died 1da2865844d1d4a696b6bf4c5b5585a01d42d09aa5ab98368a242ed9564703f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f134c801700 time 2024-04-22T06:49:40.389182+0000 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f135fe3c3dd] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: clone() 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.389+0000 7f134c801700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f134c801700 time 2024-04-22T06:49:40.389182+0000 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f135fe3c3dd] 2024-04-22T06:49:40.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: clone() 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: *** Caught signal (Aborted) ** 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: in thread 7f134c801700 thread_name:quiesce.agt 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f135ebc4d20] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: gsignal() 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: abort() 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f135fe3c437] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 13: clone() 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.390+0000 7f134c801700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: in thread 7f134c801700 thread_name:quiesce.agt 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f135ebc4d20] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: gsignal() 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: abort() 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f135fe3c437] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 13: clone() 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: -2> 2024-04-22T06:49:40.389+0000 7f134c801700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f134c801700 time 2024-04-22T06:49:40.389182+0000 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f135fe3c3dd] 2024-04-22T06:49:40.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: clone() 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 0> 2024-04-22T06:49:40.390+0000 7f134c801700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: in thread 7f134c801700 thread_name:quiesce.agt 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f135ebc4d20] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: gsignal() 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: abort() 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f135fe3c437] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 13: clone() 2024-04-22T06:49:40.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: -2> 2024-04-22T06:49:40.389+0000 7f134c801700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f134c801700 time 2024-04-22T06:49:40.389182+0000 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f135fe3c3dd] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: clone() 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 0> 2024-04-22T06:49:40.390+0000 7f134c801700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: in thread 7f134c801700 thread_name:quiesce.agt 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f135ebc4d20] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2: gsignal() 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 3: abort() 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f135fe3c437] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f135fe3c5a3] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564e3ec7fe9c] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564e3ec7ff9f] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564e3ebae0d3] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564e3ebae03c] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564e3ee2dd3f] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564e3ee96560] 2024-04-22T06:49:40.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f135ebba1ca] 2024-04-22T06:49:40.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 13: clone() 2024-04-22T06:49:40.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:49:40.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[65809]: 2024-04-22T06:49:40.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:40 smithi190 podman[66263]: 2024-04-22 06:49:40.451056809 +0000 UTC m=+0.022015782 container died 17390753be9b4590b8249ba47681bd45deb4776fcdbfadcc7e5913db8fa40eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:49:41.115 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-mon[31421]: pgmap v166: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2024-04-22T06:49:41.115 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3582033710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:49:41.116 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:40 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:49:41.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-mon[31815]: pgmap v166: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2024-04-22T06:49:41.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3582033710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:49:41.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:40 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:49:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:40 smithi079 ceph-mon[27946]: pgmap v166: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2024-04-22T06:49:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:40 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3582033710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:49:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:40 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:49:41.439 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:41 smithi193 podman[67333]: 2024-04-22 06:49:41.142049714 +0000 UTC m=+0.714855109 container remove 1da2865844d1d4a696b6bf4c5b5585a01d42d09aa5ab98368a242ed9564703f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:41.439 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:41 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:49:41.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:41 smithi190 podman[66263]: 2024-04-22 06:49:41.11813221 +0000 UTC m=+0.689091174 container remove 17390753be9b4590b8249ba47681bd45deb4776fcdbfadcc7e5913db8fa40eea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T06:49:41.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:41 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:49:41.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:41 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-22T06:49:41.954 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:41 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-22T06:49:41.954 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:41 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.106s CPU time. 2024-04-22T06:49:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.127+0000 7fd6d032b640 1 -- 172.21.15.79:0/4214145974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 msgr2=0x7fd6c811db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.127+0000 7fd6d032b640 1 --2- 172.21.15.79:0/4214145974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 0x7fd6c811db20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009860 tx=0x7fd6c402bc10 comp rx=0 tx=0).stop 2024-04-22T06:49:43.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.129+0000 7fd6d032b640 1 -- 172.21.15.79:0/4214145974 shutdown_connections 2024-04-22T06:49:43.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.129+0000 7fd6d032b640 1 --2- 172.21.15.79:0/4214145974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6c8110b20 0x7fd6c8109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.129+0000 7fd6d032b640 1 --2- 172.21.15.79:0/4214145974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 0x7fd6c811db20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.129+0000 7fd6d032b640 1 --2- 172.21.15.79:0/4214145974 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c8110740 0x7fd6c8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.129 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.129+0000 7fd6d032b640 1 -- 172.21.15.79:0/4214145974 >> 172.21.15.79:0/4214145974 conn(0x7fd6c806d020 msgr2=0x7fd6c806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.137+0000 7fd6d032b640 1 -- 172.21.15.79:0/4214145974 shutdown_connections 2024-04-22T06:49:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.138+0000 7fd6d032b640 1 -- 172.21.15.79:0/4214145974 wait complete. 2024-04-22T06:49:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 Processor -- start 2024-04-22T06:49:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 -- start start 2024-04-22T06:49:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6c8110740 0x7fd6c811ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 0x7fd6c8114e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 0x7fd6c8115880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8111810 con 0x7fd6c81153d0 2024-04-22T06:49:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8111690 con 0x7fd6c8110740 2024-04-22T06:49:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.139+0000 7fd6d032b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8111990 con 0x7fd6c811b730 2024-04-22T06:49:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.143+0000 7fd6ce8c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 0x7fd6c8115880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.143+0000 7fd6ce8c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 0x7fd6c8115880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38598/0 (socket says 172.21.15.79:38598) 2024-04-22T06:49:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.143+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3041544528 learned_addr learned my addr 172.21.15.79:0/3041544528 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.144+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6c8110740 0x7fd6c811ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.144+0000 7fd6cd8c6640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 0x7fd6c8114e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.148+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3041544528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 msgr2=0x7fd6c8114e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.148+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 0x7fd6c8114e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.148+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3041544528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6c8110740 msgr2=0x7fd6c811ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.148+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6c8110740 0x7fd6c811ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.148+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3041544528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c81bc440 con 0x7fd6c81153d0 2024-04-22T06:49:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.150+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 0x7fd6c8115880 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd6c402c150 tx=0x7fd6c4035f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.151+0000 7fd6bf7fe640 1 -- 172.21.15.79:0/3041544528 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4041030 con 0x7fd6c81153d0 2024-04-22T06:49:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.151+0000 7fd6bf7fe640 1 -- 172.21.15.79:0/3041544528 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c402c5b0 con 0x7fd6c81153d0 2024-04-22T06:49:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.151+0000 7fd6bf7fe640 1 -- 172.21.15.79:0/3041544528 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4039040 con 0x7fd6c81153d0 2024-04-22T06:49:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.151+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c81bc6d0 con 0x7fd6c81153d0 2024-04-22T06:49:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.151+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c81bcb10 con 0x7fd6c81153d0 2024-04-22T06:49:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.157+0000 7fd6bf7fe640 1 -- 172.21.15.79:0/3041544528 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd6c4044070 con 0x7fd6c81153d0 2024-04-22T06:49:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.158+0000 7fd6bf7fe640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6b40a3fa0 0x7fd6b40a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.158+0000 7fd6bf7fe640 1 -- 172.21.15.79:0/3041544528 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fd6c40f1ae0 con 0x7fd6c81153d0 2024-04-22T06:49:43.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.160+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6b40a3fa0 0x7fd6b40a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.162+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6b40a3fa0 0x7fd6b40a6460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0005fd0 tx=0x7fd6c00095c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.165+0000 7fd6d2ff6740 1 Processor -- start 2024-04-22T06:49:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.166+0000 7fd6d2ff6740 1 -- start start 2024-04-22T06:49:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.167+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940fa8d0 0x5606940f9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.167+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x5606940f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 0x5606941054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56069405e8d0 con 0x5606940fa8d0 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560693ffbe80 con 0x5606940f9b60 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6cd8c6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x5606940f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6ce8c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 0x5606941054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6ce8c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 0x5606941054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:40358/0 (socket says 172.21.15.79:40358) 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.168+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/1157513229 learned_addr learned my addr 172.21.15.79:0/1157513229 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.169+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1157513229 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56069405a890 con 0x5606941030e0 2024-04-22T06:49:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.169+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940fa8d0 0x5606940f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.170+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/1157513229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 msgr2=0x5606940f9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.170+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x5606940f9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.170+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/1157513229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940fa8d0 msgr2=0x5606940f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.170+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940fa8d0 0x5606940f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.170+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/1157513229 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56069401d320 con 0x5606941030e0 2024-04-22T06:49:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.171+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 0x5606941054d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40c60a0 tx=0x7fd6c4035f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=255e5bd89646ffb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.176+0000 7fd6bcbfe640 1 -- 172.21.15.79:0/1157513229 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40bd960 con 0x5606941030e0 2024-04-22T06:49:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.176+0000 7fd6bcbfe640 1 -- 172.21.15.79:0/1157513229 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c40c1d10 con 0x5606941030e0 2024-04-22T06:49:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.178+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1157513229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 msgr2=0x5606941054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.178+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 0x5606941054d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40c60a0 tx=0x7fd6c4035f90 comp rx=0 tx=0).stop 2024-04-22T06:49:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.180+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1157513229 shutdown_connections 2024-04-22T06:49:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.180+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x5606940f9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.180+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606941030e0 0x5606941054d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.180+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1157513229 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940fa8d0 0x5606940f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.180+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1157513229 >> 172.21.15.79:0/1157513229 conn(0x5606940c0c90 msgr2=0x5606940f4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.182+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1157513229 shutdown_connections 2024-04-22T06:49:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.183+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1157513229 wait complete. 2024-04-22T06:49:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.183+0000 7fd6d2ff6740 1 -- start start 2024-04-22T06:49:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.183+0000 7fd6d2ff6740 1 Processor -- start 2024-04-22T06:49:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940f9b60 0x5606941029d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606941030e0 0x5606941002a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56069405a890 con 0x5606940f9b60 2024-04-22T06:49:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560693fe6eb0 con 0x5606941030e0 2024-04-22T06:49:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560694105bd0 con 0x5606940fa8d0 2024-04-22T06:49:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.184+0000 7fd6d2ff6740 10 client.0 _ll_register_callbacks cb 0x560694013b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.186+0000 7fd6cd8c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.187+0000 7fd6cd8c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940ffd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:40368/0 (socket says 172.21.15.79:40368) 2024-04-22T06:49:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.187+0000 7fd6cd8c6640 1 -- 172.21.15.79:0/3616508261 learned_addr learned my addr 172.21.15.79:0/3616508261 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.188+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940f9b60 0x5606941029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.188+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606941030e0 0x5606941002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:42 smithi193 ceph-mon[31815]: pgmap v167: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 88 op/s 2024-04-22T06:49:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:42 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:49:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:42 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:49:43.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:42 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:49:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.192+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3616508261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 msgr2=0x5606940ffd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.192+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940ffd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.192+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3616508261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940f9b60 msgr2=0x5606941029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.192+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940f9b60 0x5606941029d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.191 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.192+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3616508261 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560693d71210 con 0x5606941030e0 2024-04-22T06:49:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.195+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606941030e0 0x5606941002a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40c60a0 tx=0x7fd6c40bdf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c78abd05ad8f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.195+0000 7fd6b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:49:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.196+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/3616508261 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40d6060 con 0x5606941030e0 2024-04-22T06:49:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.197+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/3616508261 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fd6c40c6140 con 0x5606941030e0 2024-04-22T06:49:43.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.199+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c40c1d10 con 0x5606941030e0 2024-04-22T06:49:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.200+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/3616508261 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd6c40c65d0 con 0x5606941030e0 2024-04-22T06:49:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.200+0000 7fd6d2ff6740 10 client.25853 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.201+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x560693e3d620 con 0x5606941030e0 2024-04-22T06:49:43.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:42 smithi190 ceph-mon[31421]: pgmap v167: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 88 op/s 2024-04-22T06:49:43.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:42 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:49:43.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:42 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:49:43.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:42 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:49:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.205+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/3616508261 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c40c1d10 con 0x5606941030e0 2024-04-22T06:49:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.205+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/3616508261 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fd6c404e060 con 0x5606941030e0 2024-04-22T06:49:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.206+0000 7fd6d2ff6740 10 client.25853 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.206+0000 7fd6d2ff6740 10 client.25853 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.206+0000 7fd6d2ff6740 20 client.25853 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.206+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694112af0 0x560694114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.206+0000 7fd6d2ff6740 4 client.25853 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.206+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560694072880 con 0x560694112af0 2024-04-22T06:49:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.212+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694112af0 0x560694114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.220+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694112af0 0x560694114ee0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=242124bd6761fdfd server_cookie=1002acae7f238804 in_seq=0 out_seq=0 2024-04-22T06:49:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.220+0000 7fd6b1ffb640 10 client.25853 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.222+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/3616508261 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560694072880 con 0x560694112af0 2024-04-22T06:49:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.223+0000 7fd6b1ffb640 10 client.25853 handle_command_reply: tid=0 2024-04-22T06:49:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 client.25853 shutdown 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606941030e0 msgr2=0x5606941002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606941030e0 0x5606941002a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40c60a0 tx=0x7fd6c40bdf70 comp rx=0 tx=0).stop 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 shutdown_connections 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694112af0 0x560694114ee0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606941030e0 0x5606941002a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3616508261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606940f9b60 0x5606941029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 >> 172.21.15.79:0/3616508261 conn(0x5606940c0c90 msgr2=0x5606940f65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.226+0000 7fd6b1ffb640 0 client.25853 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.227+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 shutdown_connections 2024-04-22T06:49:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.227+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3616508261 wait complete. 2024-04-22T06:49:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.227+0000 7fd6d2ff6740 20 client.25853 trim_cache size 0 max 16384 2024-04-22T06:49:43.246 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:42 smithi079 ceph-mon[27946]: pgmap v167: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 88 op/s 2024-04-22T06:49:43.246 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:42 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:49:43.246 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:42 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:49:43.246 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:42 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:49:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.327+0000 7fd6d2ff6740 1 Processor -- start 2024-04-22T06:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.327+0000 7fd6d2ff6740 1 -- start start 2024-04-22T06:49:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.328+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940f9b60 0x56069410e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.328+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940fa8d0 0x5606940f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.328+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x560694102490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.329+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81c7e10 con 0x5606941030e0 2024-04-22T06:49:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.329+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81c56e0 con 0x5606940fa8d0 2024-04-22T06:49:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.329+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81c3530 con 0x5606940f9b60 2024-04-22T06:49:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.329+0000 7fd6ce8c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x560694102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.330+0000 7fd6ce8c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x560694102490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38612/0 (socket says 172.21.15.79:38612) 2024-04-22T06:49:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.330+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3091999542 learned_addr learned my addr 172.21.15.79:0/3091999542 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.332+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3091999542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940f9b60 msgr2=0x56069410e960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.332+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940f9b60 0x56069410e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.332+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3091999542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940fa8d0 msgr2=0x5606940f3750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.332+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940fa8d0 0x5606940f3750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.332+0000 7fd6ce8c8640 1 -- 172.21.15.79:0/3091999542 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606940f8e30 con 0x5606941030e0 2024-04-22T06:49:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.334+0000 7fd6ce8c8640 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x560694102490 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40ba330 tx=0x7fd6c4035f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cece12b93b5073b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.339+0000 7fd6b3fff640 1 -- 172.21.15.79:0/3091999542 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40cb080 con 0x5606941030e0 2024-04-22T06:49:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.340+0000 7fd6b3fff640 1 -- 172.21.15.79:0/3091999542 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c40bd960 con 0x5606941030e0 2024-04-22T06:49:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.340+0000 7fd6b3fff640 1 -- 172.21.15.79:0/3091999542 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40bd730 con 0x5606941030e0 2024-04-22T06:49:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.341+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3091999542 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 msgr2=0x560694102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.341+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x560694102490 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40ba330 tx=0x7fd6c4035f90 comp rx=0 tx=0).stop 2024-04-22T06:49:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.342+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3091999542 shutdown_connections 2024-04-22T06:49:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.343+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940fa8d0 0x5606940f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.343+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x560694102490 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.343+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/3091999542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940f9b60 0x56069410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.343+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3091999542 >> 172.21.15.79:0/3091999542 conn(0x5606940c0c90 msgr2=0x5606940f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.348+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3091999542 shutdown_connections 2024-04-22T06:49:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.348+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/3091999542 wait complete. 2024-04-22T06:49:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.349+0000 7fd6d2ff6740 1 -- start start 2024-04-22T06:49:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.349+0000 7fd6d2ff6740 1 Processor -- start 2024-04-22T06:49:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.350+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x56069410b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.351+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.351+0000 7fd6d2ff6740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x5606940fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.351+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81c3ba0 con 0x5606941030e0 2024-04-22T06:49:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.352+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81c5010 con 0x5606940f9b60 2024-04-22T06:49:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.352+0000 7fd6d2ff6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81c4270 con 0x5606940fa8d0 2024-04-22T06:49:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.352+0000 7fd6d2ff6740 10 client.0 _ll_register_callbacks cb 0x7fd6b8009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.353+0000 7fd6cd8c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.354+0000 7fd6cd8c6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:40386/0 (socket says 172.21.15.79:40386) 2024-04-22T06:49:43.352 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.354+0000 7fd6cd8c6640 1 -- 172.21.15.79:0/1390759866 learned_addr learned my addr 172.21.15.79:0/1390759866 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:49:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.355+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x56069410b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.357+0000 7fd6cd8c6640 1 -- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 msgr2=0x56069410b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.357+0000 7fd6cd8c6640 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x56069410b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.357+0000 7fd6cd8c6640 1 -- 172.21.15.79:0/1390759866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 msgr2=0x5606940fcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.357+0000 7fd6cd8c6640 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x5606940fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.357+0000 7fd6cd8c6640 1 -- 172.21.15.79:0/1390759866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606940fd3e0 con 0x5606940fa8d0 2024-04-22T06:49:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.359+0000 7fd6cd8c6640 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940fc6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8005940 tx=0x7fd6b8013b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ae0c026274d56fb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.362+0000 7fd6b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:49:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.362+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/1390759866 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8011b90 con 0x5606940fa8d0 2024-04-22T06:49:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.362+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/1390759866 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8985+0+0 (secure 0 0 0) 0x7fd6b8011d30 con 0x5606940fa8d0 2024-04-22T06:49:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.364+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c40bd960 con 0x5606940fa8d0 2024-04-22T06:49:43.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.369+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/1390759866 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd6b8004020 con 0x5606940fa8d0 2024-04-22T06:49:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.369+0000 7fd6d2ff6740 10 client.25849 fetch_fsmap learned FSMap version 47 2024-04-22T06:49:43.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.369+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x560694178e80 con 0x5606940fa8d0 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.370+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/1390759866 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c40bd960 con 0x5606940fa8d0 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/1390759866 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 47) ==== 8338+0+0 (secure 0 0 0) 0x7fd6b800f040 con 0x5606940fa8d0 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6d2ff6740 10 client.25849 fetch_fsmap finished waiting for FSMap version 47 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6d2ff6740 10 client.25849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6d2ff6740 20 client.25849 populate_metadata read hostname 'smithi079' 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694179660 0x56069417ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6d2ff6740 4 client.25849 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.371+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560693ffe3d0 con 0x560694179660 2024-04-22T06:49:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.374+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694179660 0x56069417ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.387+0000 7fd6ce0c7640 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694179660 0x56069417ba80 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e680c494881924 server_cookie=1cd36d94863109a5 in_seq=0 out_seq=0 2024-04-22T06:49:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:43.387+0000 7fd6b1ffb640 10 client.25849 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:45.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:44 smithi193 ceph-mon[31815]: pgmap v168: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 311 op/s 2024-04-22T06:49:45.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:44 smithi190 ceph-mon[31421]: pgmap v168: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 311 op/s 2024-04-22T06:49:45.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:44 smithi079 ceph-mon[27946]: pgmap v168: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 311 op/s 2024-04-22T06:49:47.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:46 smithi193 ceph-mon[31815]: pgmap v169: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 309 op/s 2024-04-22T06:49:47.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:46 smithi190 ceph-mon[31421]: pgmap v169: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 309 op/s 2024-04-22T06:49:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:46 smithi079 ceph-mon[27946]: pgmap v169: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 309 op/s 2024-04-22T06:49:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:48 smithi193 ceph-mon[31815]: pgmap v170: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 227 MiB/s wr, 1.14k op/s 2024-04-22T06:49:49.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:48 smithi190 ceph-mon[31421]: pgmap v170: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 227 MiB/s wr, 1.14k op/s 2024-04-22T06:49:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:48 smithi079 ceph-mon[27946]: pgmap v170: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 227 MiB/s wr, 1.14k op/s 2024-04-22T06:49:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:50 smithi193 ceph-mon[31815]: pgmap v171: 97 pgs: 97 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 307 MiB/s wr, 1.34k op/s 2024-04-22T06:49:51.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:50 smithi190 ceph-mon[31421]: pgmap v171: 97 pgs: 97 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 307 MiB/s wr, 1.34k op/s 2024-04-22T06:49:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:50 smithi079 ceph-mon[27946]: pgmap v171: 97 pgs: 97 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 307 MiB/s wr, 1.34k op/s 2024-04-22T06:49:51.958 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:51 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:49:51.958 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:51 smithi193 systemd[1]: Stopped Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:51.958 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:51 smithi193 systemd[1]: Starting Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:49:51.958 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:51 smithi193 podman[67567]: 2024-04-22 06:49:51.840763605 +0000 UTC m=+0.020587562 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:51.958 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:51 smithi193 podman[67567]: 2024-04-22 06:49:51.959677342 +0000 UTC m=+0.139501290 container create 6a31b953468a0696b78a68a3f013bb268fc6cc63d9d9268c68228fdadde3b95b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD) 2024-04-22T06:49:52.001 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:51 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:49:52.001 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:51 smithi190 systemd[1]: Stopped Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:52.001 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:51 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.106s CPU time. 2024-04-22T06:49:52.001 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:51 smithi190 systemd[1]: Starting Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:49:52.287 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 podman[66474]: 2024-04-22 06:49:51.907825284 +0000 UTC m=+0.039726848 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:52.287 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 podman[66474]: 2024-04-22 06:49:52.053247854 +0000 UTC m=+0.185149407 container create 4561aeb1feeef4b27b523b92b4e71e681b9c4eff092c2a25bc4c11ecdab2821b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:52.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:52.618+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:52.625+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:49:52.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:52.625+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:49:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:52.625+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:49:52.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:49:52.625+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:49:52.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 podman[66474]: 2024-04-22 06:49:52.29010252 +0000 UTC m=+0.422004083 container init 4561aeb1feeef4b27b523b92b4e71e681b9c4eff092c2a25bc4c11ecdab2821b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T06:49:52.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 podman[66474]: 2024-04-22 06:49:52.295417845 +0000 UTC m=+0.427319391 container start 4561aeb1feeef4b27b523b92b4e71e681b9c4eff092c2a25bc4c11ecdab2821b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:52.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: starting mds.e at 2024-04-22T06:49:52.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 bash[66474]: 4561aeb1feeef4b27b523b92b4e71e681b9c4eff092c2a25bc4c11ecdab2821b 2024-04-22T06:49:52.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:49:52 smithi190 systemd[1]: Started Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:52.817 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:52 smithi193 podman[67567]: 2024-04-22 06:49:52.44686932 +0000 UTC m=+0.626693286 container init 6a31b953468a0696b78a68a3f013bb268fc6cc63d9d9268c68228fdadde3b95b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T06:49:52.817 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:52 smithi193 podman[67567]: 2024-04-22 06:49:52.454036099 +0000 UTC m=+0.633860049 container start 6a31b953468a0696b78a68a3f013bb268fc6cc63d9d9268c68228fdadde3b95b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T06:49:52.817 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mds[67587]: -- 172.21.15.193:0/2782796826 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5600d0891a40 con 0x5600d0865000 2024-04-22T06:49:52.817 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: starting mds.f at 2024-04-22T06:49:52.817 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:52 smithi193 bash[67567]: 6a31b953468a0696b78a68a3f013bb268fc6cc63d9d9268c68228fdadde3b95b 2024-04-22T06:49:52.818 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:49:52 smithi193 systemd[1]: Started Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: pgmap v172: 97 pgs: 97 active+clean; 36 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 458 MiB/s wr, 1.72k op/s 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: daemon mds.e restarted 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:boot 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: Replacing daemon mds.f as rank 2 with standby daemon mds.i 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,2=f=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:49:53.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:49:53.095 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:52 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:49:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: pgmap v172: 97 pgs: 97 active+clean; 36 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 458 MiB/s wr, 1.72k op/s 2024-04-22T06:49:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: daemon mds.e restarted 2024-04-22T06:49:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:49:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:49:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:boot 2024-04-22T06:49:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: Replacing daemon mds.f as rank 2 with standby daemon mds.i 2024-04-22T06:49:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-22T06:49:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:49:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,2=f=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:49:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:49:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:49:53.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:52 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: pgmap v172: 97 pgs: 97 active+clean; 36 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 458 MiB/s wr, 1.72k op/s 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: daemon mds.e restarted 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:boot 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: Replacing daemon mds.f as rank 2 with standby daemon mds.i 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:49:53.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,2=f=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:49:53.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:49:53.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:49:53.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:52 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6b1ffb640 1 -- 172.21.15.79:0/1390759866 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560693ffe3d0 con 0x560694179660 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6b1ffb640 10 client.25849 handle_command_reply: tid=0 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 client.25849 shutdown 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 msgr2=0x5606940fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940fc6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8005940 tx=0x7fd6b8013b70 comp rx=0 tx=0).stop 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 shutdown_connections 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560694179660 0x56069417ba80 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606941030e0 0x5606940fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606940fa8d0 0x5606940fc6d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 --2- 172.21.15.79:0/1390759866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606940f9b60 0x56069410b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 >> 172.21.15.79:0/1390759866 conn(0x5606940c0c90 msgr2=0x5606940f64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.312+0000 7fd6b1ffb640 0 client.25849 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.313+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 shutdown_connections 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.313+0000 7fd6d2ff6740 1 -- 172.21.15.79:0/1390759866 wait complete. 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.313+0000 7fd6d2ff6740 20 client.25849 trim_cache size 0 max 16384 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:49:53.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:49:53.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:49:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.325+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6b40a3fa0 msgr2=0x7fd6b40a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.325+0000 7fd6d032b640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6b40a3fa0 0x7fd6b40a6460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0005fd0 tx=0x7fd6c00095c0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.326+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 msgr2=0x7fd6c8115880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.326+0000 7fd6d032b640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 0x7fd6c8115880 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fd6c402c150 tx=0x7fd6c4035f20 comp rx=0 tx=0).stop 2024-04-22T06:49:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.326+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 shutdown_connections 2024-04-22T06:49:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.326+0000 7fd6d032b640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6c81153d0 0x7fd6c8115880 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.326+0000 7fd6d032b640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6c811b730 0x7fd6c8114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.327+0000 7fd6d032b640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd6b40a3fa0 0x7fd6b40a6460 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.327+0000 7fd6d032b640 1 --2- 172.21.15.79:0/3041544528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6c8110740 0x7fd6c811ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.327+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 >> 172.21.15.79:0/3041544528 conn(0x7fd6c806d020 msgr2=0x7fd6c811bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.330+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 shutdown_connections 2024-04-22T06:49:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:49:53.330+0000 7fd6d032b640 1 -- 172.21.15.79:0/3041544528 wait complete. 2024-04-22T06:49:54.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:54 smithi193 ceph-mon[31815]: pgmap v175: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 858 MiB/s wr, 2.83k op/s 2024-04-22T06:49:54.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:54 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:boot 2024-04-22T06:49:54.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:54 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:resolve 2024-04-22T06:49:54.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:54 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:resolve 2024-04-22T06:49:54.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:54 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:54.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:54 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:49:54.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:54 smithi190 ceph-mon[31421]: pgmap v175: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 858 MiB/s wr, 2.83k op/s 2024-04-22T06:49:54.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:54 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:boot 2024-04-22T06:49:54.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:54 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:resolve 2024-04-22T06:49:54.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:54 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:resolve 2024-04-22T06:49:54.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:54 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:54.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:54 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:49:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:54 smithi079 ceph-mon[27946]: pgmap v175: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 858 MiB/s wr, 2.83k op/s 2024-04-22T06:49:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:54 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:boot 2024-04-22T06:49:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:54 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:resolve 2024-04-22T06:49:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:54 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:resolve 2024-04-22T06:49:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:54 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:54 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:49:55.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00499986 2024-04-22T06:49:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499986 2024-04-22T06:49:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00499992 2024-04-22T06:49:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499992 2024-04-22T06:49:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:reconnect 2024-04-22T06:49:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:reconnect 2024-04-22T06:49:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:55 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00499986 2024-04-22T06:49:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499986 2024-04-22T06:49:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00499992 2024-04-22T06:49:55.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499992 2024-04-22T06:49:55.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:reconnect 2024-04-22T06:49:55.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:reconnect 2024-04-22T06:49:55.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:55 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:55.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00499986 2024-04-22T06:49:55.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499986 2024-04-22T06:49:55.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.00499992 2024-04-22T06:49:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499992 2024-04-22T06:49:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:reconnect 2024-04-22T06:49:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:reconnect 2024-04-22T06:49:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:55 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: pgmap v176: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 556 MiB/s wr, 1.58k op/s 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:rejoin 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:rejoin 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:active 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:active 2024-04-22T06:49:56.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:56 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:49:56.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: pgmap v176: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 556 MiB/s wr, 1.58k op/s 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:rejoin 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:rejoin 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:active 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:active 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:56 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:49:56.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: pgmap v176: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 556 MiB/s wr, 1.58k op/s 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:rejoin 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:rejoin 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/1413649941,v1:172.21.15.193:6839/1413649941] up:active 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1169559599,v1:172.21.15.190:6837/1169559599] up:active 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:56.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:56 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:49:58.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:49:58 smithi193 ceph-mon[31815]: pgmap v177: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 738 MiB/s wr, 2.06k op/s 2024-04-22T06:49:58.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:49:58 smithi190 ceph-mon[31421]: pgmap v177: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 738 MiB/s wr, 2.06k op/s 2024-04-22T06:49:58.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:49:58 smithi079 ceph-mon[27946]: pgmap v177: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 738 MiB/s wr, 2.06k op/s 2024-04-22T06:50:00.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:00 smithi193 ceph-mon[31815]: pgmap v178: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 607 MiB/s wr, 1.63k op/s 2024-04-22T06:50:00.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:00 smithi193 ceph-mon[31815]: overall HEALTH_OK 2024-04-22T06:50:00.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:00 smithi190 ceph-mon[31421]: pgmap v178: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 607 MiB/s wr, 1.63k op/s 2024-04-22T06:50:00.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:00 smithi190 ceph-mon[31421]: overall HEALTH_OK 2024-04-22T06:50:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:00 smithi079 ceph-mon[27946]: pgmap v178: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 607 MiB/s wr, 1.63k op/s 2024-04-22T06:50:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:00 smithi079 ceph-mon[27946]: overall HEALTH_OK 2024-04-22T06:50:02.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:02 smithi193 ceph-mon[31815]: pgmap v179: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 663 MiB/s wr, 1.65k op/s 2024-04-22T06:50:02.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:02 smithi190 ceph-mon[31421]: pgmap v179: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 663 MiB/s wr, 1.65k op/s 2024-04-22T06:50:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:02 smithi079 ceph-mon[27946]: pgmap v179: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 663 MiB/s wr, 1.65k op/s 2024-04-22T06:50:04.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:04 smithi193 ceph-mon[31815]: pgmap v180: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 660 MiB/s wr, 1.51k op/s 2024-04-22T06:50:04.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:04 smithi190 ceph-mon[31421]: pgmap v180: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 660 MiB/s wr, 1.51k op/s 2024-04-22T06:50:04.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:04 smithi079 ceph-mon[27946]: pgmap v180: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 660 MiB/s wr, 1.51k op/s 2024-04-22T06:50:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.557+0000 7fc67759e640 1 -- 172.21.15.79:0/916351803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700051f0 msgr2=0x7fc670005670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.557+0000 7fc67759e640 1 --2- 172.21.15.79:0/916351803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700051f0 0x7fc670005670 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc6780676c0 tx=0x7fc678098250 comp rx=0 tx=0).stop 2024-04-22T06:50:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.558+0000 7fc67759e640 1 -- 172.21.15.79:0/916351803 shutdown_connections 2024-04-22T06:50:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.558+0000 7fc67759e640 1 --2- 172.21.15.79:0/916351803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6700986e0 0x7fc670098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.558+0000 7fc67759e640 1 --2- 172.21.15.79:0/916351803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700051f0 0x7fc670005670 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.558+0000 7fc67759e640 1 --2- 172.21.15.79:0/916351803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc67009acf0 0x7fc670004cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.558+0000 7fc67759e640 1 -- 172.21.15.79:0/916351803 >> 172.21.15.79:0/916351803 conn(0x7fc67000a740 msgr2=0x7fc67000ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.565+0000 7fc67759e640 1 -- 172.21.15.79:0/916351803 shutdown_connections 2024-04-22T06:50:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.566+0000 7fc67759e640 1 -- 172.21.15.79:0/916351803 wait complete. 2024-04-22T06:50:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.567+0000 7fc67759e640 1 Processor -- start 2024-04-22T06:50:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 -- start start 2024-04-22T06:50:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6700051f0 0x7fc6700a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 0x7fc6700a2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc67009acf0 0x7fc6700a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc6700ac530 con 0x7fc6700986e0 2024-04-22T06:50:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc6700ac3b0 con 0x7fc67009acf0 2024-04-22T06:50:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.568+0000 7fc67759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc6700ac6b0 con 0x7fc6700051f0 2024-04-22T06:50:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.569+0000 7fc675d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 0x7fc6700a2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.569+0000 7fc675d9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 0x7fc6700a2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43126/0 (socket says 172.21.15.79:43126) 2024-04-22T06:50:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.569+0000 7fc675d9b640 1 -- 172.21.15.79:0/3900222613 learned_addr learned my addr 172.21.15.79:0/3900222613 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.569+0000 7fc676d9d640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc67009acf0 0x7fc6700a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.570+0000 7fc675d9b640 1 -- 172.21.15.79:0/3900222613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6700051f0 msgr2=0x7fc6700a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.570+0000 7fc675d9b640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6700051f0 0x7fc6700a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.570+0000 7fc675d9b640 1 -- 172.21.15.79:0/3900222613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc67009acf0 msgr2=0x7fc6700a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.570+0000 7fc675d9b640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc67009acf0 0x7fc6700a2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.570+0000 7fc675d9b640 1 -- 172.21.15.79:0/3900222613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6700a2ed0 con 0x7fc6700986e0 2024-04-22T06:50:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.571+0000 7fc675d9b640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 0x7fc6700a2350 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc678051690 tx=0x7fc678068b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.572+0000 7fc6677fe640 1 -- 172.21.15.79:0/3900222613 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67806b370 con 0x7fc6700986e0 2024-04-22T06:50:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.572+0000 7fc6677fe640 1 -- 172.21.15.79:0/3900222613 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc67806b510 con 0x7fc6700986e0 2024-04-22T06:50:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.572+0000 7fc6677fe640 1 -- 172.21.15.79:0/3900222613 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6780a9640 con 0x7fc6700986e0 2024-04-22T06:50:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.572+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6700a3160 con 0x7fc6700986e0 2024-04-22T06:50:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.572+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6700a70f0 con 0x7fc6700986e0 2024-04-22T06:50:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.586+0000 7fc67f608740 1 Processor -- start 2024-04-22T06:50:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.587+0000 7fc6677fe640 1 -- 172.21.15.79:0/3900222613 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc67809f5b0 con 0x7fc6700986e0 2024-04-22T06:50:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 -- start start 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc6677fe640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc6600a3f30 0x7fc6600a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f6c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560872ec58d0 con 0x560872f6a2d0 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560872e62e80 con 0x560872f60d50 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.589+0000 7fc67f608740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560872ec1890 con 0x560872f61ac0 2024-04-22T06:50:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.590+0000 7fc6677fe640 1 -- 172.21.15.79:0/3900222613 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7fc67815e750 con 0x7fc6700986e0 2024-04-22T06:50:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.590+0000 7fc676d9d640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc6600a3f30 0x7fc6600a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.590+0000 7fc676d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f6c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.590+0000 7fc676d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f6c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43128/0 (socket says 172.21.15.79:43128) 2024-04-22T06:50:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.590+0000 7fc676d9d640 1 -- 172.21.15.79:0/3734525730 learned_addr learned my addr 172.21.15.79:0/3734525730 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.591+0000 7fc67659c640 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.592+0000 7fc676d9d640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc6600a3f30 0x7fc6600a63f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc66c015410 tx=0x7fc66c00c7c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.593+0000 7fc676d9d640 1 -- 172.21.15.79:0/3734525730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 msgr2=0x560872f60810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.593+0000 7fc676d9d640 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f60810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.593+0000 7fc676d9d640 1 -- 172.21.15.79:0/3734525730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 msgr2=0x560872f611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.593+0000 7fc676d9d640 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.593+0000 7fc676d9d640 1 -- 172.21.15.79:0/3734525730 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560872e84320 con 0x560872f6a2d0 2024-04-22T06:50:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.594+0000 7fc676d9d640 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f6c6c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0186c0 tx=0x7fc66c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f52a0325600f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.595+0000 7fc664ff9640 1 -- 172.21.15.79:0/3734525730 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c01c040 con 0x560872f6a2d0 2024-04-22T06:50:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.595+0000 7fc664ff9640 1 -- 172.21.15.79:0/3734525730 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c003690 con 0x560872f6a2d0 2024-04-22T06:50:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.597+0000 7fc67f608740 1 -- 172.21.15.79:0/3734525730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 msgr2=0x560872f6c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.597+0000 7fc67f608740 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f6c6c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0186c0 tx=0x7fc66c01a000 comp rx=0 tx=0).stop 2024-04-22T06:50:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.598+0000 7fc67f608740 1 -- 172.21.15.79:0/3734525730 shutdown_connections 2024-04-22T06:50:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.598+0000 7fc67f608740 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.598+0000 7fc67f608740 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f6c6c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.598+0000 7fc67f608740 1 --2- 172.21.15.79:0/3734525730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f60810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.599 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.598+0000 7fc67f608740 1 -- 172.21.15.79:0/3734525730 >> 172.21.15.79:0/3734525730 conn(0x560872f27c90 msgr2=0x560872f5ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.604+0000 7fc67f608740 1 -- 172.21.15.79:0/3734525730 shutdown_connections 2024-04-22T06:50:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.605+0000 7fc67f608740 1 -- 172.21.15.79:0/3734525730 wait complete. 2024-04-22T06:50:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.607+0000 7fc67f608740 1 -- start start 2024-04-22T06:50:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.607+0000 7fc67f608740 1 Processor -- start 2024-04-22T06:50:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.608+0000 7fc67f608740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.608+0000 7fc67f608740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f66fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.608+0000 7fc67f608740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f67530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.608+0000 7fc67f608740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x560872f6cf90 con 0x560872f6a2d0 2024-04-22T06:50:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.609+0000 7fc67f608740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560872e4deb0 con 0x560872f60d50 2024-04-22T06:50:06.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.609+0000 7fc67f608740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560872f6d110 con 0x560872f61ac0 2024-04-22T06:50:06.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.609+0000 7fc67f608740 10 client.0 _ll_register_callbacks cb 0x560872e7ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:06.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.609+0000 7fc676d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f67530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.612+0000 7fc676d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f67530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43138/0 (socket says 172.21.15.79:43138) 2024-04-22T06:50:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.612+0000 7fc676d9d640 1 -- 172.21.15.79:0/419780867 learned_addr learned my addr 172.21.15.79:0/419780867 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.612+0000 7fc67659c640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.612+0000 7fc675d9b640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f66fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.613+0000 7fc676d9d640 1 -- 172.21.15.79:0/419780867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 msgr2=0x560872f66fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.613+0000 7fc676d9d640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f66fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.613+0000 7fc676d9d640 1 -- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 msgr2=0x560872f69c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.613+0000 7fc676d9d640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.613+0000 7fc676d9d640 1 -- 172.21.15.79:0/419780867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560872bd8210 con 0x560872f6a2d0 2024-04-22T06:50:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.614+0000 7fc676d9d640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f67530 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc66c016010 tx=0x7fc66c004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6ed0364a9788ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.615+0000 7fc65dffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.615+0000 7fc65dffb640 1 -- 172.21.15.79:0/419780867 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c02e020 con 0x560872f6a2d0 2024-04-22T06:50:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.615+0000 7fc65dffb640 1 -- 172.21.15.79:0/419780867 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7fc66c017840 con 0x560872f6a2d0 2024-04-22T06:50:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.615+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66c003690 con 0x560872f6a2d0 2024-04-22T06:50:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.616+0000 7fc65dffb640 1 -- 172.21.15.79:0/419780867 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fc66c01c040 con 0x560872f6a2d0 2024-04-22T06:50:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.619+0000 7fc67f608740 10 client.16953 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.619+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x560872ca4620 con 0x560872f6a2d0 2024-04-22T06:50:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.620+0000 7fc65dffb640 1 -- 172.21.15.79:0/419780867 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c00c4d0 con 0x560872f6a2d0 2024-04-22T06:50:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc65dffb640 1 -- 172.21.15.79:0/419780867 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7fc66c02a570 con 0x560872f6a2d0 2024-04-22T06:50:06.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc67f608740 10 client.16953 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc67f608740 10 client.16953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc67f608740 20 client.16953 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc67f608740 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f7a050 0x560872f7c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc67f608740 4 client.16953 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.622+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc66c003690 con 0x560872f7a050 2024-04-22T06:50:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.625+0000 7fc67659c640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f7a050 0x560872f7c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.634 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.635+0000 7fc67659c640 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f7a050 0x560872f7c440 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24c40e3c73f5bc00 server_cookie=79bacda0c73a98a6 in_seq=0 out_seq=0 2024-04-22T06:50:06.634 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.636+0000 7fc65dffb640 10 client.16953 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.638+0000 7fc65dffb640 1 -- 172.21.15.79:0/419780867 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc66c003690 con 0x560872f7a050 2024-04-22T06:50:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.638+0000 7fc65dffb640 10 client.16953 handle_command_reply: tid=0 2024-04-22T06:50:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.638+0000 7fc67f608740 1 client.16953 shutdown 2024-04-22T06:50:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.639+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 msgr2=0x560872f67530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.639+0000 7fc67f608740 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f67530 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc66c016010 tx=0x7fc66c004250 comp rx=0 tx=0).stop 2024-04-22T06:50:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 shutdown_connections 2024-04-22T06:50:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc67f608740 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f7a050 0x560872f7c440 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc67f608740 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f6a2d0 0x560872f67530 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc67f608740 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f61ac0 0x560872f66fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc67f608740 1 --2- 172.21.15.79:0/419780867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 >> 172.21.15.79:0/419780867 conn(0x560872f27c90 msgr2=0x560872f5d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.643+0000 7fc65dffb640 0 client.16953 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.647+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 shutdown_connections 2024-04-22T06:50:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.647+0000 7fc67f608740 1 -- 172.21.15.79:0/419780867 wait complete. 2024-04-22T06:50:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.650+0000 7fc67f608740 20 client.16953 trim_cache size 0 max 16384 2024-04-22T06:50:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.689+0000 7fc67f608740 1 Processor -- start 2024-04-22T06:50:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.689+0000 7fc67f608740 1 -- start start 2024-04-22T06:50:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.690+0000 7fc67f608740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f75b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.690+0000 7fc67f608740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f27c90 0x560872f69700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.690+0000 7fc67f608740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.690+0000 7fc67f608740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc658011600 con 0x560872f60d50 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.690+0000 7fc67f608740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc65800eed0 con 0x560872f27c90 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.690+0000 7fc67f608740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc65800cdd0 con 0x560872f608b0 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc67659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f75b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc675d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f27c90 0x560872f69700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43152/0 (socket says 172.21.15.79:43152) 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 -- 172.21.15.79:0/1084994801 learned_addr learned my addr 172.21.15.79:0/1084994801 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 -- 172.21.15.79:0/1084994801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 msgr2=0x560872f75b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f75b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 -- 172.21.15.79:0/1084994801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f27c90 msgr2=0x560872f69700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f27c90 0x560872f69700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.691+0000 7fc676d9d640 1 -- 172.21.15.79:0/1084994801 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560872f61540 con 0x560872f60d50 2024-04-22T06:50:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.692+0000 7fc676d9d640 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 0x560872f69c40 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc66c018fd0 tx=0x7fc66c006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40368e1efa7ff8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.693+0000 7fc65ffff640 1 -- 172.21.15.79:0/1084994801 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c03f020 con 0x560872f60d50 2024-04-22T06:50:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.693+0000 7fc65ffff640 1 -- 172.21.15.79:0/1084994801 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c003690 con 0x560872f60d50 2024-04-22T06:50:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.694+0000 7fc67f608740 1 -- 172.21.15.79:0/1084994801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 msgr2=0x560872f69c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.694+0000 7fc67f608740 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 0x560872f69c40 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc66c018fd0 tx=0x7fc66c006ce0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.694+0000 7fc67f608740 1 -- 172.21.15.79:0/1084994801 shutdown_connections 2024-04-22T06:50:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.694+0000 7fc67f608740 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f27c90 0x560872f69700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.695+0000 7fc67f608740 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f60d50 0x560872f69c40 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.695+0000 7fc67f608740 1 --2- 172.21.15.79:0/1084994801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f75b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.695+0000 7fc67f608740 1 -- 172.21.15.79:0/1084994801 >> 172.21.15.79:0/1084994801 conn(0x560872f62a20 msgr2=0x560872f75480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.695+0000 7fc67f608740 1 -- 172.21.15.79:0/1084994801 shutdown_connections 2024-04-22T06:50:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.695+0000 7fc67f608740 1 -- 172.21.15.79:0/1084994801 wait complete. 2024-04-22T06:50:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.695+0000 7fc67f608740 1 -- start start 2024-04-22T06:50:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 Processor -- start 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 0x560872f7cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f7d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc65800d390 con 0x560872f27c90 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc65800e800 con 0x560872f60d50 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc65800da60 con 0x560872f608b0 2024-04-22T06:50:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.696+0000 7fc67f608740 10 client.0 _ll_register_callbacks cb 0x7fc668006ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.697+0000 7fc67659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 0x560872f7cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.697+0000 7fc67659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 0x560872f7cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43154/0 (socket says 172.21.15.79:43154) 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.697+0000 7fc67659c640 1 -- 172.21.15.79:0/1597357908 learned_addr learned my addr 172.21.15.79:0/1597357908 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.698+0000 7fc67659c640 1 -- 172.21.15.79:0/1597357908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 msgr2=0x560872f7d020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.698+0000 7fc67659c640 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f7d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.698+0000 7fc67659c640 1 -- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 msgr2=0x560872f7d560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.698+0000 7fc67659c640 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f7d560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.698+0000 7fc67659c640 1 -- 172.21.15.79:0/1597357908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560872fdf260 con 0x560872f27c90 2024-04-22T06:50:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.699+0000 7fc67659c640 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 0x560872f7cae0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc668006cf0 tx=0x7fc668009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b774a025b5ac865b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.699+0000 7fc65dffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.700+0000 7fc65dffb640 1 -- 172.21.15.79:0/1597357908 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc668005e20 con 0x560872f27c90 2024-04-22T06:50:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.700+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66c003690 con 0x560872f27c90 2024-04-22T06:50:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.700+0000 7fc65dffb640 1 -- 172.21.15.79:0/1597357908 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7fc6680190c0 con 0x560872f27c90 2024-04-22T06:50:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.702+0000 7fc65dffb640 1 -- 172.21.15.79:0/1597357908 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fc66802b050 con 0x560872f27c90 2024-04-22T06:50:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.702+0000 7fc67f608740 10 client.16959 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.703+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x560872fe0590 con 0x560872f27c90 2024-04-22T06:50:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.703+0000 7fc65dffb640 1 -- 172.21.15.79:0/1597357908 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc66c003690 con 0x560872f27c90 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc65dffb640 1 -- 172.21.15.79:0/1597357908 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7fc668020020 con 0x560872f27c90 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc67f608740 10 client.16959 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc67f608740 10 client.16959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc67f608740 20 client.16959 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc67f608740 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f61ac0 0x560872fe2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc67f608740 4 client.16959 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.704+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560872ed9880 con 0x560872f61ac0 2024-04-22T06:50:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.706+0000 7fc675d9b640 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f61ac0 0x560872fe2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.712+0000 7fc675d9b640 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f61ac0 0x560872fe2d20 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9172293f765198a server_cookie=9d5d71414df9ae65 in_seq=0 out_seq=0 2024-04-22T06:50:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.712+0000 7fc65dffb640 10 client.16959 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.879+0000 7fc65dffb640 1 -- 172.21.15.79:0/1597357908 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560872ed9880 con 0x560872f61ac0 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.879+0000 7fc65dffb640 10 client.16959 handle_command_reply: tid=0 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.879+0000 7fc67f608740 1 client.16959 shutdown 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.879+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 msgr2=0x560872f7cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.879+0000 7fc67f608740 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 0x560872f7cae0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc668006cf0 tx=0x7fc668009ae0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 shutdown_connections 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560872f61ac0 0x560872fe2d20 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560872f60d50 0x560872f7d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560872f608b0 0x560872f7d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 --2- 172.21.15.79:0/1597357908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x560872f27c90 0x560872f7cae0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 >> 172.21.15.79:0/1597357908 conn(0x560872f62a20 msgr2=0x560872f5d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc65dffb640 0 client.16959 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 shutdown_connections 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 1 -- 172.21.15.79:0/1597357908 wait complete. 2024-04-22T06:50:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.880+0000 7fc67f608740 20 client.16959 trim_cache size 0 max 16384 2024-04-22T06:50:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:50:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:50:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:50:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:50:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc6600a3f30 msgr2=0x7fc6600a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc6600a3f30 0x7fc6600a63f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc66c015410 tx=0x7fc66c00c7c0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 msgr2=0x7fc6700a2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 0x7fc6700a2350 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc678051690 tx=0x7fc678068b00 comp rx=0 tx=0).stop 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 shutdown_connections 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc6600a3f30 0x7fc6600a63f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc67009acf0 0x7fc6700a2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc6700986e0 0x7fc6700a2350 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 --2- 172.21.15.79:0/3900222613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6700051f0 0x7fc6700a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.881+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 >> 172.21.15.79:0/3900222613 conn(0x7fc67000a740 msgr2=0x7fc6700976c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.882+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 shutdown_connections 2024-04-22T06:50:06.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:06.882+0000 7fc67759e640 1 -- 172.21.15.79:0/3900222613 wait complete. 2024-04-22T06:50:06.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:06 smithi193 ceph-mon[31815]: pgmap v181: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 553 MiB/s wr, 1.21k op/s 2024-04-22T06:50:06.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:06 smithi190 ceph-mon[31421]: pgmap v181: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 553 MiB/s wr, 1.21k op/s 2024-04-22T06:50:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:06 smithi079 ceph-mon[27946]: pgmap v181: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 553 MiB/s wr, 1.21k op/s 2024-04-22T06:50:07.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:07.644+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:07.653+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:50:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:07.653+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:50:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:07.653+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:50:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:07.653+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:50:08.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:08 smithi193 ceph-mon[31815]: pgmap v182: 97 pgs: 97 active+clean; 41 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 802 MiB/s wr, 2.12k op/s 2024-04-22T06:50:08.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:08 smithi190 ceph-mon[31421]: pgmap v182: 97 pgs: 97 active+clean; 41 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 802 MiB/s wr, 2.12k op/s 2024-04-22T06:50:08.993 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:08 smithi079 ceph-mon[27946]: pgmap v182: 97 pgs: 97 active+clean; 41 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 802 MiB/s wr, 2.12k op/s 2024-04-22T06:50:10.795 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:10 smithi079 ceph-mon[27946]: pgmap v183: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 679 MiB/s wr, 1.95k op/s 2024-04-22T06:50:10.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:10 smithi193 ceph-mon[31815]: pgmap v183: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 679 MiB/s wr, 1.95k op/s 2024-04-22T06:50:10.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:10 smithi190 ceph-mon[31421]: pgmap v183: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 679 MiB/s wr, 1.95k op/s 2024-04-22T06:50:12.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:12 smithi193 ceph-mon[31815]: pgmap v184: 97 pgs: 97 active+clean; 42 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 739 MiB/s wr, 2.95k op/s 2024-04-22T06:50:12.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:12 smithi190 ceph-mon[31421]: pgmap v184: 97 pgs: 97 active+clean; 42 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 739 MiB/s wr, 2.95k op/s 2024-04-22T06:50:12.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:12 smithi079 ceph-mon[27946]: pgmap v184: 97 pgs: 97 active+clean; 42 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 739 MiB/s wr, 2.95k op/s 2024-04-22T06:50:14.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:14 smithi193 ceph-mon[31815]: pgmap v185: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 769 MiB/s wr, 4.13k op/s 2024-04-22T06:50:14.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:14 smithi190 ceph-mon[31421]: pgmap v185: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 769 MiB/s wr, 4.13k op/s 2024-04-22T06:50:14.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:14 smithi079 ceph-mon[27946]: pgmap v185: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 769 MiB/s wr, 4.13k op/s 2024-04-22T06:50:16.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:16 smithi193 ceph-mon[31815]: pgmap v186: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 631 MiB/s wr, 3.85k op/s 2024-04-22T06:50:16.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:16 smithi190 ceph-mon[31421]: pgmap v186: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 631 MiB/s wr, 3.85k op/s 2024-04-22T06:50:16.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:16 smithi079 ceph-mon[27946]: pgmap v186: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 631 MiB/s wr, 3.85k op/s 2024-04-22T06:50:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:18 smithi193 ceph-mon[31815]: pgmap v187: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 881 MiB/s wr, 5.61k op/s 2024-04-22T06:50:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:18 smithi190 ceph-mon[31421]: pgmap v187: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 881 MiB/s wr, 5.61k op/s 2024-04-22T06:50:18.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:18 smithi079 ceph-mon[27946]: pgmap v187: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 881 MiB/s wr, 5.61k op/s 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 -- 172.21.15.79:0/338509413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 msgr2=0x7f4378115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 --2- 172.21.15.79:0/338509413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f4378115a70 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4368009900 tx=0x7f436802cf50 comp rx=0 tx=0).stop 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 -- 172.21.15.79:0/338509413 shutdown_connections 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 --2- 172.21.15.79:0/338509413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f4378115a70 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 --2- 172.21.15.79:0/338509413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f437810e910 0x7f437810ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 --2- 172.21.15.79:0/338509413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f437810dca0 0x7f437810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 -- 172.21.15.79:0/338509413 >> 172.21.15.79:0/338509413 conn(0x7f437806ce90 msgr2=0x7f437806d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 -- 172.21.15.79:0/338509413 shutdown_connections 2024-04-22T06:50:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.104+0000 7f437d8b3640 1 -- 172.21.15.79:0/338509413 wait complete. 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.105+0000 7f437d8b3640 1 Processor -- start 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.105+0000 7f437d8b3640 1 -- start start 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.105+0000 7f437d8b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f437810dca0 0x7f4378112580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d8b3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f437810e910 0x7f4378112ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d8b3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f43781acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d8b3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4378118260 con 0x7f437810f2b0 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d8b3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43781180e0 con 0x7f437810dca0 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d8b3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43781183e0 con 0x7f437810e910 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f43781acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f43781acfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57498/0 (socket says 172.21.15.79:57498) 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 -- 172.21.15.79:0/3447616685 learned_addr learned my addr 172.21.15.79:0/3447616685 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 -- 172.21.15.79:0/3447616685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f437810e910 msgr2=0x7f4378112ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f437810e910 0x7f4378112ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 -- 172.21.15.79:0/3447616685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f437810dca0 msgr2=0x7f4378112580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.106+0000 7f437d0b2640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f437810dca0 0x7f4378112580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.107+0000 7f437d0b2640 1 -- 172.21.15.79:0/3447616685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43781ad710 con 0x7f437810f2b0 2024-04-22T06:50:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.107+0000 7f437d0b2640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f43781acfe0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4368009430 tx=0x7f436802d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.109+0000 7f4375ffb640 1 -- 172.21.15.79:0/3447616685 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436802dc90 con 0x7f437810f2b0 2024-04-22T06:50:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.109+0000 7f4375ffb640 1 -- 172.21.15.79:0/3447616685 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4368043070 con 0x7f437810f2b0 2024-04-22T06:50:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.109+0000 7f4375ffb640 1 -- 172.21.15.79:0/3447616685 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368003cd0 con 0x7f437810f2b0 2024-04-22T06:50:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.110+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43781ad9a0 con 0x7f437810f2b0 2024-04-22T06:50:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.110+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43781b4090 con 0x7f437810f2b0 2024-04-22T06:50:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.115+0000 7f43817dd740 1 Processor -- start 2024-04-22T06:50:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.116+0000 7f4375ffb640 1 -- 172.21.15.79:0/3447616685 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f43680052a0 con 0x7f437810f2b0 2024-04-22T06:50:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.117+0000 7f43817dd740 1 -- start start 2024-04-22T06:50:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.118+0000 7f4375ffb640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43640a3e60 0x7f43640a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f4375ffb640 1 -- 172.21.15.79:0/3447616685 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f43680f28e0 con 0x7f437810f2b0 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f43817dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c9946810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f43817dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c99471d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f43817dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c99526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f43817dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5561c98ab8d0 con 0x5561c9946d50 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f43817dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5561c9848e80 con 0x5561c99502d0 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f43817dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5561c98a7890 con 0x5561c9947ac0 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f4377fff640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43640a3e60 0x7f43640a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f4377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c99471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f4377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c99471d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57508/0 (socket says 172.21.15.79:57508) 2024-04-22T06:50:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.119+0000 7f4377fff640 1 -- 172.21.15.79:0/1168783625 learned_addr learned my addr 172.21.15.79:0/1168783625 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 -- 172.21.15.79:0/1168783625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 msgr2=0x5561c9946810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c9946810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 -- 172.21.15.79:0/1168783625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 msgr2=0x5561c99526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c99526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 -- 172.21.15.79:0/1168783625 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c986a320 con 0x5561c9946d50 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43640a3e60 0x7f43640a6320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4370011810 tx=0x7f437000c9e0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.120+0000 7f4377fff640 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c99471d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4370009040 tx=0x7f4370018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f2cd68f54c7698 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.121+0000 7f436effd640 1 -- 172.21.15.79:0/1168783625 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370004dd0 con 0x5561c9946d50 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.121+0000 7f436effd640 1 -- 172.21.15.79:0/1168783625 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4370023070 con 0x5561c9946d50 2024-04-22T06:50:20.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.121+0000 7f436effd640 1 -- 172.21.15.79:0/1168783625 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370006330 con 0x5561c9946d50 2024-04-22T06:50:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 -- 172.21.15.79:0/1168783625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 msgr2=0x5561c99471d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c99471d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4370009040 tx=0x7f4370018000 comp rx=0 tx=0).stop 2024-04-22T06:50:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 -- 172.21.15.79:0/1168783625 shutdown_connections 2024-04-22T06:50:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c99526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c99471d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 --2- 172.21.15.79:0/1168783625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c9946810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 -- 172.21.15.79:0/1168783625 >> 172.21.15.79:0/1168783625 conn(0x5561c990dc90 msgr2=0x5561c9941a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 -- 172.21.15.79:0/1168783625 shutdown_connections 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.122+0000 7f43817dd740 1 -- 172.21.15.79:0/1168783625 wait complete. 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 -- start start 2024-04-22T06:50:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 Processor -- start 2024-04-22T06:50:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c994fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c994d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c994d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5561c9953040 con 0x5561c9947ac0 2024-04-22T06:50:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5561c9833eb0 con 0x5561c99502d0 2024-04-22T06:50:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5561c99531c0 con 0x5561c9946d50 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.123+0000 7f43817dd740 10 client.0 _ll_register_callbacks cb 0x5561c9860b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.124+0000 7f4377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c994d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.124+0000 7f4377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c994d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57510/0 (socket says 172.21.15.79:57510) 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.124+0000 7f4377fff640 1 -- 172.21.15.79:0/3793266334 learned_addr learned my addr 172.21.15.79:0/3793266334 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.124+0000 7f437c8b1640 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c994fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.124+0000 7f4377fff640 1 -- 172.21.15.79:0/3793266334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 msgr2=0x5561c994fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4377fff640 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c994fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4377fff640 1 -- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 msgr2=0x5561c994d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4377fff640 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c994d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4377fff640 1 -- 172.21.15.79:0/3793266334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c95be210 con 0x5561c9947ac0 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4377fff640 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c994d020 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f43700090b0 tx=0x7f4370004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08bc45c6de48876 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:20.124 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4357fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f4357fff640 1 -- 172.21.15.79:0/3793266334 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437002d070 con 0x5561c9947ac0 2024-04-22T06:50:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.125+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4370023070 con 0x5561c9947ac0 2024-04-22T06:50:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.126+0000 7f4357fff640 1 -- 172.21.15.79:0/3793266334 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f437001e480 con 0x5561c9947ac0 2024-04-22T06:50:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.126+0000 7f4357fff640 1 -- 172.21.15.79:0/3793266334 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f4370037050 con 0x5561c9947ac0 2024-04-22T06:50:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.126+0000 7f43817dd740 10 client.16971 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.126+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5561c968a620 con 0x5561c9947ac0 2024-04-22T06:50:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.127+0000 7f4357fff640 1 -- 172.21.15.79:0/3793266334 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4370023070 con 0x5561c9947ac0 2024-04-22T06:50:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f4357fff640 1 -- 172.21.15.79:0/3793266334 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f4370029440 con 0x5561c9947ac0 2024-04-22T06:50:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f43817dd740 10 client.16971 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f43817dd740 10 client.16971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f43817dd740 20 client.16971 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f43817dd740 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99607d0 0x5561c9962c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f43817dd740 4 client.16971 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.128+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561c98bf880 con 0x5561c99607d0 2024-04-22T06:50:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.129+0000 7f437c8b1640 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99607d0 0x5561c9962c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.136+0000 7f437c8b1640 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99607d0 0x5561c9962c50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f03b07fda62ad054 server_cookie=bb8a4484770a695c in_seq=0 out_seq=0 2024-04-22T06:50:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.136+0000 7f4357fff640 10 client.16971 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.142+0000 7f4357fff640 1 -- 172.21.15.79:0/3793266334 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561c98bf880 con 0x5561c99607d0 2024-04-22T06:50:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.143+0000 7f4357fff640 10 client.16971 handle_command_reply: tid=0 2024-04-22T06:50:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.143+0000 7f43817dd740 1 client.16971 shutdown 2024-04-22T06:50:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.143+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 msgr2=0x5561c994d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.143+0000 7f43817dd740 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c994d020 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f43700090b0 tx=0x7f4370004f00 comp rx=0 tx=0).stop 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 shutdown_connections 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99607d0 0x5561c9962c50 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c994d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c994d020 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 --2- 172.21.15.79:0/3793266334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c994fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 >> 172.21.15.79:0/3793266334 conn(0x5561c990dc90 msgr2=0x5561c9943790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f4357fff640 0 client.16971 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 shutdown_connections 2024-04-22T06:50:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.144+0000 7f43817dd740 1 -- 172.21.15.79:0/3793266334 wait complete. 2024-04-22T06:50:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.145+0000 7f43817dd740 20 client.16971 trim_cache size 0 max 16384 2024-04-22T06:50:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.199+0000 7f43817dd740 1 Processor -- start 2024-04-22T06:50:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 -- start start 2024-04-22T06:50:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c995bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c994f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c994fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43781bf6c0 con 0x5561c9946d50 2024-04-22T06:50:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43781bcf90 con 0x5561c99502d0 2024-04-22T06:50:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.202+0000 7f43817dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43781bade0 con 0x5561c9947ac0 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.203+0000 7f4377fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c994f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.203+0000 7f4377fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c994f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:44102/0 (socket says 172.21.15.79:44102) 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.203+0000 7f4377fff640 1 -- 172.21.15.79:0/920840872 learned_addr learned my addr 172.21.15.79:0/920840872 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.204+0000 7f4377fff640 1 -- 172.21.15.79:0/920840872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 msgr2=0x5561c994fc90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.204+0000 7f4377fff640 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c994fc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.204+0000 7f4377fff640 1 -- 172.21.15.79:0/920840872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 msgr2=0x5561c995bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.204+0000 7f4377fff640 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c995bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.204+0000 7f4377fff640 1 -- 172.21.15.79:0/920840872 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c9946020 con 0x5561c9947ac0 2024-04-22T06:50:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.204+0000 7f4377fff640 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c994f750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4370011420 tx=0x7f43700115e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afd256e5f5479206 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.206+0000 7f436e7fc640 1 -- 172.21.15.79:0/920840872 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437002d030 con 0x5561c9947ac0 2024-04-22T06:50:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.207+0000 7f436e7fc640 1 -- 172.21.15.79:0/920840872 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4370023070 con 0x5561c9947ac0 2024-04-22T06:50:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.207+0000 7f436e7fc640 1 -- 172.21.15.79:0/920840872 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370029c70 con 0x5561c9947ac0 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 -- 172.21.15.79:0/920840872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 msgr2=0x5561c994f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c994f750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4370011420 tx=0x7f43700115e0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 -- 172.21.15.79:0/920840872 shutdown_connections 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c994fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9947ac0 0x5561c994f750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 --2- 172.21.15.79:0/920840872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9946d50 0x5561c995bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 -- 172.21.15.79:0/920840872 >> 172.21.15.79:0/920840872 conn(0x5561c990dc90 msgr2=0x5561c9948270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 -- 172.21.15.79:0/920840872 shutdown_connections 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.208+0000 7f43817dd740 1 -- 172.21.15.79:0/920840872 wait complete. 2024-04-22T06:50:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.209+0000 7f43817dd740 1 -- start start 2024-04-22T06:50:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.209+0000 7f43817dd740 1 Processor -- start 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c99584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c99589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c99492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43781bb450 con 0x5561c9947ac0 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43781bc8c0 con 0x5561c99502d0 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43781bbb20 con 0x5561c9946d50 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.210+0000 7f43817dd740 10 client.0 _ll_register_callbacks cb 0x7f437000cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.212+0000 7f4377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c99589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.213+0000 7f437c8b1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c99584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.213+0000 7f437c8b1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c99584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:44118/0 (socket says 172.21.15.79:44118) 2024-04-22T06:50:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.213+0000 7f437c8b1640 1 -- 172.21.15.79:0/2720227653 learned_addr learned my addr 172.21.15.79:0/2720227653 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.214+0000 7f437c8b1640 1 -- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 msgr2=0x5561c99492d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:50:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.214+0000 7f437c8b1640 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c99492d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.214+0000 7f437c8b1640 1 -- 172.21.15.79:0/2720227653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 msgr2=0x5561c99589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.214+0000 7f437c8b1640 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c99589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.214+0000 7f437c8b1640 1 -- 172.21.15.79:0/2720227653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c9949810 con 0x5561c9946d50 2024-04-22T06:50:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.216+0000 7f437c8b1640 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c99584a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f436000bb20 tx=0x7f43600143b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b9c97783cc6e0b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.217+0000 7f4357fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:50:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.217+0000 7f4357fff640 1 -- 172.21.15.79:0/2720227653 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360014d70 con 0x5561c9946d50 2024-04-22T06:50:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.219+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4370023070 con 0x5561c9946d50 2024-04-22T06:50:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.220+0000 7f4357fff640 1 -- 172.21.15.79:0/2720227653 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f43600170e0 con 0x5561c9946d50 2024-04-22T06:50:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.221+0000 7f4357fff640 1 -- 172.21.15.79:0/2720227653 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f4360029050 con 0x5561c9946d50 2024-04-22T06:50:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.221+0000 7f43817dd740 10 client.25876 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.221+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5561c99d7570 con 0x5561c9946d50 2024-04-22T06:50:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f4357fff640 1 -- 172.21.15.79:0/2720227653 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4370023070 con 0x5561c9946d50 2024-04-22T06:50:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f4357fff640 1 -- 172.21.15.79:0/2720227653 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f4360020a50 con 0x5561c9946d50 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f43817dd740 10 client.25876 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f43817dd740 10 client.25876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f43817dd740 20 client.25876 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f43817dd740 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99d7da0 0x5561c99da1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f43817dd740 4 client.25876 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.222+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561c984b3d0 con 0x5561c99d7da0 2024-04-22T06:50:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.223+0000 7f4377fff640 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99d7da0 0x5561c99da1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.225+0000 7f4377fff640 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99d7da0 0x5561c99da1c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d0f1af7212ad19 server_cookie=19881e0908b7a571 in_seq=0 out_seq=0 2024-04-22T06:50:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:20.225+0000 7f4357fff640 10 client.25876 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:20.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:20 smithi193 ceph-mon[31815]: pgmap v188: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 703 MiB/s wr, 5.13k op/s 2024-04-22T06:50:20.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:20 smithi190 ceph-mon[31421]: pgmap v188: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 703 MiB/s wr, 5.13k op/s 2024-04-22T06:50:20.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:20 smithi079 ceph-mon[27946]: pgmap v188: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 703 MiB/s wr, 5.13k op/s 2024-04-22T06:50:22.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:22.670+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:22.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:22.674+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-22T06:50:22.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:22.674+0000 7fcdb2ffd640 1 -- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 msgr2=0x555864e93510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:50:22.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:22.674+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:50:22.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:22.674+0000 7fcdb2ffd640 1 --2- 172.21.15.79:0/2325839592 >> [v2:172.21.15.193:6838/1588799940,v1:172.21.15.193:6839/1588799940] conn(0x555864e91090 0x555864e93510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:50:22.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:22 smithi193 ceph-mon[31815]: pgmap v189: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 781 MiB/s wr, 5.20k op/s 2024-04-22T06:50:22.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:22 smithi190 ceph-mon[31421]: pgmap v189: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 781 MiB/s wr, 5.20k op/s 2024-04-22T06:50:22.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:22 smithi079 ceph-mon[27946]: pgmap v189: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 781 MiB/s wr, 5.20k op/s 2024-04-22T06:50:24.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:24 smithi193 ceph-mon[31815]: pgmap v190: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 4.54k op/s 2024-04-22T06:50:24.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:24 smithi190 ceph-mon[31421]: pgmap v190: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 4.54k op/s 2024-04-22T06:50:24.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:24 smithi079 ceph-mon[27946]: pgmap v190: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 806 MiB/s wr, 4.54k op/s 2024-04-22T06:50:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:50:26.854 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 48, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 70, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 73, in _check_damage rdmg = self.fs.get_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1832, in get_damage result[rank] = self.get_damage(rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1835, in get_damage return self.rank_tell(['damage', 'ls'], rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1371, in rank_tell return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1358, in mds_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls' 2024-04-22T06:50:26.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:26 smithi193 ceph-mon[31815]: pgmap v191: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 627 MiB/s wr, 3.03k op/s 2024-04-22T06:50:26.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:26 smithi190 ceph-mon[31421]: pgmap v191: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 627 MiB/s wr, 3.03k op/s 2024-04-22T06:50:26.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:26 smithi079 ceph-mon[27946]: pgmap v191: 97 pgs: 97 active+clean; 45 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 627 MiB/s wr, 3.03k op/s 2024-04-22T06:50:28.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:28 smithi193 ceph-mon[31815]: pgmap v192: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 852 MiB/s wr, 3.49k op/s 2024-04-22T06:50:28.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:28 smithi190 ceph-mon[31421]: pgmap v192: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 852 MiB/s wr, 3.49k op/s 2024-04-22T06:50:28.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:28 smithi079 ceph-mon[27946]: pgmap v192: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 852 MiB/s wr, 3.49k op/s 2024-04-22T06:50:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.674+0000 7f4357fff640 1 -- 172.21.15.79:0/2720227653 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561c984b3d0 con 0x5561c99d7da0 2024-04-22T06:50:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.674+0000 7f4357fff640 10 client.25876 handle_command_reply: tid=0 2024-04-22T06:50:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.674+0000 7f43817dd740 1 client.25876 shutdown 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 msgr2=0x5561c99584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c99584a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f436000bb20 tx=0x7f43600143b0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 shutdown_connections 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5561c99d7da0 0x5561c99da1c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5561c99502d0 0x5561c99492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5561c9947ac0 0x5561c99589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 --2- 172.21.15.79:0/2720227653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561c9946d50 0x5561c99584a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 >> 172.21.15.79:0/2720227653 conn(0x5561c990dc90 msgr2=0x5561c994b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.679+0000 7f4357fff640 0 client.25876 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.680+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 shutdown_connections 2024-04-22T06:50:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.680+0000 7f43817dd740 1 -- 172.21.15.79:0/2720227653 wait complete. 2024-04-22T06:50:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.680+0000 7f43817dd740 20 client.25876 trim_cache size 0 max 16384 2024-04-22T06:50:29.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:50:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:50:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:50:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:50:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.706+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43640a3e60 msgr2=0x7f43640a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.706+0000 7f437d8b3640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43640a3e60 0x7f43640a6320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4370011810 tx=0x7f437000c9e0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.706+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 msgr2=0x7f43781acfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.706+0000 7f437d8b3640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f43781acfe0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4368009430 tx=0x7f436802d890 comp rx=0 tx=0).stop 2024-04-22T06:50:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.715+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 shutdown_connections 2024-04-22T06:50:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.715+0000 7f437d8b3640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f437810f2b0 0x7f43781acfe0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.715+0000 7f437d8b3640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f437810e910 0x7f4378112ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.715+0000 7f437d8b3640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43640a3e60 0x7f43640a6320 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.715+0000 7f437d8b3640 1 --2- 172.21.15.79:0/3447616685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f437810dca0 0x7f4378112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.715+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 >> 172.21.15.79:0/3447616685 conn(0x7f437806ce90 msgr2=0x7f437810b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.721+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 shutdown_connections 2024-04-22T06:50:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:29.721+0000 7f437d8b3640 1 -- 172.21.15.79:0/3447616685 wait complete. 2024-04-22T06:50:30.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:30 smithi193 ceph-mon[31815]: pgmap v193: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 675 MiB/s wr, 1.80k op/s 2024-04-22T06:50:30.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:30 smithi190 ceph-mon[31421]: pgmap v193: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 675 MiB/s wr, 1.80k op/s 2024-04-22T06:50:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:30 smithi079 ceph-mon[27946]: pgmap v193: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 675 MiB/s wr, 1.80k op/s 2024-04-22T06:50:31.118 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 33.7 2024-04-22T06:50:31.119 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:50:31.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.442+0000 7f0157585640 1 -- 172.21.15.79:0/2751346092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0150101b60 msgr2=0x7f015010dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:31.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.442+0000 7f0157585640 1 --2- 172.21.15.79:0/2751346092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0150101b60 0x7f015010dc90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0148009cd0 tx=0x7f014802c120 comp rx=0 tx=0).stop 2024-04-22T06:50:31.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.443+0000 7f0157585640 1 -- 172.21.15.79:0/2751346092 shutdown_connections 2024-04-22T06:50:31.441 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.443+0000 7f0157585640 1 --2- 172.21.15.79:0/2751346092 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0150101b60 0x7f015010dc90 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.443+0000 7f0157585640 1 --2- 172.21.15.79:0/2751346092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01501011a0 0x7f0150101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.443+0000 7f0157585640 1 --2- 172.21.15.79:0/2751346092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0150105e30 0x7f0150106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.443+0000 7f0157585640 1 -- 172.21.15.79:0/2751346092 >> 172.21.15.79:0/2751346092 conn(0x7f0150077e30 msgr2=0x7f0150078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:31.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.444+0000 7f0157585640 1 -- 172.21.15.79:0/2751346092 shutdown_connections 2024-04-22T06:50:31.443 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.445+0000 7f0157585640 1 -- 172.21.15.79:0/2751346092 wait complete. 2024-04-22T06:50:31.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.446+0000 7f0157585640 1 Processor -- start 2024-04-22T06:50:31.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.446+0000 7f0157585640 1 -- start start 2024-04-22T06:50:31.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0157585640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 0x7f015010db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:31.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0157585640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0150105e30 0x7f0150109ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:31.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0157585640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015010a2c0 0x7f015010a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:31.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0157585640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0150110140 con 0x7f01501011a0 2024-04-22T06:50:31.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0157585640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f015010ffc0 con 0x7f0150105e30 2024-04-22T06:50:31.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0157585640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01501102c0 con 0x7f015010a2c0 2024-04-22T06:50:31.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.447+0000 7f0155321640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 0x7f015010db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:31.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 0x7f015010db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51844/0 (socket says 172.21.15.79:51844) 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 -- 172.21.15.79:0/3730675224 learned_addr learned my addr 172.21.15.79:0/3730675224 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0154b20640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0150105e30 0x7f0150109ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 -- 172.21.15.79:0/3730675224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015010a2c0 msgr2=0x7f015010a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015010a2c0 0x7f015010a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 -- 172.21.15.79:0/3730675224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0150105e30 msgr2=0x7f0150109ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0150105e30 0x7f0150109ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 -- 172.21.15.79:0/3730675224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0150115b80 con 0x7f01501011a0 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.448+0000 7f0155321640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 0x7f015010db70 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f014000b3d0 tx=0x7f014000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:31.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.449+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140015020 con 0x7f01501011a0 2024-04-22T06:50:31.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.449+0000 7f0157585640 1 -- 172.21.15.79:0/3730675224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0150115e70 con 0x7f01501011a0 2024-04-22T06:50:31.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.449+0000 7f0157585640 1 -- 172.21.15.79:0/3730675224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0150116410 con 0x7f01501011a0 2024-04-22T06:50:31.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.450+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0140011ce0 con 0x7f01501011a0 2024-04-22T06:50:31.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.451+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140010e20 con 0x7f01501011a0 2024-04-22T06:50:31.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.452+0000 7f0157585640 1 -- 172.21.15.79:0/3730675224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0150102570 con 0x7f01501011a0 2024-04-22T06:50:31.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.453+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0140018020 con 0x7f01501011a0 2024-04-22T06:50:31.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.455+0000 7f01467fc640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f013c0a3de0 0x7f013c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:31.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.457+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f0140099390 con 0x7f01501011a0 2024-04-22T06:50:31.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.457+0000 7f0154b20640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f013c0a3de0 0x7f013c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:31.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.458+0000 7f0154b20640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f013c0a3de0 0x7f013c0a62a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f015010b3c0 tx=0x7f014c008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:31.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.476+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f014004cff0 con 0x7f01501011a0 2024-04-22T06:50:31.980 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:50:31.980 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":54,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:49:56.426792+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25817,"mds_2":25810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25810":{"gid":25810,"name":"i","rank":2,"incarnation":49,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/1413649941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1413649941},{"type":"v1","addr":"172.21.15.193:6839","nonce":1413649941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25817":{"gid":25817,"name":"h","rank":1,"incarnation":49,"state":"up:active","state_seq":23,"addr":"172.21.15.190:6837/1169559599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1169559599},{"type":"v1","addr":"172.21.15.190:6837","nonce":1169559599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25855":{"gid":25855,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1167513219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1167513219},{"type":"v1","addr":"172.21.15.190:6835","nonce":1167513219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25865":{"gid":25865,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/99277862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":99277862},{"type":"v1","addr":"172.21.15.193:6835","nonce":99277862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,25817,25810]},"id":1}]} 2024-04-22T06:50:31.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.977+0000 7f0157585640 1 -- 172.21.15.79:0/3730675224 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0150068de0 con 0x7f01501011a0 2024-04-22T06:50:31.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.979+0000 7f01467fc640 1 -- 172.21.15.79:0/3730675224 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11878 (secure 0 0 0) 0x7f01400d3020 con 0x7f01501011a0 2024-04-22T06:50:31.981 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 54 2024-04-22T06:50:31.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.990+0000 7f013bfff640 1 -- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f013c0a3de0 msgr2=0x7f013c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:31.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.990+0000 7f013bfff640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f013c0a3de0 0x7f013c0a62a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f015010b3c0 tx=0x7f014c008040 comp rx=0 tx=0).stop 2024-04-22T06:50:31.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.991+0000 7f013bfff640 1 -- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 msgr2=0x7f015010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:31.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.991+0000 7f013bfff640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 0x7f015010db70 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f014000b3d0 tx=0x7f014000b890 comp rx=0 tx=0).stop 2024-04-22T06:50:31.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.993+0000 7f013bfff640 1 -- 172.21.15.79:0/3730675224 shutdown_connections 2024-04-22T06:50:31.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.993+0000 7f013bfff640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f013c0a3de0 0x7f013c0a62a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.993+0000 7f013bfff640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0150105e30 0x7f0150109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.994+0000 7f013bfff640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f015010a2c0 0x7f015010a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.994+0000 7f013bfff640 1 --2- 172.21.15.79:0/3730675224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01501011a0 0x7f015010db70 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:31.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.994+0000 7f013bfff640 1 -- 172.21.15.79:0/3730675224 >> 172.21.15.79:0/3730675224 conn(0x7f0150077e30 msgr2=0x7f01500fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:31.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.997+0000 7f013bfff640 1 -- 172.21.15.79:0/3730675224 shutdown_connections 2024-04-22T06:50:31.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:31.997+0000 7f013bfff640 1 -- 172.21.15.79:0/3730675224 wait complete. 2024-04-22T06:50:32.041 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 93.7 --await' 2024-04-22T06:50:32.041 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 93.7 --await 2024-04-22T06:50:32.326 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 -- 172.21.15.79:0/1588692572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c109480 msgr2=0x7f924c111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 --2- 172.21.15.79:0/1588692572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c109480 0x7f924c111050 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9244009ca0 tx=0x7f924402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 -- 172.21.15.79:0/1588692572 shutdown_connections 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 --2- 172.21.15.79:0/1588692572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c104d60 0x7f924c108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 --2- 172.21.15.79:0/1588692572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c109480 0x7f924c111050 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 --2- 172.21.15.79:0/1588692572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c1043b0 0x7f924c104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 -- 172.21.15.79:0/1588692572 >> 172.21.15.79:0/1588692572 conn(0x7f924c0ffc10 msgr2=0x7f924c102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.327+0000 7f925412b640 1 -- 172.21.15.79:0/1588692572 shutdown_connections 2024-04-22T06:50:32.327 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.328+0000 7f925412b640 1 -- 172.21.15.79:0/1588692572 wait complete. 2024-04-22T06:50:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 Processor -- start 2024-04-22T06:50:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 -- start start 2024-04-22T06:50:32.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 0x7f924c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c109480 0x7f924c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c19a820 0x7f924c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f924c114440 con 0x7f924c1043b0 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1142c0 con 0x7f924c109480 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f925412b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1145c0 con 0x7f924c19a820 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 0x7f924c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 0x7f924c1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51848/0 (socket says 172.21.15.79:51848) 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f9251ec7640 1 -- 172.21.15.79:0/335955591 learned_addr learned my addr 172.21.15.79:0/335955591 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.329+0000 7f92526c8640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c19a820 0x7f924c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9251ec7640 1 -- 172.21.15.79:0/335955591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c19a820 msgr2=0x7f924c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9251ec7640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c19a820 0x7f924c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9251ec7640 1 -- 172.21.15.79:0/335955591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c109480 msgr2=0x7f924c1a0bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:50:32.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9251ec7640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c109480 0x7f924c1a0bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9251ec7640 1 -- 172.21.15.79:0/335955591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c19b3b0 con 0x7f924c1043b0 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f92526c8640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c19a820 0x7f924c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9251ec7640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 0x7f924c1a06b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f924800b4a0 tx=0x7f924800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f9242ffd640 1 -- 172.21.15.79:0/335955591 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248014070 con 0x7f924c1043b0 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.330+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c19b6a0 con 0x7f924c1043b0 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.331+0000 7f9242ffd640 1 -- 172.21.15.79:0/335955591 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9248010ac0 con 0x7f924c1043b0 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.331+0000 7f9242ffd640 1 -- 172.21.15.79:0/335955591 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924801e060 con 0x7f924c1043b0 2024-04-22T06:50:32.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.331+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c1a7880 con 0x7f924c1043b0 2024-04-22T06:50:32.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.333+0000 7f9242ffd640 1 -- 172.21.15.79:0/335955591 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9248017070 con 0x7f924c1043b0 2024-04-22T06:50:32.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.335+0000 7f9256df8740 1 Processor -- start 2024-04-22T06:50:32.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.335+0000 7f9256df8740 1 -- start start 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9256df8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b028e9c0 0x55a2b028eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9256df8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0297980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9256df8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b0297ec0 0x55a2b029a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9256df8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a2b01ef0b0 con 0x55a2b0297ec0 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9256df8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2b01a0680 con 0x55a2b028f2e0 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92526c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b028e9c0 0x55a2b028eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9242ffd640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92380a3de0 0x7f92380a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92526c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b028e9c0 0x55a2b028eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:34762/0 (socket says 172.21.15.79:34762) 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92526c8640 1 -- 172.21.15.79:0/206618454 learned_addr learned my addr 172.21.15.79:0/206618454 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92516c6640 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b0297ec0 0x55a2b029a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f9256df8740 1 -- 172.21.15.79:0/206618454 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a2b01acc60 con 0x55a2b028e9c0 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92526c8640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92380a3de0 0x7f92380a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92516c6640 1 -- 172.21.15.79:0/206618454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b028e9c0 msgr2=0x55a2b028eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.336+0000 7f92516c6640 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b028e9c0 0x55a2b028eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f92516c6640 1 -- 172.21.15.79:0/206618454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 msgr2=0x55a2b0297980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f92516c6640 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0297980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f92516c6640 1 -- 172.21.15.79:0/206618454 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2affd1c90 con 0x55a2b0297ec0 2024-04-22T06:50:32.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f9242ffd640 1 -- 172.21.15.79:0/335955591 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f92480cfdc0 con 0x7f924c1043b0 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f92516c6640 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b0297ec0 0x55a2b029a2b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f924c19bb70 tx=0x7f923c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f923889bb33122c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f92526c8640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92380a3de0 0x7f92380a62a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9244009ca0 tx=0x7f924402c0f0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.337+0000 7f9237fff640 1 -- 172.21.15.79:0/206618454 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c022070 con 0x55a2b0297ec0 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.338+0000 7f9237fff640 1 -- 172.21.15.79:0/206618454 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f923c0035f0 con 0x55a2b0297ec0 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.338+0000 7f9237fff640 1 -- 172.21.15.79:0/206618454 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c022070 con 0x55a2b0297ec0 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 -- 172.21.15.79:0/206618454 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b0297ec0 msgr2=0x55a2b029a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b0297ec0 0x55a2b029a2b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f924c19bb70 tx=0x7f923c009a10 comp rx=0 tx=0).stop 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 -- 172.21.15.79:0/206618454 shutdown_connections 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0297980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b0297ec0 0x55a2b029a2b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 --2- 172.21.15.79:0/206618454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b028e9c0 0x55a2b028eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 -- 172.21.15.79:0/206618454 >> 172.21.15.79:0/206618454 conn(0x55a2b02890f0 msgr2=0x55a2b02894d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:32.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 -- 172.21.15.79:0/206618454 shutdown_connections 2024-04-22T06:50:32.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.339+0000 7f9256df8740 1 -- 172.21.15.79:0/206618454 wait complete. 2024-04-22T06:50:32.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.340+0000 7f9256df8740 1 -- start start 2024-04-22T06:50:32.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.340+0000 7f9256df8740 1 Processor -- start 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.340+0000 7f9256df8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 0x55a2b0295030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.340+0000 7f9256df8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028f2e0 0x55a2b0295570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9256df8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b0295ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f92516c6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028f2e0 0x55a2b0295570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 0x55a2b0295030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 0x55a2b0295030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.79:36516/0 (socket says 172.21.15.79:36516) 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f92516c6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028f2e0 0x55a2b0295570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51862/0 (socket says 172.21.15.79:51862) 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 -- 172.21.15.79:0/2040230657 learned_addr learned my addr 172.21.15.79:0/2040230657 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a2b01f2400 con 0x55a2b028f2e0 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2b01ae840 con 0x55a2b028e9c0 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f92526c8640 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b0295ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a2b01f7720 con 0x55a2b0297ec0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9256df8740 10 client.0 _ll_register_callbacks cb 0x55a2b01ffea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 -- 172.21.15.79:0/2040230657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 msgr2=0x55a2b0295ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b0295ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 -- 172.21.15.79:0/2040230657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028f2e0 msgr2=0x55a2b0295570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028f2e0 0x55a2b0295570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.341+0000 7f9251ec7640 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2b0032420 con 0x55a2b028e9c0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f9251ec7640 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 0x55a2b0295030 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f924809bb20 tx=0x7f924809c290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1846040d58506ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f92357fa640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f92357fa640 1 -- 172.21.15.79:0/2040230657 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480a4800 con 0x55a2b028e9c0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f92357fa640 1 -- 172.21.15.79:0/2040230657 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f9248017360 con 0x55a2b028e9c0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f923c0035f0 con 0x55a2b028e9c0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f92357fa640 1 -- 172.21.15.79:0/2040230657 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f9248098200 con 0x55a2b028e9c0 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f9256df8740 10 client.25874 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:32.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.342+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55a2afec5580 con 0x55a2b028e9c0 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f92357fa640 1 -- 172.21.15.79:0/2040230657 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92480cd4b0 con 0x55a2b028e9c0 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f92357fa640 1 -- 172.21.15.79:0/2040230657 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f924809afe0 con 0x55a2b028e9c0 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f9256df8740 10 client.25874 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f9256df8740 10 client.25874 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f9256df8740 20 client.25874 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f9256df8740 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b02a7c20 0x55a2b02aa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f9256df8740 4 client.25874 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f923c0035f0 con 0x55a2b02a7c20 2024-04-22T06:50:32.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.343+0000 7f92516c6640 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b02a7c20 0x55a2b02aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.344+0000 7f92357fa640 10 client.25874 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.344+0000 7f92516c6640 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b02a7c20 0x55a2b02aa030 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3a86a895d3691d8 server_cookie=74a511a62fc10d86 in_seq=0 out_seq=0 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f92357fa640 1 -- 172.21.15.79:0/2040230657 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f923c0035f0 con 0x55a2b02a7c20 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f92357fa640 10 client.25874 handle_command_reply: tid=0 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 client.25874 shutdown 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 msgr2=0x55a2b0295030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 0x55a2b0295030 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f924809bb20 tx=0x7f924809c290 comp rx=0 tx=0).stop 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 shutdown_connections 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b02a7c20 0x55a2b02aa030 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b0295ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f92357fa640 0 client.25874 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:32.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028f2e0 0x55a2b0295570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 --2- 172.21.15.79:0/2040230657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028e9c0 0x55a2b0295030 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 >> 172.21.15.79:0/2040230657 conn(0x55a2b02890f0 msgr2=0x55a2b028ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:32.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 shutdown_connections 2024-04-22T06:50:32.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 1 -- 172.21.15.79:0/2040230657 wait complete. 2024-04-22T06:50:32.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.347+0000 7f9256df8740 20 client.25874 trim_cache size 0 max 16384 2024-04-22T06:50:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 Processor -- start 2024-04-22T06:50:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 -- start start 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b02a3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0293f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b02958d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1ac2f0 con 0x55a2b028e9c0 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1aa140 con 0x55a2b028f2e0 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9256df8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1abc20 con 0x55a2b0297ec0 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b02a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b02a3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51872/0 (socket says 172.21.15.79:51872) 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f9251ec7640 1 -- 172.21.15.79:0/813071896 learned_addr learned my addr 172.21.15.79:0/813071896 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f92516c6640 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0293f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.376+0000 7f92526c8640 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b02958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f9251ec7640 1 -- 172.21.15.79:0/813071896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 msgr2=0x55a2b02958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f9251ec7640 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b02958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f9251ec7640 1 -- 172.21.15.79:0/813071896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 msgr2=0x55a2b0293f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f9251ec7640 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0293f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f9251ec7640 1 -- 172.21.15.79:0/813071896 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2b0289cc0 con 0x55a2b028e9c0 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f92526c8640 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b02958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f9251ec7640 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b02a3870 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9248097870 tx=0x7f92480047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c7a27e3ddd13ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f92377fe640 1 -- 172.21.15.79:0/813071896 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480983a0 con 0x55a2b028e9c0 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f92377fe640 1 -- 172.21.15.79:0/813071896 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92480a4800 con 0x55a2b028e9c0 2024-04-22T06:50:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.377+0000 7f92377fe640 1 -- 172.21.15.79:0/813071896 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480cd680 con 0x55a2b028e9c0 2024-04-22T06:50:32.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 -- 172.21.15.79:0/813071896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 msgr2=0x55a2b02a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b02a3870 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9248097870 tx=0x7f92480047d0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 -- 172.21.15.79:0/813071896 shutdown_connections 2024-04-22T06:50:32.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b0293f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b02958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 --2- 172.21.15.79:0/813071896 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b02a3870 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 -- 172.21.15.79:0/813071896 >> 172.21.15.79:0/813071896 conn(0x55a2b02890f0 msgr2=0x55a2b02881d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 -- 172.21.15.79:0/813071896 shutdown_connections 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 -- 172.21.15.79:0/813071896 wait complete. 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 -- start start 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.378+0000 7f9256df8740 1 Processor -- start 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b029b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b029bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b029c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1aae80 con 0x55a2b028e9c0 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1aa7b0 con 0x55a2b028f2e0 2024-04-22T06:50:32.381 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1ab550 con 0x55a2b0297ec0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9256df8740 10 client.0 _ll_register_callbacks cb 0x7f924809b7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b029b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b029b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51882/0 (socket says 172.21.15.79:51882) 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 -- 172.21.15.79:0/2982325060 learned_addr learned my addr 172.21.15.79:0/2982325060 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 -- 172.21.15.79:0/2982325060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 msgr2=0x55a2b029c430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b029c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 -- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 msgr2=0x55a2b029bef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b029bef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.379+0000 7f9251ec7640 1 -- 172.21.15.79:0/2982325060 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2b030d3c0 con 0x55a2b028e9c0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f9251ec7640 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b029b9b0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f92480cf040 tx=0x7f924809c290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5bb297dee93cee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f92357fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f92357fa640 1 -- 172.21.15.79:0/2982325060 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248017bb0 con 0x55a2b028e9c0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f92357fa640 1 -- 172.21.15.79:0/2982325060 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f9248017d50 con 0x55a2b028e9c0 2024-04-22T06:50:32.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92480a4800 con 0x55a2b028e9c0 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f92357fa640 1 -- 172.21.15.79:0/2982325060 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f92480ab050 con 0x55a2b028e9c0 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f9256df8740 10 client.16995 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55a2b030e660 con 0x55a2b028e9c0 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f92357fa640 1 -- 172.21.15.79:0/2982325060 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92480a4800 con 0x55a2b028e9c0 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f92357fa640 1 -- 172.21.15.79:0/2982325060 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f924809ac50 con 0x55a2b028e9c0 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.380+0000 7f9256df8740 10 client.16995 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.381+0000 7f9256df8740 10 client.16995 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.381+0000 7f9256df8740 20 client.16995 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.381+0000 7f9256df8740 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b030ea60 0x55a2b0310ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.381+0000 7f9256df8740 4 client.16995 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 93.7, "await": true}] 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.381+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 93.7, "await": true}) -- 0x55a2b0200b10 con 0x55a2b030ea60 2024-04-22T06:50:32.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.382+0000 7f92516c6640 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b030ea60 0x55a2b0310ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:32.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.385+0000 7f92357fa640 10 client.16995 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:32.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.385+0000 7f92516c6640 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b030ea60 0x55a2b0310ec0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49ffc54bc5e5fcd3 server_cookie=5fbbb30038dc6871 in_seq=0 out_seq=0 2024-04-22T06:50:32.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f92357fa640 1 -- 172.21.15.79:0/2982325060 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55a2b0200b10 con 0x55a2b030ea60 2024-04-22T06:50:32.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f92357fa640 10 client.16995 handle_command_reply: tid=0 2024-04-22T06:50:32.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 client.16995 shutdown 2024-04-22T06:50:32.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 msgr2=0x55a2b029b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b029b9b0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f92480cf040 tx=0x7f924809c290 comp rx=0 tx=0).stop 2024-04-22T06:50:32.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 shutdown_connections 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a2b030ea60 0x55a2b0310ec0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a2b028f2e0 0x55a2b029bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2b0297ec0 0x55a2b029c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 --2- 172.21.15.79:0/2982325060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a2b028e9c0 0x55a2b029b9b0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f92357fa640 0 client.16995 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 >> 172.21.15.79:0/2982325060 conn(0x55a2b02890f0 msgr2=0x55a2b028a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 shutdown_connections 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 1 -- 172.21.15.79:0/2982325060 wait complete. 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.655+0000 7f9256df8740 20 client.16995 trim_cache size 0 max 16384 2024-04-22T06:50:32.655 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 54, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 20, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "6e8daeb4": { 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 20, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 93.7, 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:50:32.656 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:50:32.657 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92380a3de0 msgr2=0x7f92380a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92380a3de0 0x7f92380a62a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9244009ca0 tx=0x7f924402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 msgr2=0x7f924c1a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 0x7f924c1a06b0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f924800b4a0 tx=0x7f924800b960 comp rx=0 tx=0).stop 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 shutdown_connections 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f92380a3de0 0x7f92380a62a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c109480 0x7f924c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f924c19a820 0x7f924c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 --2- 172.21.15.79:0/335955591 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f924c1043b0 0x7f924c1a06b0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 >> 172.21.15.79:0/335955591 conn(0x7f924c0ffc10 msgr2=0x7f924c100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:32.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 shutdown_connections 2024-04-22T06:50:32.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:50:32.656+0000 7f925412b640 1 -- 172.21.15.79:0/335955591 wait complete. 2024-04-22T06:50:32.667 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 6e8daeb4, quiesce took 1.5 seconds. Will release after: 33.7 2024-04-22T06:50:32.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:32 smithi193 ceph-mon[31815]: pgmap v194: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 748 MiB/s wr, 1.68k op/s 2024-04-22T06:50:32.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:32 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3730675224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:50:32.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:32 smithi190 ceph-mon[31421]: pgmap v194: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 748 MiB/s wr, 1.68k op/s 2024-04-22T06:50:32.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:32 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3730675224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:50:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:32 smithi079 ceph-mon[27946]: pgmap v194: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 748 MiB/s wr, 1.68k op/s 2024-04-22T06:50:32.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:32 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3730675224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:50:34.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:34 smithi193 ceph-mon[31815]: pgmap v195: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 734 MiB/s wr, 1.57k op/s 2024-04-22T06:50:34.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:34 smithi190 ceph-mon[31421]: pgmap v195: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 734 MiB/s wr, 1.57k op/s 2024-04-22T06:50:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:34 smithi079 ceph-mon[27946]: pgmap v195: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 734 MiB/s wr, 1.57k op/s 2024-04-22T06:50:36.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:36 smithi193 ceph-mon[31815]: pgmap v196: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 586 MiB/s wr, 1.15k op/s 2024-04-22T06:50:36.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:36 smithi190 ceph-mon[31421]: pgmap v196: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 586 MiB/s wr, 1.15k op/s 2024-04-22T06:50:36.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:36 smithi079 ceph-mon[27946]: pgmap v196: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 586 MiB/s wr, 1.15k op/s 2024-04-22T06:50:38.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:38 smithi193 ceph-mon[31815]: pgmap v197: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 647 MiB/s wr, 1.20k op/s 2024-04-22T06:50:38.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:38 smithi190 ceph-mon[31421]: pgmap v197: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 647 MiB/s wr, 1.20k op/s 2024-04-22T06:50:38.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:38 smithi079 ceph-mon[27946]: pgmap v197: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 647 MiB/s wr, 1.20k op/s 2024-04-22T06:50:40.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:40 smithi193 ceph-mon[31815]: pgmap v198: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 422 MiB/s wr, 736 op/s 2024-04-22T06:50:40.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:40 smithi190 ceph-mon[31421]: pgmap v198: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 422 MiB/s wr, 736 op/s 2024-04-22T06:50:40.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:40 smithi079 ceph-mon[27946]: pgmap v198: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 422 MiB/s wr, 736 op/s 2024-04-22T06:50:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 -- 172.21.15.79:0/1244727625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098101aa0 msgr2=0x7f8098105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 --2- 172.21.15.79:0/1244727625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098101aa0 0x7f8098105a90 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8088009900 tx=0x7f808802bc30 comp rx=0 tx=0).stop 2024-04-22T06:50:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 -- 172.21.15.79:0/1244727625 shutdown_connections 2024-04-22T06:50:42.865 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 --2- 172.21.15.79:0/1244727625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8098106170 0x7f809810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 --2- 172.21.15.79:0/1244727625 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098101aa0 0x7f8098105a90 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 --2- 172.21.15.79:0/1244727625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80981010f0 0x7f80981014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 -- 172.21.15.79:0/1244727625 >> 172.21.15.79:0/1244727625 conn(0x7f8098077fd0 msgr2=0x7f80980783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 -- 172.21.15.79:0/1244727625 shutdown_connections 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 -- 172.21.15.79:0/1244727625 wait complete. 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 Processor -- start 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.865+0000 7f809e787640 1 -- start start 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80981010f0 0x7f8098197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8098106170 0x7f8098197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 0x7f80981a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f8098111090 con 0x7f8098197da0 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8098110f10 con 0x7f80981010f0 2024-04-22T06:50:42.866 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8098111210 con 0x7f8098106170 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 0x7f80981a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 0x7f80981a69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46586/0 (socket says 172.21.15.79:46586) 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 -- 172.21.15.79:0/2910177261 learned_addr learned my addr 172.21.15.79:0/2910177261 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f80977fe640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8098106170 0x7f8098197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f8097fff640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80981010f0 0x7f8098197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 -- 172.21.15.79:0/2910177261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8098106170 msgr2=0x7f8098197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8098106170 0x7f8098197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 -- 172.21.15.79:0/2910177261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80981010f0 msgr2=0x7f8098197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80981010f0 0x7f8098197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 -- 172.21.15.79:0/2910177261 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80981a6f60 con 0x7f8098197da0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809cd24640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 0x7f80981a69a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f809000eb40 tx=0x7f809000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f80957fa640 1 -- 172.21.15.79:0/2910177261 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8090004b20 con 0x7f8098197da0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f80957fa640 1 -- 172.21.15.79:0/2910177261 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8090004cc0 con 0x7f8098197da0 2024-04-22T06:50:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80981a7250 con 0x7f8098197da0 2024-04-22T06:50:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f80957fa640 1 -- 172.21.15.79:0/2910177261 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8090010690 con 0x7f8098197da0 2024-04-22T06:50:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.866+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80981a7770 con 0x7f8098197da0 2024-04-22T06:50:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80957fa640 1 -- 172.21.15.79:0/2910177261 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8090005020 con 0x7f8098197da0 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 Processor -- start 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 -- start start 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd2428d0 0x5593fd241620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd241fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd24d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5593fd1a68d0 con 0x5593fd24b0e0 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5593fd143e80 con 0x5593fd241b60 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80a1370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5593fd1a2890 con 0x5593fd2428d0 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd24d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd24d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46596/0 (socket says 172.21.15.79:46596) 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80977fe640 1 -- 172.21.15.79:0/1588021766 learned_addr learned my addr 172.21.15.79:0/1588021766 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80957fa640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80840a3e40 0x7f80840a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f8097fff640 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd241fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.869+0000 7f80977fe640 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd2428d0 0x5593fd241620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f809cd24640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80840a3e40 0x7f80840a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80977fe640 1 -- 172.21.15.79:0/1588021766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd2428d0 msgr2=0x5593fd241620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80977fe640 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd2428d0 0x5593fd241620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80977fe640 1 -- 172.21.15.79:0/1588021766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 msgr2=0x5593fd241fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80977fe640 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd241fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80977fe640 1 -- 172.21.15.79:0/1588021766 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593fd165320 con 0x5593fd24b0e0 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80957fa640 1 -- 172.21.15.79:0/2910177261 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f80900d0f30 con 0x7f8098197da0 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f80977fe640 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd24d4d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8088009900 tx=0x7f8088005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dac55e463f7f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.876 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f808e7fc640 1 -- 172.21.15.79:0/1588021766 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088047020 con 0x5593fd24b0e0 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f809cd24640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80840a3e40 0x7f80840a6300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f809009cb80 tx=0x7f8090067740 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f808e7fc640 1 -- 172.21.15.79:0/1588021766 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f808802cc90 con 0x5593fd24b0e0 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.870+0000 7f808e7fc640 1 -- 172.21.15.79:0/1588021766 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80880046c0 con 0x5593fd24b0e0 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 -- 172.21.15.79:0/1588021766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 msgr2=0x5593fd24d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd24d4d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8088009900 tx=0x7f8088005e80 comp rx=0 tx=0).stop 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 -- 172.21.15.79:0/1588021766 shutdown_connections 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd241fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd24d4d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 --2- 172.21.15.79:0/1588021766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd2428d0 0x5593fd241620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 -- 172.21.15.79:0/1588021766 >> 172.21.15.79:0/1588021766 conn(0x5593fd208c90 msgr2=0x5593fd23c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 -- 172.21.15.79:0/1588021766 shutdown_connections 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 -- 172.21.15.79:0/1588021766 wait complete. 2024-04-22T06:50:42.877 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 -- start start 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.871+0000 7f80a1370740 1 Processor -- start 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd24aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd247e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 0x5593fd248450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5593fd24ddb0 con 0x5593fd2428d0 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5593fd12eeb0 con 0x5593fd241b60 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5593fd24df30 con 0x5593fd24b0e0 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 10 client.0 _ll_register_callbacks cb 0x5593fd15bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd247e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd247e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46604/0 (socket says 172.21.15.79:46604) 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 -- 172.21.15.79:0/1802806341 learned_addr learned my addr 172.21.15.79:0/1802806341 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:42.878 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80977fe640 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd24aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 -- 172.21.15.79:0/1802806341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 msgr2=0x5593fd248450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 0x5593fd248450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 -- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 msgr2=0x5593fd24aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd24aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 -- 172.21.15.79:0/1802806341 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593fceb9210 con 0x5593fd2428d0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f8097fff640 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd247e40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f80800126d0 tx=0x7f8080012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4944d9156934b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80777fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80777fe640 1 -- 172.21.15.79:0/1802806341 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080004050 con 0x5593fd2428d0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80777fe640 1 -- 172.21.15.79:0/1802806341 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f80800041f0 con 0x5593fd2428d0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f808802cc90 con 0x5593fd2428d0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80777fe640 1 -- 172.21.15.79:0/1802806341 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f8080025050 con 0x5593fd2428d0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 10 client.17007 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.872+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5593fcf85620 con 0x5593fd2428d0 2024-04-22T06:50:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80777fe640 1 -- 172.21.15.79:0/1802806341 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f808802cc90 con 0x5593fd2428d0 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80777fe640 1 -- 172.21.15.79:0/1802806341 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f8080003970 con 0x5593fd2428d0 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80a1370740 10 client.17007 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80a1370740 10 client.17007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80a1370740 20 client.17007 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80a1370740 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd25b110 0x5593fd25d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80a1370740 4 client.17007 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593fd1ba880 con 0x5593fd25b110 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80977fe640 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd25b110 0x5593fd25d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80777fe640 10 client.17007 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.873+0000 7f80977fe640 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd25b110 0x5593fd25d520 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65fccdd158672245 server_cookie=6ac316f5b125f938 in_seq=0 out_seq=0 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80777fe640 1 -- 172.21.15.79:0/1802806341 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593fd1ba880 con 0x5593fd25b110 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80777fe640 10 client.17007 handle_command_reply: tid=0 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 client.17007 shutdown 2024-04-22T06:50:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 msgr2=0x5593fd247e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd247e40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f80800126d0 tx=0x7f8080012b90 comp rx=0 tx=0).stop 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 shutdown_connections 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd25b110 0x5593fd25d520 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 0x5593fd248450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd247e40 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 --2- 172.21.15.79:0/1802806341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd24aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80777fe640 0 client.17007 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 >> 172.21.15.79:0/1802806341 conn(0x5593fd208c90 msgr2=0x5593fd23e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 shutdown_connections 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.874+0000 7f80a1370740 1 -- 172.21.15.79:0/1802806341 wait complete. 2024-04-22T06:50:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.875+0000 7f80a1370740 20 client.17007 trim_cache size 0 max 16384 2024-04-22T06:50:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.905+0000 7f80a1370740 1 Processor -- start 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 -- start start 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd256cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd2435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 0x5593fd24a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f80981b2bc0 con 0x5593fd2428d0 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f80981b0490 con 0x5593fd241b60 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80a1370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f80981ae300 con 0x5593fd24b0e0 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd2435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd2435d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46606/0 (socket says 172.21.15.79:46606) 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 -- 172.21.15.79:0/2890740720 learned_addr learned my addr 172.21.15.79:0/2890740720 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f80977fe640 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd256cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 -- 172.21.15.79:0/2890740720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 msgr2=0x5593fd24a570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 0x5593fd24a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 -- 172.21.15.79:0/2890740720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 msgr2=0x5593fd256cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd256cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.906+0000 7f8097fff640 1 -- 172.21.15.79:0/2890740720 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593fd240e30 con 0x5593fd2428d0 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.907+0000 7f8097fff640 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd2435d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8080003d90 tx=0x7f8080010640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26df58a58719dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.907+0000 7f808dffb640 1 -- 172.21.15.79:0/2890740720 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808002e070 con 0x5593fd2428d0 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.907+0000 7f808dffb640 1 -- 172.21.15.79:0/2890740720 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8080003aa0 con 0x5593fd2428d0 2024-04-22T06:50:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.907+0000 7f808dffb640 1 -- 172.21.15.79:0/2890740720 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800050c0 con 0x5593fd2428d0 2024-04-22T06:50:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 -- 172.21.15.79:0/2890740720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 msgr2=0x5593fd2435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd2435d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8080003d90 tx=0x7f8080010640 comp rx=0 tx=0).stop 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 -- 172.21.15.79:0/2890740720 shutdown_connections 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd24b0e0 0x5593fd24a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd2428d0 0x5593fd2435d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 --2- 172.21.15.79:0/2890740720 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd241b60 0x5593fd256cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 -- 172.21.15.79:0/2890740720 >> 172.21.15.79:0/2890740720 conn(0x5593fd208c90 msgr2=0x5593fd242ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 -- 172.21.15.79:0/2890740720 shutdown_connections 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 -- 172.21.15.79:0/2890740720 wait complete. 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.908+0000 7f80a1370740 1 -- start start 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 Processor -- start 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd241b60 0x5593fd24ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd2428d0 0x5593fd24f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd2c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f80981ae950 con 0x5593fd24b0e0 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f80981afdc0 con 0x5593fd2428d0 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f80981af020 con 0x5593fd241b60 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80a1370740 10 client.0 _ll_register_callbacks cb 0x7f80800097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd2c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd2c06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46612/0 (socket says 172.21.15.79:46612) 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 -- 172.21.15.79:0/3354045432 learned_addr learned my addr 172.21.15.79:0/3354045432 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f8097fff640 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd2428d0 0x5593fd24f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 -- 172.21.15.79:0/3354045432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd241b60 msgr2=0x5593fd24ef30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd241b60 0x5593fd24ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 -- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd2428d0 msgr2=0x5593fd24f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd2428d0 0x5593fd24f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.909+0000 7f80977fe640 1 -- 172.21.15.79:0/3354045432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593fd24fd40 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80977fe640 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd2c06a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8088000c00 tx=0x7f8088004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ab6d8c8ad7df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80777fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80777fe640 1 -- 172.21.15.79:0/3354045432 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088047040 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80777fe640 1 -- 172.21.15.79:0/3354045432 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f808802c7e0 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8080003aa0 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80777fe640 1 -- 172.21.15.79:0/3354045432 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f808802ca00 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 10 client.17013 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5593fd2c1cf0 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80777fe640 1 -- 172.21.15.79:0/3354045432 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8080003aa0 con 0x5593fd24b0e0 2024-04-22T06:50:42.913 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80777fe640 1 -- 172.21.15.79:0/3354045432 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f80880047c0 con 0x5593fd24b0e0 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 10 client.17013 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 20 client.17013 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd2c2030 0x5593fd2c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 4 client.17013 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.910+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593fd1463d0 con 0x5593fd2c2030 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.911+0000 7f8097fff640 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd2c2030 0x5593fd2c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.912+0000 7f80777fe640 10 client.17013 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.912+0000 7f8097fff640 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd2c2030 0x5593fd2c4420 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe0e616ccc3bdc8 server_cookie=2ad60de0eafad547 in_seq=0 out_seq=0 2024-04-22T06:50:42.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:42 smithi193 ceph-mon[31815]: pgmap v199: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 367 MiB/s wr, 681 op/s 2024-04-22T06:50:42.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:42 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:50:42.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:42 smithi190 ceph-mon[31421]: pgmap v199: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 367 MiB/s wr, 681 op/s 2024-04-22T06:50:42.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:42 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:50:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.954+0000 7f80777fe640 1 -- 172.21.15.79:0/3354045432 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593fd1463d0 con 0x5593fd2c2030 2024-04-22T06:50:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.954+0000 7f80777fe640 10 client.17013 handle_command_reply: tid=0 2024-04-22T06:50:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.956+0000 7f80a1370740 1 client.17013 shutdown 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.960+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 msgr2=0x5593fd2c06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd2c06a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8088000c00 tx=0x7f8088004160 comp rx=0 tx=0).stop 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 shutdown_connections 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5593fd2c2030 0x5593fd2c4420 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5593fd2428d0 0x5593fd24f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5593fd24b0e0 0x5593fd2c06a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 --2- 172.21.15.79:0/3354045432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593fd241b60 0x5593fd24ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.961+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 >> 172.21.15.79:0/3354045432 conn(0x5593fd208c90 msgr2=0x5593fd23e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.962+0000 7f80777fe640 0 client.17013 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:42.961 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.962+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 shutdown_connections 2024-04-22T06:50:42.961 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.962+0000 7f80a1370740 1 -- 172.21.15.79:0/3354045432 wait complete. 2024-04-22T06:50:42.961 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.962+0000 7f80a1370740 20 client.17013 trim_cache size 0 max 16384 2024-04-22T06:50:42.962 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:50:42.962 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:50:42.962 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:50:42.962 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:50:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.973+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80840a3e40 msgr2=0x7f80840a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.974+0000 7f809e787640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80840a3e40 0x7f80840a6300 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f809009cb80 tx=0x7f8090067740 comp rx=0 tx=0).stop 2024-04-22T06:50:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.974+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 msgr2=0x7f80981a69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.974+0000 7f809e787640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 0x7f80981a69a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f809000eb40 tx=0x7f809000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.975 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.976+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 shutdown_connections 2024-04-22T06:50:42.975 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.977+0000 7f809e787640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8098197da0 0x7f80981a69a0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.977+0000 7f809e787640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8098106170 0x7f8098197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.977+0000 7f809e787640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80840a3e40 0x7f80840a6300 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.977+0000 7f809e787640 1 --2- 172.21.15.79:0/2910177261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80981010f0 0x7f8098197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.977+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 >> 172.21.15.79:0/2910177261 conn(0x7f8098077fd0 msgr2=0x7f80980ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.987+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 shutdown_connections 2024-04-22T06:50:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:42.987+0000 7f809e787640 1 -- 172.21.15.79:0/2910177261 wait complete. 2024-04-22T06:50:42.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:42 smithi079 ceph-mon[27946]: pgmap v199: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 367 MiB/s wr, 681 op/s 2024-04-22T06:50:42.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:42 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:50:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:43 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:50:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:43 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:50:43.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:43 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:50:43.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:43 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:50:43.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:43 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:50:43.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:43 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:50:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:43 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:50:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:43 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:50:43.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:43 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:50:44.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:44 smithi193 ceph-mon[31815]: pgmap v200: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 236 MiB/s wr, 382 op/s 2024-04-22T06:50:44.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:44 smithi190 ceph-mon[31421]: pgmap v200: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 236 MiB/s wr, 382 op/s 2024-04-22T06:50:44.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:44 smithi079 ceph-mon[27946]: pgmap v200: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 236 MiB/s wr, 382 op/s 2024-04-22T06:50:46.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:46 smithi193 ceph-mon[31815]: pgmap v201: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 91 MiB/s wr, 77 op/s 2024-04-22T06:50:46.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:46 smithi190 ceph-mon[31421]: pgmap v201: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 91 MiB/s wr, 77 op/s 2024-04-22T06:50:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:46 smithi079 ceph-mon[27946]: pgmap v201: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 91 MiB/s wr, 77 op/s 2024-04-22T06:50:48.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:48 smithi190 ceph-mon[31421]: pgmap v202: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 115 MiB/s wr, 101 op/s 2024-04-22T06:50:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:48 smithi079 ceph-mon[27946]: pgmap v202: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 115 MiB/s wr, 101 op/s 2024-04-22T06:50:49.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:48 smithi193 ceph-mon[31815]: pgmap v202: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 115 MiB/s wr, 101 op/s 2024-04-22T06:50:50.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:50 smithi190 ceph-mon[31421]: pgmap v203: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 59 MiB/s wr, 57 op/s 2024-04-22T06:50:50.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:50 smithi079 ceph-mon[27946]: pgmap v203: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 59 MiB/s wr, 57 op/s 2024-04-22T06:50:51.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:50 smithi193 ceph-mon[31815]: pgmap v203: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 59 MiB/s wr, 57 op/s 2024-04-22T06:50:52.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:52 smithi190 ceph-mon[31421]: pgmap v204: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 74 MiB/s wr, 70 op/s 2024-04-22T06:50:52.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:52 smithi079 ceph-mon[27946]: pgmap v204: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 74 MiB/s wr, 70 op/s 2024-04-22T06:50:53.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:52 smithi193 ceph-mon[31815]: pgmap v204: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 74 MiB/s wr, 70 op/s 2024-04-22T06:50:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:54 smithi190 ceph-mon[31421]: pgmap v205: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 71 MiB/s wr, 63 op/s 2024-04-22T06:50:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:54 smithi079 ceph-mon[27946]: pgmap v205: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 71 MiB/s wr, 63 op/s 2024-04-22T06:50:55.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:54 smithi193 ceph-mon[31815]: pgmap v205: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 71 MiB/s wr, 63 op/s 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 -- 172.21.15.79:0/3413405626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4814102b80 msgr2=0x7f4814106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 --2- 172.21.15.79:0/3413405626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4814102b80 0x7f4814106bc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4808009900 tx=0x7f480802bc30 comp rx=0 tx=0).stop 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 -- 172.21.15.79:0/3413405626 shutdown_connections 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 --2- 172.21.15.79:0/3413405626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48141072a0 0x7f481410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 --2- 172.21.15.79:0/3413405626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4814102b80 0x7f4814106bc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 --2- 172.21.15.79:0/3413405626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48141021d0 0x7f48141025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 -- 172.21.15.79:0/3413405626 >> 172.21.15.79:0/3413405626 conn(0x7f48140793e0 msgr2=0x7f48140797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 -- 172.21.15.79:0/3413405626 shutdown_connections 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 -- 172.21.15.79:0/3413405626 wait complete. 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.130+0000 7f481945c640 1 Processor -- start 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 -- start start 2024-04-22T06:50:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48141021d0 0x7f4814198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48141072a0 0x7f4814198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 0x7f481419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48141121e0 con 0x7f481419cec0 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4814112060 con 0x7f48141072a0 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f481945c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4814112360 con 0x7f48141021d0 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 0x7f481419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 0x7f481419d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39096/0 (socket says 172.21.15.79:39096) 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 -- 172.21.15.79:0/1131362745 learned_addr learned my addr 172.21.15.79:0/1131362745 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f4812ffd640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48141021d0 0x7f4814198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48127fc640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48141072a0 0x7f4814198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 -- 172.21.15.79:0/1131362745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48141021d0 msgr2=0x7f4814198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48141021d0 0x7f4814198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 -- 172.21.15.79:0/1131362745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48141072a0 msgr2=0x7f4814198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48141072a0 0x7f4814198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 -- 172.21.15.79:0/1131362745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481419d9b0 con 0x7f481419cec0 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.131+0000 7f48137fe640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 0x7f481419d370 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f480c00cab0 tx=0x7f480c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.132+0000 7f47fbfff640 1 -- 172.21.15.79:0/1131362745 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c00ae20 con 0x7f481419cec0 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.132+0000 7f47fbfff640 1 -- 172.21.15.79:0/1131362745 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f480c00f900 con 0x7f481419cec0 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.132+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f481419dca0 con 0x7f481419cec0 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.132+0000 7f47fbfff640 1 -- 172.21.15.79:0/1131362745 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c0189f0 con 0x7f481419cec0 2024-04-22T06:50:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.132+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48141a5460 con 0x7f481419cec0 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f47fbfff640 1 -- 172.21.15.79:0/1131362745 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f480c00fab0 con 0x7f481419cec0 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 Processor -- start 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 -- start start 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f47fbfff640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47fc0a3e40 0x7f47fc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562516f058d0 con 0x562516fa0b60 2024-04-22T06:50:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.134+0000 7f481c052740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562516ea2e80 con 0x562516fa18d0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f481c052740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562516f01890 con 0x562516faa0e0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f47fbfff640 1 -- 172.21.15.79:0/1131362745 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f480c0d10b0 con 0x7f481419cec0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47fc0a3e40 0x7f47fc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39100/0 (socket says 172.21.15.79:39100) 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 -- 172.21.15.79:0/1789818620 learned_addr learned my addr 172.21.15.79:0/1789818620 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48137fe640 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f4812ffd640 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 -- 172.21.15.79:0/1789818620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 msgr2=0x562516fac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fac4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 -- 172.21.15.79:0/1789818620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 msgr2=0x562516fa0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 -- 172.21.15.79:0/1789818620 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562516ec4320 con 0x562516fa0b60 2024-04-22T06:50:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48137fe640 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47fc0a3e40 0x7f47fc0a6300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4808009230 tx=0x7f4808004110 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f48127fc640 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa0fe0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f481406b630 tx=0x7f4808047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4beff55dc1d472 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f47f97fa640 1 -- 172.21.15.79:0/1789818620 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808053070 con 0x562516fa0b60 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f47f97fa640 1 -- 172.21.15.79:0/1789818620 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48080054d0 con 0x562516fa0b60 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.135+0000 7f47f97fa640 1 -- 172.21.15.79:0/1789818620 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808003910 con 0x562516fa0b60 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 -- 172.21.15.79:0/1789818620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 msgr2=0x562516fa0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa0fe0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f481406b630 tx=0x7f4808047000 comp rx=0 tx=0).stop 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 -- 172.21.15.79:0/1789818620 shutdown_connections 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa0fe0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 --2- 172.21.15.79:0/1789818620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 -- 172.21.15.79:0/1789818620 >> 172.21.15.79:0/1789818620 conn(0x562516f67c90 msgr2=0x562516f9b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 -- 172.21.15.79:0/1789818620 shutdown_connections 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 -- 172.21.15.79:0/1789818620 wait complete. 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.136+0000 7f481c052740 1 -- start start 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 Processor -- start 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fa9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fa6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fa7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562516facdb0 con 0x562516fa18d0 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562516e8deb0 con 0x562516fa0b60 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562516facf30 con 0x562516faa0e0 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f481c052740 10 client.0 _ll_register_callbacks cb 0x562516ebab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fa6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fa6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39114/0 (socket says 172.21.15.79:39114) 2024-04-22T06:50:56.143 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 -- 172.21.15.79:0/3654351267 learned_addr learned my addr 172.21.15.79:0/3654351267 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f4812ffd640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fa9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48137fe640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fa7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 -- 172.21.15.79:0/3654351267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 msgr2=0x562516fa7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fa7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 -- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 msgr2=0x562516fa9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fa9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 -- 172.21.15.79:0/3654351267 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562516c18210 con 0x562516fa18d0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f48127fc640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fa6e70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f480802bc70 tx=0x7f4808004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613f74ca151ccc63 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.137+0000 7f47f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f47f6ff5640 1 -- 172.21.15.79:0/3654351267 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480805c020 con 0x562516fa18d0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f47f6ff5640 1 -- 172.21.15.79:0/3654351267 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f4808003ae0 con 0x562516fa18d0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48080054d0 con 0x562516fa18d0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f47f6ff5640 1 -- 172.21.15.79:0/3654351267 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f4808067050 con 0x562516fa18d0 2024-04-22T06:50:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 10 client.17025 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x562516ce4620 con 0x562516fa18d0 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f47f6ff5640 1 -- 172.21.15.79:0/3654351267 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48080054d0 con 0x562516fa18d0 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f47f6ff5640 1 -- 172.21.15.79:0/3654351267 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f4808043a20 con 0x562516fa18d0 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 10 client.17025 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 10 client.17025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 20 client.17025 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562516fb9f60 0x562516fbc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 4 client.17025 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562516f19880 con 0x562516fb9f60 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.138+0000 7f4812ffd640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562516fb9f60 0x562516fbc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.139+0000 7f47f6ff5640 10 client.17025 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.139+0000 7f4812ffd640 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562516fb9f60 0x562516fbc350 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a9ab12cd983b64 server_cookie=6db19ed899a26cd8 in_seq=0 out_seq=0 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f47f6ff5640 1 -- 172.21.15.79:0/3654351267 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562516f19880 con 0x562516fb9f60 2024-04-22T06:50:56.145 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f47f6ff5640 10 client.17025 handle_command_reply: tid=0 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 client.17025 shutdown 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 msgr2=0x562516fa6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fa6e70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f480802bc70 tx=0x7f4808004d00 comp rx=0 tx=0).stop 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 shutdown_connections 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562516fb9f60 0x562516fbc350 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fa7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fa6e70 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 --2- 172.21.15.79:0/3654351267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fa9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 >> 172.21.15.79:0/3654351267 conn(0x562516f67c90 msgr2=0x562516f9d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f47f6ff5640 0 client.17025 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 shutdown_connections 2024-04-22T06:50:56.146 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 1 -- 172.21.15.79:0/3654351267 wait complete. 2024-04-22T06:50:56.147 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.140+0000 7f481c052740 20 client.17025 trim_cache size 0 max 16384 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 Processor -- start 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 -- start start 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48141b0940 con 0x562516fa0b60 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48141ae210 con 0x562516fa18d0 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f481c052740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48141ac060 con 0x562516faa0e0 2024-04-22T06:50:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f4812ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f4812ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39122/0 (socket says 172.21.15.79:39122) 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f4812ffd640 1 -- 172.21.15.79:0/1135964304 learned_addr learned my addr 172.21.15.79:0/1135964304 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.155+0000 7f48127fc640 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f4812ffd640 1 -- 172.21.15.79:0/1135964304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 msgr2=0x562516fbef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f4812ffd640 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f4812ffd640 1 -- 172.21.15.79:0/1135964304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 msgr2=0x562516fa9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f4812ffd640 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f4812ffd640 1 -- 172.21.15.79:0/1135964304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562516f9fe30 con 0x562516fa0b60 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f48127fc640 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f4812ffd640 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa95a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f48040058c0 tx=0x7f480400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40f886ddf502da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f47f8ff9640 1 -- 172.21.15.79:0/1135964304 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804005020 con 0x562516fa0b60 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f47f8ff9640 1 -- 172.21.15.79:0/1135964304 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4804024030 con 0x562516fa0b60 2024-04-22T06:50:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.156+0000 7f47f8ff9640 1 -- 172.21.15.79:0/1135964304 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804020350 con 0x562516fa0b60 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- 172.21.15.79:0/1135964304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 msgr2=0x562516fa95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa95a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f48040058c0 tx=0x7f480400a250 comp rx=0 tx=0).stop 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- 172.21.15.79:0/1135964304 shutdown_connections 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa18d0 0x562516fa9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- 172.21.15.79:0/1135964304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa0b60 0x562516fa95a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- 172.21.15.79:0/1135964304 >> 172.21.15.79:0/1135964304 conn(0x562516f67c90 msgr2=0x562516f96760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- 172.21.15.79:0/1135964304 shutdown_connections 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- 172.21.15.79:0/1135964304 wait complete. 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- start start 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 Processor -- start 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fbf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fbdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.160 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48141ac6d0 con 0x562516fa18d0 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48141adb40 con 0x562516fa0b60 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48141acda0 con 0x562516faa0e0 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f481c052740 10 client.0 _ll_register_callbacks cb 0x7f4808051070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f48127fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fbdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f48127fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fbdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39132/0 (socket says 172.21.15.79:39132) 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.157+0000 7f48127fc640 1 -- 172.21.15.79:0/2893738858 learned_addr learned my addr 172.21.15.79:0/2893738858 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48137fe640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f4812ffd640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fbf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48127fc640 1 -- 172.21.15.79:0/2893738858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 msgr2=0x562516fbe180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48127fc640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbe180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48127fc640 1 -- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 msgr2=0x562516fbf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48127fc640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fbf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48127fc640 1 -- 172.21.15.79:0/2893738858 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562516fbe740 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f48127fc640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fbdc40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x562516fa8440 tx=0x7f4808005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31f7b7ce34d6e0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f47f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f47f6ff5640 1 -- 172.21.15.79:0/2893738858 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808070060 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f47f6ff5640 1 -- 172.21.15.79:0/2893738858 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f48080057f0 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4804024030 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f47f6ff5640 1 -- 172.21.15.79:0/2893738858 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f4808077050 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f481c052740 10 client.17031 fetch_fsmap learned FSMap version 54 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x56251701ff00 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.158+0000 7f47f6ff5640 1 -- 172.21.15.79:0/2893738858 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4804024030 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f47f6ff5640 1 -- 172.21.15.79:0/2893738858 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f48080438e0 con 0x562516fa18d0 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f481c052740 10 client.17031 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f481c052740 10 client.17031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f481c052740 20 client.17031 populate_metadata read hostname 'smithi079' 2024-04-22T06:50:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f481c052740 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562517020650 0x562517022a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:56.163 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f481c052740 4 client.17031 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:56.163 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562516ea53d0 con 0x562517020650 2024-04-22T06:50:56.163 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f4812ffd640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562517020650 0x562517022a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:56.163 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f47f6ff5640 10 client.17031 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:56.163 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.159+0000 7f4812ffd640 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562517020650 0x562517022a70 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858de5c2b7994642 server_cookie=13ba6ad91890439d in_seq=0 out_seq=0 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f47f6ff5640 1 -- 172.21.15.79:0/2893738858 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562516ea53d0 con 0x562517020650 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f47f6ff5640 10 client.17031 handle_command_reply: tid=0 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 client.17031 shutdown 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 msgr2=0x562516fbdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fbdc40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x562516fa8440 tx=0x7f4808005700 comp rx=0 tx=0).stop 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 shutdown_connections 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562517020650 0x562517022a70 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562516faa0e0 0x562516fbe180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562516fa18d0 0x562516fbdc40 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f47f6ff5640 0 client.17031 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:50:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 --2- 172.21.15.79:0/2893738858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562516fa0b60 0x562516fbf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 >> 172.21.15.79:0/2893738858 conn(0x562516f67c90 msgr2=0x562516fa1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 shutdown_connections 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 1 -- 172.21.15.79:0/2893738858 wait complete. 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.170+0000 7f481c052740 20 client.17031 trim_cache size 0 max 16384 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:50:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:50:56.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47fc0a3e40 msgr2=0x7f47fc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47fc0a3e40 0x7f47fc0a6300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4808009230 tx=0x7f4808004110 comp rx=0 tx=0).stop 2024-04-22T06:50:56.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 msgr2=0x7f481419d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:56.171 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 0x7f481419d370 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f480c00cab0 tx=0x7f480c00cf70 comp rx=0 tx=0).stop 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 shutdown_connections 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47fc0a3e40 0x7f47fc0a6300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48141072a0 0x7f4814198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f481419cec0 0x7f481419d370 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 --2- 172.21.15.79:0/1131362745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48141021d0 0x7f4814198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 >> 172.21.15.79:0/1131362745 conn(0x7f48140793e0 msgr2=0x7f48140ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 shutdown_connections 2024-04-22T06:50:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:50:56.171+0000 7f481945c640 1 -- 172.21.15.79:0/1131362745 wait complete. 2024-04-22T06:50:56.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:56 smithi079 ceph-mon[27946]: pgmap v206: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 58 MiB/s wr, 53 op/s 2024-04-22T06:50:57.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:56 smithi193 ceph-mon[31815]: pgmap v206: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 58 MiB/s wr, 53 op/s 2024-04-22T06:50:57.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:56 smithi190 ceph-mon[31421]: pgmap v206: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 58 MiB/s wr, 53 op/s 2024-04-22T06:50:58.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:50:58 smithi079 ceph-mon[27946]: pgmap v207: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 71 MiB/s wr, 66 op/s 2024-04-22T06:50:59.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:50:58 smithi193 ceph-mon[31815]: pgmap v207: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 71 MiB/s wr, 66 op/s 2024-04-22T06:50:59.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:50:58 smithi190 ceph-mon[31421]: pgmap v207: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 71 MiB/s wr, 66 op/s 2024-04-22T06:51:00.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:00 smithi079 ceph-mon[27946]: pgmap v208: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 52 MiB/s wr, 46 op/s 2024-04-22T06:51:01.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:00 smithi193 ceph-mon[31815]: pgmap v208: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 52 MiB/s wr, 46 op/s 2024-04-22T06:51:01.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:00 smithi190 ceph-mon[31421]: pgmap v208: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 52 MiB/s wr, 46 op/s 2024-04-22T06:51:02.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:02 smithi079 ceph-mon[27946]: pgmap v209: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 61 MiB/s wr, 64 op/s 2024-04-22T06:51:02.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:02 smithi193 ceph-mon[31815]: pgmap v209: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 61 MiB/s wr, 64 op/s 2024-04-22T06:51:03.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:02 smithi190 ceph-mon[31421]: pgmap v209: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 61 MiB/s wr, 64 op/s 2024-04-22T06:51:04.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:04 smithi079 ceph-mon[27946]: pgmap v210: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 60 MiB/s wr, 82 op/s 2024-04-22T06:51:05.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:04 smithi190 ceph-mon[31421]: pgmap v210: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 60 MiB/s wr, 82 op/s 2024-04-22T06:51:05.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:04 smithi193 ceph-mon[31815]: pgmap v210: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 60 MiB/s wr, 82 op/s 2024-04-22T06:51:06.368 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 -- 172.21.15.79:0/1408895280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c106180 msgr2=0x7fcc4c10dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/1408895280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c106180 0x7fcc4c10dd50 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009ca0 tx=0x7fcc4402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 -- 172.21.15.79:0/1408895280 shutdown_connections 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/1408895280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c101a60 0x7fcc4c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/1408895280 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c106180 0x7fcc4c10dd50 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/1408895280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 -- 172.21.15.79:0/1408895280 >> 172.21.15.79:0/1408895280 conn(0x7fcc4c0771d0 msgr2=0x7fcc4c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 -- 172.21.15.79:0/1408895280 shutdown_connections 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.481+0000 7fcc52ff4640 1 -- 172.21.15.79:0/1408895280 wait complete. 2024-04-22T06:51:06.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 Processor -- start 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 -- start start 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc4c106180 0x7fcc4c197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c197d80 0x7fcc4c1a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c1110f0 con 0x7fcc4c1010b0 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c110f70 con 0x7fcc4c197d80 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc52ff4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c111270 con 0x7fcc4c106180 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc50d90640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc50d90640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47622/0 (socket says 172.21.15.79:47622) 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc50d90640 1 -- 172.21.15.79:0/3065336052 learned_addr learned my addr 172.21.15.79:0/3065336052 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.482+0000 7fcc50d90640 1 -- 172.21.15.79:0/3065336052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc4c106180 msgr2=0x7fcc4c197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc4bfff640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc4c106180 0x7fcc4c197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc51591640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c197d80 0x7fcc4c1a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc50d90640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc4c106180 0x7fcc4c197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc50d90640 1 -- 172.21.15.79:0/3065336052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c197d80 msgr2=0x7fcc4c1a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc50d90640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c197d80 0x7fcc4c1a6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc50d90640 1 -- 172.21.15.79:0/3065336052 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c1a6f40 con 0x7fcc4c1010b0 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc51591640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c197d80 0x7fcc4c1a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc50d90640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c197300 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000d7a0 tx=0x7fcc4000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40017070 con 0x7fcc4c1010b0 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc400054d0 con 0x7fcc4c1010b0 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c1a7230 con 0x7fcc4c1010b0 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40004c60 con 0x7fcc4c1010b0 2024-04-22T06:51:06.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.483+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c1a7730 con 0x7fcc4c1010b0 2024-04-22T06:51:06.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.484+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4c069230 con 0x7fcc4c1010b0 2024-04-22T06:51:06.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.485+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc40004e00 con 0x7fcc4c1010b0 2024-04-22T06:51:06.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.485+0000 7fcc49ffb640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc3c0a3d20 0x7fcc3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.486+0000 7fcc4bfff640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc3c0a3d20 0x7fcc3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.486+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7fcc400d0590 con 0x7fcc4c1010b0 2024-04-22T06:51:06.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.486+0000 7fcc4bfff640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc3c0a3d20 0x7fcc3c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c199460 tx=0x7fcc38005ed0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.490+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc4009c9b0 con 0x7fcc4c1010b0 2024-04-22T06:51:06.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.706+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc4c109470 con 0x7fcc4c1010b0 2024-04-22T06:51:06.705 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.707+0000 7fcc49ffb640 1 -- 172.21.15.79:0/3065336052 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11878 (secure 0 0 0) 0x7fcc40096e20 con 0x7fcc4c1010b0 2024-04-22T06:51:06.706 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:51:06.706 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":54,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:49:56.426792+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25817,"mds_2":25810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24424":{"gid":24424,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/678626858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":678626858},{"type":"v1","addr":"172.21.15.193:6837","nonce":678626858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25810":{"gid":25810,"name":"i","rank":2,"incarnation":49,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/1413649941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1413649941},{"type":"v1","addr":"172.21.15.193:6839","nonce":1413649941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25817":{"gid":25817,"name":"h","rank":1,"incarnation":49,"state":"up:active","state_seq":23,"addr":"172.21.15.190:6837/1169559599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1169559599},{"type":"v1","addr":"172.21.15.190:6837","nonce":1169559599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25855":{"gid":25855,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1167513219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1167513219},{"type":"v1","addr":"172.21.15.190:6835","nonce":1167513219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25865":{"gid":25865,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/99277862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":99277862},{"type":"v1","addr":"172.21.15.193:6835","nonce":99277862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,25817,25810]},"id":1}]} 2024-04-22T06:51:06.706 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 54 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc3c0a3d20 msgr2=0x7fcc3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc3c0a3d20 0x7fcc3c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c199460 tx=0x7fcc38005ed0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 msgr2=0x7fcc4c197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c197300 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000d7a0 tx=0x7fcc4000dc60 comp rx=0 tx=0).stop 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 shutdown_connections 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcc3c0a3d20 0x7fcc3c0a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc4c197d80 0x7fcc4c1a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc4c106180 0x7fcc4c197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 --2- 172.21.15.79:0/3065336052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc4c1010b0 0x7fcc4c197300 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 >> 172.21.15.79:0/3065336052 conn(0x7fcc4c0771d0 msgr2=0x7fcc4c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 shutdown_connections 2024-04-22T06:51:06.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.709+0000 7fcc52ff4640 1 -- 172.21.15.79:0/3065336052 wait complete. 2024-04-22T06:51:06.720 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id 6e8daeb4 --release --await' 2024-04-22T06:51:06.720 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id 6e8daeb4 --release --await 2024-04-22T06:51:06.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 -- 172.21.15.79:0/557910484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec1005c0 msgr2=0x7f43ec100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 --2- 172.21.15.79:0/557910484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec1005c0 0x7f43ec100a40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00aa10 tx=0x7f43dc02cf70 comp rx=0 tx=0).stop 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 -- 172.21.15.79:0/557910484 shutdown_connections 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 --2- 172.21.15.79:0/557910484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43ec100f80 0x7f43ec10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 --2- 172.21.15.79:0/557910484 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec1005c0 0x7f43ec100a40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 --2- 172.21.15.79:0/557910484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ec106680 0x7f43ec106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 -- 172.21.15.79:0/557910484 >> 172.21.15.79:0/557910484 conn(0x7f43ec077410 msgr2=0x7f43ec0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 -- 172.21.15.79:0/557910484 shutdown_connections 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.826+0000 7f43f33f4640 1 -- 172.21.15.79:0/557910484 wait complete. 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 Processor -- start 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 -- start start 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43ec1005c0 0x7f43ec1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ec100f80 0x7f43ec1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 0x7f43ec19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec10fff0 con 0x7f43ec106680 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec10fe70 con 0x7f43ec100f80 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f33f4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec110170 con 0x7f43ec1005c0 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 0x7f43ec19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 0x7f43ec19ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47652/0 (socket says 172.21.15.79:47652) 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 -- 172.21.15.79:0/187610218 learned_addr learned my addr 172.21.15.79:0/187610218 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f098f640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ec100f80 0x7f43ec1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 -- 172.21.15.79:0/187610218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43ec1005c0 msgr2=0x7f43ec1964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43ec1005c0 0x7f43ec1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 -- 172.21.15.79:0/187610218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ec100f80 msgr2=0x7f43ec1969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ec100f80 0x7f43ec1969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 -- 172.21.15.79:0/187610218 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ec19b4a0 con 0x7f43ec106680 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.827+0000 7f43f1991640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 0x7f43ec19ad70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f43e400eaa0 tx=0x7f43e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.828+0000 7f43e27fc640 1 -- 172.21.15.79:0/187610218 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e400cdb0 con 0x7f43ec106680 2024-04-22T06:51:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.828+0000 7f43e27fc640 1 -- 172.21.15.79:0/187610218 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43e4005020 con 0x7f43ec106680 2024-04-22T06:51:06.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.828+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ec19b790 con 0x7f43ec106680 2024-04-22T06:51:06.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.828+0000 7f43e27fc640 1 -- 172.21.15.79:0/187610218 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e40116d0 con 0x7f43ec106680 2024-04-22T06:51:06.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.828+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ec1a5230 con 0x7f43ec106680 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.830+0000 7f43e27fc640 1 -- 172.21.15.79:0/187610218 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f43e40051d0 con 0x7f43ec106680 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.830+0000 7f43f5fee740 1 Processor -- start 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.830+0000 7f43f5fee740 1 -- start start 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492310 0x5653584926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535849b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x56535849dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f5fee740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5653583f7500 con 0x56535849b810 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f5fee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56535839d6a0 con 0x565358492310 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f5fee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5653583f34c0 con 0x565358492c30 2024-04-22T06:51:06.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43e27fc640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d80a3da0 0x7f43d80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x56535849dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x56535849dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47662/0 (socket says 172.21.15.79:47662) 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 -- 172.21.15.79:0/2978984197 learned_addr learned my addr 172.21.15.79:0/2978984197 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f1991640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d80a3da0 0x7f43d80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f1190640 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535849b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492310 0x5653584926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 -- 172.21.15.79:0/2978984197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 msgr2=0x56535849b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43e27fc640 1 -- 172.21.15.79:0/187610218 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f43e40d0cb0 con 0x7f43ec106680 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535849b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 -- 172.21.15.79:0/2978984197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492310 msgr2=0x5653584926f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492310 0x5653584926f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 -- 172.21.15.79:0/2978984197 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653583bc5e0 con 0x56535849b810 2024-04-22T06:51:06.836 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f098f640 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x56535849dc00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f43dc035870 tx=0x7f43dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1975ec32b488b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43f1991640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d80a3da0 0x7f43d80a6260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f43e40993a0 tx=0x7f43e40c7ae0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43d77fe640 1 -- 172.21.15.79:0/2978984197 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0359f0 con 0x56535849b810 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43d77fe640 1 -- 172.21.15.79:0/2978984197 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc035b90 con 0x56535849b810 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.831+0000 7f43d77fe640 1 -- 172.21.15.79:0/2978984197 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0035f0 con 0x56535849b810 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 -- 172.21.15.79:0/2978984197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 msgr2=0x56535849dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x56535849dc00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f43dc035870 tx=0x7f43dc000bc0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 -- 172.21.15.79:0/2978984197 shutdown_connections 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x56535849dc00 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535849b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 --2- 172.21.15.79:0/2978984197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492310 0x5653584926f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 -- 172.21.15.79:0/2978984197 >> 172.21.15.79:0/2978984197 conn(0x56535848ca40 msgr2=0x56535848ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 -- 172.21.15.79:0/2978984197 shutdown_connections 2024-04-22T06:51:06.837 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 -- 172.21.15.79:0/2978984197 wait complete. 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.832+0000 7f43f5fee740 1 -- start start 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 Processor -- start 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358498670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x565358498bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56535849b810 0x5653584990f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56535849e540 con 0x565358492310 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5653583b0a60 con 0x565358492c30 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56535849e6c0 con 0x56535849b810 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f5fee740 10 client.0 _ll_register_callbacks cb 0x565358109110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358498670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358498670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47664/0 (socket says 172.21.15.79:47664) 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 -- 172.21.15.79:0/431276194 learned_addr learned my addr 172.21.15.79:0/431276194 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f1190640 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x565358498bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.838 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 -- 172.21.15.79:0/431276194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56535849b810 msgr2=0x5653584990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56535849b810 0x5653584990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 -- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 msgr2=0x565358498bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x565358498bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.833+0000 7f43f098f640 1 -- 172.21.15.79:0/431276194 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565358109920 con 0x565358492310 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f098f640 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358498670 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00a6b0 tx=0x7f43dc00a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb15c613d8aade96 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43d4ff9640 1 -- 172.21.15.79:0/431276194 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0046c0 con 0x565358492310 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43d4ff9640 1 -- 172.21.15.79:0/431276194 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f43dc004860 con 0x565358492310 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43dc035b90 con 0x565358492310 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43d4ff9640 1 -- 172.21.15.79:0/431276194 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f43dc004a60 con 0x565358492310 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 10 client.17049 fetch_fsmap learned FSMap version 54 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5653581d5d30 con 0x565358492310 2024-04-22T06:51:06.839 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43d4ff9640 1 -- 172.21.15.79:0/431276194 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc035b90 con 0x565358492310 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43d4ff9640 1 -- 172.21.15.79:0/431276194 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f43dc04d070 con 0x565358492310 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 10 client.17049 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 10 client.17049 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 20 client.17049 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653584ab7c0 0x5653584adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 4 client.17049 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.834+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653583caa10 con 0x5653584ab7c0 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.835+0000 7f43f1190640 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653584ab7c0 0x5653584adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.835+0000 7f43d4ff9640 10 client.17049 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.835+0000 7f43f1190640 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653584ab7c0 0x5653584adbb0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e183b23195d419 server_cookie=4b2fef59c3575017 in_seq=0 out_seq=0 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43d4ff9640 1 -- 172.21.15.79:0/431276194 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653583caa10 con 0x5653584ab7c0 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43d4ff9640 10 client.17049 handle_command_reply: tid=0 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 client.17049 shutdown 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 msgr2=0x565358498670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.840 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358498670 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00a6b0 tx=0x7f43dc00a6e0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 shutdown_connections 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653584ab7c0 0x5653584adbb0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x565358498bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56535849b810 0x5653584990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 --2- 172.21.15.79:0/431276194 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358498670 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 >> 172.21.15.79:0/431276194 conn(0x56535848ca40 msgr2=0x56535848e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43d4ff9640 0 client.17049 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 shutdown_connections 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 1 -- 172.21.15.79:0/431276194 wait complete. 2024-04-22T06:51:06.841 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.836+0000 7f43f5fee740 20 client.17049 trim_cache size 0 max 16384 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 Processor -- start 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 -- start start 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492310 0x56535849ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x5653584b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x5653584958d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec1b0ac0 con 0x56535849b810 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec1ae390 con 0x565358492c30 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f5fee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec1ac1e0 con 0x565358492310 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x5653584958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x5653584958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47670/0 (socket says 172.21.15.79:47670) 2024-04-22T06:51:06.852 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 -- 172.21.15.79:0/3649752543 learned_addr learned my addr 172.21.15.79:0/3649752543 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 -- 172.21.15.79:0/3649752543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492310 msgr2=0x56535849ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492310 0x56535849ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 -- 172.21.15.79:0/3649752543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 msgr2=0x5653584b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x5653584b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 -- 172.21.15.79:0/3649752543 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653584916b0 con 0x56535849b810 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.853+0000 7f43f098f640 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x5653584958d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0059d0 tx=0x7f43dc035690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c1b0fb5590efc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43d6ffd640 1 -- 172.21.15.79:0/3649752543 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc05d070 con 0x56535849b810 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43d6ffd640 1 -- 172.21.15.79:0/3649752543 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc00a890 con 0x56535849b810 2024-04-22T06:51:06.853 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43d6ffd640 1 -- 172.21.15.79:0/3649752543 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc00d000 con 0x56535849b810 2024-04-22T06:51:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 -- 172.21.15.79:0/3649752543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 msgr2=0x5653584958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x5653584958d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0059d0 tx=0x7f43dc035690 comp rx=0 tx=0).stop 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 -- 172.21.15.79:0/3649752543 shutdown_connections 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565358492c30 0x5653584b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x56535849b810 0x5653584958d0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3649752543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492310 0x56535849ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 -- 172.21.15.79:0/3649752543 >> 172.21.15.79:0/3649752543 conn(0x56535848ca40 msgr2=0x56535848e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 -- 172.21.15.79:0/3649752543 shutdown_connections 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 -- 172.21.15.79:0/3649752543 wait complete. 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.854+0000 7f43f5fee740 1 -- start start 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 Processor -- start 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358509b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535850a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56535849b810 0x5653585149d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec1ac850 con 0x565358492310 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec1adcc0 con 0x56535849b810 2024-04-22T06:51:06.857 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec1acf20 con 0x565358492c30 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f5fee740 10 client.0 _ll_register_callbacks cb 0x7f43dc0355e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358509b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358509b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47674/0 (socket says 172.21.15.79:47674) 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 -- 172.21.15.79:0/3235430822 learned_addr learned my addr 172.21.15.79:0/3235430822 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56535849b810 0x5653585149d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f1190640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535850a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 -- 172.21.15.79:0/3235430822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 msgr2=0x56535850a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535850a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 -- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56535849b810 msgr2=0x5653585149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56535849b810 0x5653585149d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f098f640 1 -- 172.21.15.79:0/3235430822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565358514f10 con 0x565358492310 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.855+0000 7f43f1190640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535850a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f098f640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358509b40 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f43dc035700 tx=0x7f43dc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9840f93ec255b0ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43d4ff9640 1 -- 172.21.15.79:0/3235430822 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc05d070 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43d4ff9640 1 -- 172.21.15.79:0/3235430822 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9157+0+0 (secure 0 0 0) 0x7f43dc005460 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43dc00a890 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43d4ff9640 1 -- 172.21.15.79:0/3235430822 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f43dc00d210 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 10 client.17055 fetch_fsmap learned FSMap version 54 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x565358516230 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43d4ff9640 1 -- 172.21.15.79:0/3235430822 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc00a890 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43d4ff9640 1 -- 172.21.15.79:0/3235430822 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7f43dc04d070 con 0x565358492310 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 10 client.17055 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 10 client.17055 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 20 client.17055 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653585167f0 0x565358518c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 4 client.17055 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6e8daeb4", "release": true, "await": true}] 2024-04-22T06:51:06.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6e8daeb4", "release": true, "await": true}) -- 0x565358404f20 con 0x5653585167f0 2024-04-22T06:51:06.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.856+0000 7f43f1190640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653585167f0 0x565358518c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.857+0000 7f43d4ff9640 10 client.17055 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:06.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.857+0000 7f43f1190640 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653585167f0 0x565358518c10 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca30158b0d26189 server_cookie=f904203d17b45a32 in_seq=0 out_seq=0 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.861+0000 7f43d4ff9640 1 -- 172.21.15.79:0/3235430822 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x565358404f20 con 0x5653585167f0 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.861+0000 7f43d4ff9640 10 client.17055 handle_command_reply: tid=0 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.861+0000 7f43f5fee740 1 client.17055 shutdown 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 msgr2=0x565358509b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358509b40 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f43dc035700 tx=0x7f43dc005b50 comp rx=0 tx=0).stop 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 shutdown_connections 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5653585167f0 0x565358518c10 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56535849b810 0x5653585149d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565358492c30 0x56535850a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 --2- 172.21.15.79:0/3235430822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x565358492310 0x565358509b40 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 >> 172.21.15.79:0/3235430822 conn(0x56535848ca40 msgr2=0x565358498630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43d4ff9640 0 client.17055 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 shutdown_connections 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 1 -- 172.21.15.79:0/3235430822 wait complete. 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.862+0000 7f43f5fee740 20 client.17055 trim_cache size 0 max 16384 2024-04-22T06:51:06.862 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 54, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 22, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "6e8daeb4": { 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 22, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 93.7, 2024-04-22T06:51:06.863 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:51:06.864 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d80a3da0 msgr2=0x7f43d80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d80a3da0 0x7f43d80a6260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f43e40993a0 tx=0x7f43e40c7ae0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 msgr2=0x7f43ec19ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 0x7f43ec19ad70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f43e400eaa0 tx=0x7f43e400ef60 comp rx=0 tx=0).stop 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 shutdown_connections 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f43d80a3da0 0x7f43d80a6260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ec100f80 0x7f43ec1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43ec106680 0x7f43ec19ad70 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 --2- 172.21.15.79:0/187610218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43ec1005c0 0x7f43ec1964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.863+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 >> 172.21.15.79:0/187610218 conn(0x7f43ec077410 msgr2=0x7f43ec0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.864+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 shutdown_connections 2024-04-22T06:51:06.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:51:06.864+0000 7f43f33f4640 1 -- 172.21.15.79:0/187610218 wait complete. 2024-04-22T06:51:06.886 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 6e8daeb4, seconds elapsed: 35.8 2024-04-22T06:51:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:06 smithi079 ceph-mon[27946]: pgmap v211: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 MiB/s wr, 71 op/s 2024-04-22T06:51:06.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:06 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3065336052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:51:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-mon[31815]: pgmap v211: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 MiB/s wr, 71 op/s 2024-04-22T06:51:07.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3065336052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:51:07.189 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f73539e6700 time 2024-04-22T06:51:06.859151+0000 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73670213dd] 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.190 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: clone() 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:06.859+0000 7f73539e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f73539e6700 time 2024-04-22T06:51:06.859151+0000 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73670213dd] 2024-04-22T06:51:07.191 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.192 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.192 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.192 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.192 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.192 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: clone() 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: *** Caught signal (Aborted) ** 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: in thread 7f73539e6700 thread_name:quiesce.agt 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7365da9d20] 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: gsignal() 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: abort() 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7367021437] 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.193 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 13: clone() 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:06.860+0000 7f73539e6700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: in thread 7f73539e6700 thread_name:quiesce.agt 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.194 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7365da9d20] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: gsignal() 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: abort() 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7367021437] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 13: clone() 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:07.195 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: -1> 2024-04-22T06:51:06.859+0000 7f73539e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f73539e6700 time 2024-04-22T06:51:06.859151+0000 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73670213dd] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.196 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: clone() 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 0> 2024-04-22T06:51:06.860+0000 7f73539e6700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: in thread 7f73539e6700 thread_name:quiesce.agt 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7365da9d20] 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: gsignal() 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: abort() 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7367021437] 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.197 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 13: clone() 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:07.198 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: -2> 2024-04-22T06:51:06.859+0000 7f73539e6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f73539e6700 time 2024-04-22T06:51:06.859151+0000 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f73670213dd] 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.199 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: clone() 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: -1> 2024-04-22T06:51:06.860+0000 7f73539e6700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: in thread 7f73539e6700 thread_name:quiesce.agt 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.200 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7365da9d20] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2: gsignal() 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 3: abort() 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7367021437] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f73670215a3] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560b60792e9c] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560b60792f9f] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560b606c10d3] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560b606c103c] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560b60940d3f] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560b609a9560] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7365d9f1ca] 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 13: clone() 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67133]: 2024-04-22T06:51:07.201 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:06 smithi193 podman[67801]: 2024-04-22 06:51:06.923991735 +0000 UTC m=+0.030602522 container died 60088dbe46cb57c31b342a2d18a2502038138bab5cf0b55ab7d906d7d630d1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T06:51:07.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-mon[31421]: pgmap v211: 97 pgs: 97 active+clean; 49 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 MiB/s wr, 71 op/s 2024-04-22T06:51:07.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3065336052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f185fadc700 time 2024-04-22T06:51:06.858813+0000 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f18731173dd] 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: clone() 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:06.858+0000 7f185fadc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f185fadc700 time 2024-04-22T06:51:06.858813+0000 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f18731173dd] 2024-04-22T06:51:07.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: clone() 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.205 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: *** Caught signal (Aborted) ** 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: in thread 7f185fadc700 thread_name:quiesce.agt 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1871e9fd20] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: gsignal() 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: abort() 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1873117437] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.206 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 13: clone() 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:06.860+0000 7f185fadc700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: in thread 7f185fadc700 thread_name:quiesce.agt 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1871e9fd20] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: gsignal() 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: abort() 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1873117437] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.207 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 13: clone() 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: -1> 2024-04-22T06:51:06.858+0000 7f185fadc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f185fadc700 time 2024-04-22T06:51:06.858813+0000 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f18731173dd] 2024-04-22T06:51:07.208 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.209 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: clone() 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 0> 2024-04-22T06:51:06.860+0000 7f185fadc700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: in thread 7f185fadc700 thread_name:quiesce.agt 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1871e9fd20] 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: gsignal() 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: abort() 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1873117437] 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.210 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 13: clone() 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: -2> 2024-04-22T06:51:06.858+0000 7f185fadc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f185fadc700 time 2024-04-22T06:51:06.858813+0000 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f18731173dd] 2024-04-22T06:51:07.211 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: clone() 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: -1> 2024-04-22T06:51:06.860+0000 7f185fadc700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: in thread 7f185fadc700 thread_name:quiesce.agt 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1871e9fd20] 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2: gsignal() 2024-04-22T06:51:07.212 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 3: abort() 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1873117437] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f18731175a3] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558daef2ae9c] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558daef2af9f] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558daee590d3] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558daee5903c] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558daf0d8d3f] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558daf141560] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1871e951ca] 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 13: clone() 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66096]: 2024-04-22T06:51:07.213 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:06 smithi190 podman[66670]: 2024-04-22 06:51:06.922920568 +0000 UTC m=+0.025891614 container died 19d9ccf2235e3d19a446d9778c2312222b34721b5b375d3448eab0042d05d07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T06:51:07.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:07 smithi193 podman[67801]: 2024-04-22 06:51:07.634367939 +0000 UTC m=+0.740978714 container remove 60088dbe46cb57c31b342a2d18a2502038138bab5cf0b55ab7d906d7d630d1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T06:51:07.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:07 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:51:08.425 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:08 smithi190 podman[66670]: 2024-04-22 06:51:08.049009908 +0000 UTC m=+1.151980953 container remove 19d9ccf2235e3d19a446d9778c2312222b34721b5b375d3448eab0042d05d07c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T06:51:08.425 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:08 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:51:08.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:07 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-22T06:51:08.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:07 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Consumed 1.037s CPU time. 2024-04-22T06:51:08.702 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:08 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:51:08.703 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:08 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.083s CPU time. 2024-04-22T06:51:08.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:08 smithi079 ceph-mon[27946]: pgmap v212: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 68 MiB/s wr, 312 op/s 2024-04-22T06:51:09.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:08 smithi193 ceph-mon[31815]: pgmap v212: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 68 MiB/s wr, 312 op/s 2024-04-22T06:51:09.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:08 smithi190 ceph-mon[31421]: pgmap v212: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 68 MiB/s wr, 312 op/s 2024-04-22T06:51:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.487+0000 7efd21406640 1 -- 172.21.15.79:0/2764861268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110b20 msgr2=0x7efd1c109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.487+0000 7efd21406640 1 --2- 172.21.15.79:0/2764861268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110b20 0x7efd1c109490 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7efd10009930 tx=0x7efd1002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.486 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 -- 172.21.15.79:0/2764861268 shutdown_connections 2024-04-22T06:51:09.486 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 --2- 172.21.15.79:0/2764861268 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd1c11b730 0x7efd1c11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.486 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 --2- 172.21.15.79:0/2764861268 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110b20 0x7efd1c109490 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.486 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 --2- 172.21.15.79:0/2764861268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd1c110740 0x7efd1c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.487 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 -- 172.21.15.79:0/2764861268 >> 172.21.15.79:0/2764861268 conn(0x7efd1c06d020 msgr2=0x7efd1c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:09.487 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 -- 172.21.15.79:0/2764861268 shutdown_connections 2024-04-22T06:51:09.487 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.488+0000 7efd21406640 1 -- 172.21.15.79:0/2764861268 wait complete. 2024-04-22T06:51:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.489+0000 7efd21406640 1 Processor -- start 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 -- start start 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 0x7efd1c1a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd1c11b730 0x7efd1c1a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd1c1ac010 0x7efd1c1ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c121150 con 0x7efd1c110740 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c120fd0 con 0x7efd1c1ac010 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd21406640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1212d0 con 0x7efd1c11b730 2024-04-22T06:51:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.490+0000 7efd1affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 0x7efd1c1a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 0x7efd1c1a7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47684/0 (socket says 172.21.15.79:47684) 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 -- 172.21.15.79:0/2108754893 learned_addr learned my addr 172.21.15.79:0/2108754893 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 -- 172.21.15.79:0/2108754893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd1c11b730 msgr2=0x7efd1c1a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd1c11b730 0x7efd1c1a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 -- 172.21.15.79:0/2108754893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd1c1ac010 msgr2=0x7efd1c1ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd1c1ac010 0x7efd1c1ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 -- 172.21.15.79:0/2108754893 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1c1acb00 con 0x7efd1c110740 2024-04-22T06:51:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.491+0000 7efd1affd640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 0x7efd1c1a7750 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00b450 tx=0x7efd0c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.492+0000 7efd03fff640 1 -- 172.21.15.79:0/2108754893 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c015020 con 0x7efd1c110740 2024-04-22T06:51:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.492+0000 7efd03fff640 1 -- 172.21.15.79:0/2108754893 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd0c011d40 con 0x7efd1c110740 2024-04-22T06:51:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.492+0000 7efd03fff640 1 -- 172.21.15.79:0/2108754893 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c00f340 con 0x7efd1c110740 2024-04-22T06:51:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.492+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd1c1acdf0 con 0x7efd1c110740 2024-04-22T06:51:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.492+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd1c1b4590 con 0x7efd1c110740 2024-04-22T06:51:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.497+0000 7efd23ff4740 1 Processor -- start 2024-04-22T06:51:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.497+0000 7efd23ff4740 1 -- start start 2024-04-22T06:51:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.497+0000 7efd03fff640 1 -- 172.21.15.79:0/2108754893 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efd0c018020 con 0x7efd1c110740 2024-04-22T06:51:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.499+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.499+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 0x5649709b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.499+0000 7efd03fff640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efd040a3e90 0x7efd040a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.499+0000 7efd03fff640 1 -- 172.21.15.79:0/2108754893 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9245+0+0 (secure 0 0 0) 0x7efd0c0d03e0 con 0x7efd1c110740 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.499+0000 7efd1affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 0x5649709b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.499+0000 7efd1affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 0x5649709b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47694/0 (socket says 172.21.15.79:47694) 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.500+0000 7efd1b7fe640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efd040a3e90 0x7efd040a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.500+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.500+0000 7efd23ff4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5649709168d0 con 0x5649709b1b60 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.500+0000 7efd23ff4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5649708b3e80 con 0x5649709b28d0 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.500+0000 7efd23ff4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564970912890 con 0x5649709bb0e0 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.500+0000 7efd1affd640 1 -- 172.21.15.79:0/2073809308 learned_addr learned my addr 172.21.15.79:0/2073809308 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1b7fe640 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1affd640 1 -- 172.21.15.79:0/2073809308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 msgr2=0x5649709bd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1affd640 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1affd640 1 -- 172.21.15.79:0/2073809308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 msgr2=0x5649709b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1affd640 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1affd640 1 -- 172.21.15.79:0/2073809308 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649708d5320 con 0x5649709b1b60 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.501+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.502+0000 7efd1affd640 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 0x5649709b1fe0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0cda70 tx=0x7efd0c0a1e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44831b2364ca0dfd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.502+0000 7efd1b7fe640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efd040a3e90 0x7efd040a6350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efd1c10e1f0 tx=0x7efd140137b0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.502+0000 7efd017fa640 1 -- 172.21.15.79:0/2073809308 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0a0610 con 0x5649709b1b60 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.502+0000 7efd017fa640 1 -- 172.21.15.79:0/2073809308 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd0c0a1540 con 0x5649709b1b60 2024-04-22T06:51:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.502+0000 7efd017fa640 1 -- 172.21.15.79:0/2073809308 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0a3730 con 0x5649709b1b60 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 -- 172.21.15.79:0/2073809308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 msgr2=0x5649709b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 0x5649709b1fe0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7efd0c0cda70 tx=0x7efd0c0a1e60 comp rx=0 tx=0).stop 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 -- 172.21.15.79:0/2073809308 shutdown_connections 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b1b60 0x5649709b1fe0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 --2- 172.21.15.79:0/2073809308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.503+0000 7efd23ff4740 1 -- 172.21.15.79:0/2073809308 >> 172.21.15.79:0/2073809308 conn(0x564970978c90 msgr2=0x5649709ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:09.506 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 -- 172.21.15.79:0/2073809308 shutdown_connections 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 -- 172.21.15.79:0/2073809308 wait complete. 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 -- start start 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 Processor -- start 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b28d0 0x5649709b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 0x5649709c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5649709bde10 con 0x5649709bb0e0 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56497089eeb0 con 0x5649709b8970 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649709bdf90 con 0x5649709b28d0 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.504+0000 7efd23ff4740 10 client.0 _ll_register_callbacks cb 0x5649708cbb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47698/0 (socket says 172.21.15.79:47698) 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 -- 172.21.15.79:0/17865065 learned_addr learned my addr 172.21.15.79:0/17865065 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 -- 172.21.15.79:0/17865065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b28d0 msgr2=0x5649709b7ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b28d0 0x5649709b7ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 -- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 msgr2=0x5649709c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 0x5649709c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 -- 172.21.15.79:0/17865065 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564970629210 con 0x5649709bb0e0 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709b8430 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7efd10009850 tx=0x7efd1002daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349cdf52fba0c042 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.505+0000 7efcfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.506+0000 7efcfeff5640 1 -- 172.21.15.79:0/17865065 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10042020 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.506+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd0c0a1540 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.506+0000 7efcfeff5640 1 -- 172.21.15.79:0/17865065 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9245+0+0 (secure 0 0 0) 0x7efd10005250 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.506+0000 7efcfeff5640 1 -- 172.21.15.79:0/17865065 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7efd1004d050 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.506+0000 7efd23ff4740 10 client.17067 fetch_fsmap learned FSMap version 54 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.506+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5649706f5620 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efcfeff5640 1 -- 172.21.15.79:0/17865065 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd0c0a1540 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efcfeff5640 1 -- 172.21.15.79:0/17865065 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7efd100035f0 con 0x5649709bb0e0 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efd23ff4740 10 client.17067 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efd23ff4740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:51:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efd23ff4740 20 client.17067 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:09.510 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efd23ff4740 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709cb5f0 0x5649709cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.510 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efd23ff4740 4 client.17067 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:09.510 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.507+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56497092a880 con 0x5649709cb5f0 2024-04-22T06:51:09.511 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.508+0000 7efd1affd640 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709cb5f0 0x5649709cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.511 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.509+0000 7efd1affd640 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709cb5f0 0x5649709cda00 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d907e9c1c9025b9 server_cookie=95ddf2b6e55e332b in_seq=0 out_seq=0 2024-04-22T06:51:09.511 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.509+0000 7efcfeff5640 10 client.17067 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.513+0000 7efcfeff5640 1 -- 172.21.15.79:0/17865065 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56497092a880 con 0x5649709cb5f0 2024-04-22T06:51:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.513+0000 7efcfeff5640 10 client.17067 handle_command_reply: tid=0 2024-04-22T06:51:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.513+0000 7efd23ff4740 1 client.17067 shutdown 2024-04-22T06:51:09.512 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.514+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 msgr2=0x5649709b8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:09.513 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.514+0000 7efd23ff4740 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709b8430 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7efd10009850 tx=0x7efd1002daa0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.514 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 shutdown_connections 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709cb5f0 0x5649709cda00 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 0x5649709c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709b8430 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 --2- 172.21.15.79:0/17865065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b28d0 0x5649709b7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 >> 172.21.15.79:0/17865065 conn(0x564970978c90 msgr2=0x5649709ae5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efcfeff5640 0 client.17067 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 shutdown_connections 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 1 -- 172.21.15.79:0/17865065 wait complete. 2024-04-22T06:51:09.515 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.515+0000 7efd23ff4740 20 client.17067 trim_cache size 0 max 16384 2024-04-22T06:51:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.540+0000 7efd23ff4740 1 Processor -- start 2024-04-22T06:51:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.541+0000 7efd23ff4740 1 -- start start 2024-04-22T06:51:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.541+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.541+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b8970 0x5649709d3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.541+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.541+0000 7efd23ff4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1bfd50 con 0x5649709bb0e0 2024-04-22T06:51:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.541+0000 7efd23ff4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1bd620 con 0x5649709b28d0 2024-04-22T06:51:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.542+0000 7efd23ff4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1bb490 con 0x5649709b8970 2024-04-22T06:51:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.542+0000 7efd1b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.542+0000 7efd1b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47712/0 (socket says 172.21.15.79:47712) 2024-04-22T06:51:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.542+0000 7efd1b7fe640 1 -- 172.21.15.79:0/3288514090 learned_addr learned my addr 172.21.15.79:0/3288514090 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.542+0000 7efd1affd640 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.542+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b8970 0x5649709d3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.543+0000 7efd1b7fe640 1 -- 172.21.15.79:0/3288514090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b8970 msgr2=0x5649709d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.543+0000 7efd1b7fe640 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b8970 0x5649709d3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.543+0000 7efd1b7fe640 1 -- 172.21.15.79:0/3288514090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 msgr2=0x5649709d3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.543+0000 7efd1b7fe640 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709d3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.543+0000 7efd1b7fe640 1 -- 172.21.15.79:0/3288514090 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649709a7f30 con 0x5649709bb0e0 2024-04-22T06:51:09.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.544+0000 7efd1b7fe640 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709c6d10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7efd14001e20 tx=0x7efd14016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e537de55d902bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:09.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.544+0000 7efd00ff9640 1 -- 172.21.15.79:0/3288514090 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14027070 con 0x5649709bb0e0 2024-04-22T06:51:09.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.544+0000 7efd00ff9640 1 -- 172.21.15.79:0/3288514090 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd14007840 con 0x5649709bb0e0 2024-04-22T06:51:09.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.544+0000 7efd00ff9640 1 -- 172.21.15.79:0/3288514090 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14022410 con 0x5649709bb0e0 2024-04-22T06:51:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.547+0000 7efd23ff4740 1 -- 172.21.15.79:0/3288514090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 msgr2=0x5649709c6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.547+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709c6d10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7efd14001e20 tx=0x7efd14016000 comp rx=0 tx=0).stop 2024-04-22T06:51:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 -- 172.21.15.79:0/3288514090 shutdown_connections 2024-04-22T06:51:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709bb0e0 0x5649709c6d10 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709b8970 0x5649709d3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3288514090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b28d0 0x5649709d3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 -- 172.21.15.79:0/3288514090 >> 172.21.15.79:0/3288514090 conn(0x564970978c90 msgr2=0x5649709b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 -- 172.21.15.79:0/3288514090 shutdown_connections 2024-04-22T06:51:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.548+0000 7efd23ff4740 1 -- 172.21.15.79:0/3288514090 wait complete. 2024-04-22T06:51:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.549+0000 7efd23ff4740 1 -- start start 2024-04-22T06:51:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.550+0000 7efd23ff4740 1 Processor -- start 2024-04-22T06:51:09.550 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.552+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 0x5649709bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.552+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 0x5649709bba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.553+0000 7efd1affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 0x5649709bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.553+0000 7efd1affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 0x5649709bb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47716/0 (socket says 172.21.15.79:47716) 2024-04-22T06:51:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.554+0000 7efd23ff4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.554+0000 7efd23ff4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1bbae0 con 0x5649709b28d0 2024-04-22T06:51:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.554+0000 7efd23ff4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1bcf50 con 0x5649709b8970 2024-04-22T06:51:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.554+0000 7efd23ff4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c1bc1b0 con 0x5649709bb0e0 2024-04-22T06:51:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.554+0000 7efd23ff4740 10 client.0 _ll_register_callbacks cb 0x7efd10009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.554+0000 7efd1affd640 1 -- 172.21.15.79:0/3630560099 learned_addr learned my addr 172.21.15.79:0/3630560099 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.555+0000 7efd1affd640 1 -- 172.21.15.79:0/3630560099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 msgr2=0x5649709bbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.555+0000 7efd1affd640 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.555+0000 7efd1affd640 1 -- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 msgr2=0x5649709bba00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:51:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.556+0000 7efd1affd640 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 0x5649709bba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.556+0000 7efd1affd640 1 -- 172.21.15.79:0/3630560099 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649709c2220 con 0x5649709b28d0 2024-04-22T06:51:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.557+0000 7efd1affd640 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 0x5649709bb4c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7efd0c09c1a0 tx=0x7efd0c098500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1cf200e6384b01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.558+0000 7efcfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.560+0000 7efcfeff5640 1 -- 172.21.15.79:0/3630560099 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c09c260 con 0x5649709b28d0 2024-04-22T06:51:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.560+0000 7efcfeff5640 1 -- 172.21.15.79:0/3630560099 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9245+0+0 (secure 0 0 0) 0x7efd0c09a560 con 0x5649709b28d0 2024-04-22T06:51:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.560+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd14007840 con 0x5649709b28d0 2024-04-22T06:51:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.561+0000 7efcfeff5640 1 -- 172.21.15.79:0/3630560099 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7efd0c0a0610 con 0x5649709b28d0 2024-04-22T06:51:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.561+0000 7efd23ff4740 10 client.17073 fetch_fsmap learned FSMap version 54 2024-04-22T06:51:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.562+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5649709c3510 con 0x5649709b28d0 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.562+0000 7efcfeff5640 1 -- 172.21.15.79:0/3630560099 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd14007840 con 0x5649709b28d0 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efcfeff5640 1 -- 172.21.15.79:0/3630560099 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 54) ==== 8338+0+0 (secure 0 0 0) 0x7efd0c0988d0 con 0x5649709b28d0 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efd23ff4740 10 client.17073 fetch_fsmap finished waiting for FSMap version 54 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efd23ff4740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efd23ff4740 20 client.17073 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709c37a0 0x564970a33b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efd23ff4740 4 client.17073 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:09.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.563+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649708b63d0 con 0x5649709c37a0 2024-04-22T06:51:09.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.565+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709c37a0 0x564970a33b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.571+0000 7efd1a7fc640 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709c37a0 0x564970a33b90 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd2e505d6289384 server_cookie=cc711d40faaf20a in_seq=0 out_seq=0 2024-04-22T06:51:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:09.572+0000 7efcfeff5640 10 client.17073 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:10.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:09 smithi193 ceph-mon[31815]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:51:10.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:09 smithi190 ceph-mon[31421]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:51:10.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:09 smithi079 ceph-mon[27946]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:51:11.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:10 smithi079 ceph-mon[27946]: pgmap v214: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 89 MiB/s wr, 615 op/s 2024-04-22T06:51:11.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:10 smithi193 ceph-mon[31815]: pgmap v214: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 89 MiB/s wr, 615 op/s 2024-04-22T06:51:11.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:10 smithi190 ceph-mon[31421]: pgmap v214: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 89 MiB/s wr, 615 op/s 2024-04-22T06:51:13.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:12 smithi193 ceph-mon[31815]: pgmap v215: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 257 MiB/s wr, 1.57k op/s 2024-04-22T06:51:13.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:12 smithi190 ceph-mon[31421]: pgmap v215: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 257 MiB/s wr, 1.57k op/s 2024-04-22T06:51:13.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:12 smithi079 ceph-mon[27946]: pgmap v215: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 257 MiB/s wr, 1.57k op/s 2024-04-22T06:51:15.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:14 smithi193 ceph-mon[31815]: pgmap v216: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 368 MiB/s wr, 2.28k op/s 2024-04-22T06:51:15.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:14 smithi190 ceph-mon[31421]: pgmap v216: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 368 MiB/s wr, 2.28k op/s 2024-04-22T06:51:15.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:14 smithi079 ceph-mon[27946]: pgmap v216: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 368 MiB/s wr, 2.28k op/s 2024-04-22T06:51:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efcfeff5640 1 -- 172.21.15.79:0/3630560099 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649708b63d0 con 0x5649709c37a0 2024-04-22T06:51:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efcfeff5640 10 client.17073 handle_command_reply: tid=0 2024-04-22T06:51:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 client.17073 shutdown 2024-04-22T06:51:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 msgr2=0x5649709bb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 0x5649709bb4c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7efd0c09c1a0 tx=0x7efd0c098500 comp rx=0 tx=0).stop 2024-04-22T06:51:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 shutdown_connections 2024-04-22T06:51:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5649709c37a0 0x564970a33b90 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5649709b8970 0x5649709bba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649709bb0e0 0x5649709bbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 --2- 172.21.15.79:0/3630560099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5649709b28d0 0x5649709bb4c0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.824+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 >> 172.21.15.79:0/3630560099 conn(0x564970978c90 msgr2=0x5649709adc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.825+0000 7efcfeff5640 0 client.17073 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.825+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 shutdown_connections 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.825+0000 7efd23ff4740 1 -- 172.21.15.79:0/3630560099 wait complete. 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.825+0000 7efd23ff4740 20 client.17073 trim_cache size 0 max 16384 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:51:15.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:51:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efd040a3e90 msgr2=0x7efd040a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efd040a3e90 0x7efd040a6350 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efd1c10e1f0 tx=0x7efd140137b0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 msgr2=0x7efd1c1a7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 0x7efd1c1a7750 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00b450 tx=0x7efd0c00b910 comp rx=0 tx=0).stop 2024-04-22T06:51:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 shutdown_connections 2024-04-22T06:51:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7efd040a3e90 0x7efd040a6350 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd1c1ac010 0x7efd1c1ac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd1c11b730 0x7efd1c1a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 --2- 172.21.15.79:0/2108754893 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efd1c110740 0x7efd1c1a7750 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 >> 172.21.15.79:0/2108754893 conn(0x7efd1c06d020 msgr2=0x7efd1c10e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.829+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 shutdown_connections 2024-04-22T06:51:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:15.830+0000 7efd21406640 1 -- 172.21.15.79:0/2108754893 wait complete. 2024-04-22T06:51:17.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:16 smithi193 ceph-mon[31815]: pgmap v217: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 415 MiB/s wr, 2.53k op/s 2024-04-22T06:51:17.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:16 smithi190 ceph-mon[31421]: pgmap v217: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 415 MiB/s wr, 2.53k op/s 2024-04-22T06:51:17.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:16 smithi079 ceph-mon[27946]: pgmap v217: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 415 MiB/s wr, 2.53k op/s 2024-04-22T06:51:18.475 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:18 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:51:18.475 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:18 smithi193 systemd[1]: Stopped Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:18.475 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:18 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Consumed 1.037s CPU time. 2024-04-22T06:51:18.475 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:18 smithi193 systemd[1]: Starting Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:51:18.782 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:18 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:51:18.782 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:18 smithi190 systemd[1]: Stopped Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:18.782 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:18 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.083s CPU time. 2024-04-22T06:51:18.782 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:18 smithi190 systemd[1]: Starting Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:51:18.804 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:18 smithi193 podman[67903]: 2024-04-22 06:51:18.381517833 +0000 UTC m=+0.034716391 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:51:19.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:18 smithi193 ceph-mon[31815]: pgmap v218: 97 pgs: 97 active+clean; 53 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 619 MiB/s wr, 3.26k op/s 2024-04-22T06:51:19.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:18 smithi190 ceph-mon[31421]: pgmap v218: 97 pgs: 97 active+clean; 53 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 619 MiB/s wr, 3.26k op/s 2024-04-22T06:51:19.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:18 smithi190 podman[66756]: 2024-04-22 06:51:18.690678856 +0000 UTC m=+0.033864656 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:51:19.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:18 smithi079 ceph-mon[27946]: pgmap v218: 97 pgs: 97 active+clean; 53 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 619 MiB/s wr, 3.26k op/s 2024-04-22T06:51:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: Replacing daemon mds.i as rank 2 with standby daemon mds.e 2024-04-22T06:51:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: Replacing daemon mds.h as rank 1 with standby daemon mds.f 2024-04-22T06:51:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:20.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:51:20.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:51:20.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:19 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=e=up:replay} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:20.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: Replacing daemon mds.i as rank 2 with standby daemon mds.e 2024-04-22T06:51:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: Replacing daemon mds.h as rank 1 with standby daemon mds.f 2024-04-22T06:51:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:51:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:51:20.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:19 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=e=up:replay} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: Replacing daemon mds.i as rank 2 with standby daemon mds.e 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: Replacing daemon mds.h as rank 1 with standby daemon mds.f 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:51:20.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:19 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=e=up:replay} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:20.824 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:20 smithi193 podman[67903]: 2024-04-22 06:51:20.519190742 +0000 UTC m=+2.172389298 container create 93d7c27cf50a56f622538e7240da7cb55ccf431f5d8d62a537073a02e541ac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:51:20.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:20 smithi190 ceph-mon[31421]: pgmap v219: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 686 MiB/s wr, 3.42k op/s 2024-04-22T06:51:20.953 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:20 smithi190 podman[66756]: 2024-04-22 06:51:20.626497912 +0000 UTC m=+1.969683705 container create 74377163fcad3dc5106d81795dd503e663692d2f838215a96ecd317381a1ad8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T06:51:21.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:20 smithi193 ceph-mon[31815]: pgmap v219: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 686 MiB/s wr, 3.42k op/s 2024-04-22T06:51:21.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:20 smithi079 ceph-mon[27946]: pgmap v219: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 686 MiB/s wr, 3.42k op/s 2024-04-22T06:51:22.164 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:21 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:resolve 2024-04-22T06:51:22.164 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:21 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:resolve 2024-04-22T06:51:22.164 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:21 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=e=up:resolve} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:22.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:21 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:resolve 2024-04-22T06:51:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:resolve 2024-04-22T06:51:22.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=e=up:resolve} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:21 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:resolve 2024-04-22T06:51:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:21 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:resolve 2024-04-22T06:51:22.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:21 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=e=up:resolve} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:23.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: pgmap v221: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 686 MiB/s wr, 3.46k op/s 2024-04-22T06:51:23.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:reconnect 2024-04-22T06:51:23.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:reconnect 2024-04-22T06:51:23.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=e=up:reconnect} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:23.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499986 2024-04-22T06:51:23.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00899986 2024-04-22T06:51:23.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0149996 2024-04-22T06:51:23.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:22 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0159997 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: pgmap v221: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 686 MiB/s wr, 3.46k op/s 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:reconnect 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:reconnect 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=e=up:reconnect} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499986 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00899986 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0149996 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0159997 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:22 smithi190 podman[66756]: 2024-04-22 06:51:22.813768115 +0000 UTC m=+4.156953919 container init 74377163fcad3dc5106d81795dd503e663692d2f838215a96ecd317381a1ad8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T06:51:23.203 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:22 smithi190 podman[66756]: 2024-04-22 06:51:22.822931175 +0000 UTC m=+4.166116971 container start 74377163fcad3dc5106d81795dd503e663692d2f838215a96ecd317381a1ad8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:51:23.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: starting mds.h at 2024-04-22T06:51:23.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:22 smithi190 bash[66756]: 74377163fcad3dc5106d81795dd503e663692d2f838215a96ecd317381a1ad8b 2024-04-22T06:51:23.204 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:51:22 smithi190 systemd[1]: Started Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:23.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: pgmap v221: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 686 MiB/s wr, 3.46k op/s 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:reconnect 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:reconnect 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=e=up:reconnect} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00499986 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.00899986 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0149996 2024-04-22T06:51:23.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:22 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.0159997 2024-04-22T06:51:23.898 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:23 smithi193 podman[67903]: 2024-04-22 06:51:23.633824656 +0000 UTC m=+5.287023222 container init 93d7c27cf50a56f622538e7240da7cb55ccf431f5d8d62a537073a02e541ac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:51:23.899 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:23 smithi193 podman[67903]: 2024-04-22 06:51:23.640033624 +0000 UTC m=+5.293232187 container start 93d7c27cf50a56f622538e7240da7cb55ccf431f5d8d62a537073a02e541ac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T06:51:23.899 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: starting mds.i at 2024-04-22T06:51:24.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:23 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:rejoin 2024-04-22T06:51:24.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:23 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:rejoin 2024-04-22T06:51:24.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:23 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=e=up:rejoin} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:24.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:23 smithi193 ceph-mon[31815]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-22T06:51:24.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:23 smithi193 ceph-mon[31815]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-22T06:51:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:23 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:rejoin 2024-04-22T06:51:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:23 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:rejoin 2024-04-22T06:51:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:23 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=e=up:rejoin} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:23 smithi190 ceph-mon[31421]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-22T06:51:24.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:23 smithi190 ceph-mon[31421]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-22T06:51:24.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:23 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:rejoin 2024-04-22T06:51:24.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:23 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:rejoin 2024-04-22T06:51:24.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:23 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=e=up:rejoin} 1 up:standby-replay 6 up:standby 2024-04-22T06:51:24.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:23 smithi079 ceph-mon[27946]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-22T06:51:24.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:23 smithi079 ceph-mon[27946]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-22T06:51:25.181 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: pgmap v222: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 690 MiB/s wr, 3.25k op/s 2024-04-22T06:51:25.181 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:51:25.181 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:51:25.181 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:boot 2024-04-22T06:51:25.181 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:active 2024-04-22T06:51:25.181 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:active 2024-04-22T06:51:25.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:boot 2024-04-22T06:51:25.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:25.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:51:25.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:51:25.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:51:25.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:24 smithi193 ceph-mon[31815]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: pgmap v222: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 690 MiB/s wr, 3.25k op/s 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:boot 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:active 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:active 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:boot 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:51:25.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:24 smithi190 ceph-mon[31421]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: pgmap v222: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 690 MiB/s wr, 3.25k op/s 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:boot 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1167513219,v1:172.21.15.190:6835/1167513219] up:active 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/99277862,v1:172.21.15.193:6835/99277862] up:active 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:boot 2024-04-22T06:51:25.244 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:25.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:51:25.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:51:25.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:51:25.245 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:24 smithi079 ceph-mon[27946]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:51:25.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:25 smithi193 bash[67903]: 93d7c27cf50a56f622538e7240da7cb55ccf431f5d8d62a537073a02e541ac3b 2024-04-22T06:51:25.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:51:25 smithi193 systemd[1]: Started Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:27.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:26 smithi193 ceph-mon[31815]: pgmap v224: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 569 MiB/s wr, 2.67k op/s 2024-04-22T06:51:27.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:26 smithi190 ceph-mon[31421]: pgmap v224: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 569 MiB/s wr, 2.67k op/s 2024-04-22T06:51:27.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:26 smithi079 ceph-mon[27946]: pgmap v224: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 569 MiB/s wr, 2.67k op/s 2024-04-22T06:51:29.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:28 smithi193 ceph-mon[31815]: pgmap v225: 97 pgs: 97 active+clean; 58 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 719 MiB/s wr, 2.87k op/s 2024-04-22T06:51:29.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:28 smithi190 ceph-mon[31421]: pgmap v225: 97 pgs: 97 active+clean; 58 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 719 MiB/s wr, 2.87k op/s 2024-04-22T06:51:29.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:28 smithi079 ceph-mon[27946]: pgmap v225: 97 pgs: 97 active+clean; 58 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 719 MiB/s wr, 2.87k op/s 2024-04-22T06:51:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 -- 172.21.15.79:0/3150513016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14201070a0 msgr2=0x7f142010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 --2- 172.21.15.79:0/3150513016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14201070a0 0x7f142010ec70 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1418009ca0 tx=0x7f141802c120 comp rx=0 tx=0).stop 2024-04-22T06:51:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 -- 172.21.15.79:0/3150513016 shutdown_connections 2024-04-22T06:51:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 --2- 172.21.15.79:0/3150513016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14201070a0 0x7f142010ec70 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 --2- 172.21.15.79:0/3150513016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1420102980 0x7f14201069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 --2- 172.21.15.79:0/3150513016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1420101fd0 0x7f14201023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.564+0000 7f141f59e640 1 -- 172.21.15.79:0/3150513016 >> 172.21.15.79:0/3150513016 conn(0x7f14200fd890 msgr2=0x7f14200ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.566+0000 7f141f59e640 1 -- 172.21.15.79:0/3150513016 shutdown_connections 2024-04-22T06:51:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.566+0000 7f141f59e640 1 -- 172.21.15.79:0/3150513016 wait complete. 2024-04-22T06:51:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.567+0000 7f141f59e640 1 Processor -- start 2024-04-22T06:51:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 -- start start 2024-04-22T06:51:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 0x7f142019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14201070a0 0x7f142019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142019ef60 0x7f142019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1420075230 con 0x7f1420101fd0 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f14200750b0 con 0x7f142019ef60 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141f59e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14200753b0 con 0x7f14201070a0 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 0x7f142019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 0x7f142019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54758/0 (socket says 172.21.15.79:54758) 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141e59c640 1 -- 172.21.15.79:0/2100824535 learned_addr learned my addr 172.21.15.79:0/2100824535 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.568+0000 7f141e59c640 1 -- 172.21.15.79:0/2100824535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14201070a0 msgr2=0x7f142019ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f141e59c640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14201070a0 0x7f142019ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f141e59c640 1 -- 172.21.15.79:0/2100824535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142019ef60 msgr2=0x7f142019f410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f141e59c640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142019ef60 0x7f142019f410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f141e59c640 1 -- 172.21.15.79:0/2100824535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142019fa50 con 0x7f1420101fd0 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f141e59c640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 0x7f142019a6f0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f141000b4a0 tx=0x7f141000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f140f7fe640 1 -- 172.21.15.79:0/2100824535 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410014070 con 0x7f1420101fd0 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f140f7fe640 1 -- 172.21.15.79:0/2100824535 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1410010ac0 con 0x7f1420101fd0 2024-04-22T06:51:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f140f7fe640 1 -- 172.21.15.79:0/2100824535 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410017c30 con 0x7f1420101fd0 2024-04-22T06:51:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.569+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142019fd40 con 0x7f1420101fd0 2024-04-22T06:51:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.570+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14201a7530 con 0x7f1420101fd0 2024-04-22T06:51:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.574+0000 7f14277f0740 1 Processor -- start 2024-04-22T06:51:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.575+0000 7f14277f0740 1 -- start start 2024-04-22T06:51:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f14277f0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e893810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f14277f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e893d50 0x55e83e8941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f141ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e893810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f141ed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e893810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54774/0 (socket says 172.21.15.79:54774) 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f141ed9d640 1 -- 172.21.15.79:0/1763144405 learned_addr learned my addr 172.21.15.79:0/1763144405 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f14277f0740 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 0x55e83e89f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e83e7f88d0 con 0x55e83e894ac0 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e83e795e80 con 0x55e83e89d2d0 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.576+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e83e7f4890 con 0x55e83e893d50 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.577+0000 7f141ed9d640 1 -- 172.21.15.79:0/1763144405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e893d50 msgr2=0x55e83e8941d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.577+0000 7f141ed9d640 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e893d50 0x55e83e8941d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.577+0000 7f141ed9d640 1 -- 172.21.15.79:0/1763144405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 msgr2=0x55e83e89f6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.577+0000 7f141ed9d640 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 0x55e83e89f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.577+0000 7f141ed9d640 1 -- 172.21.15.79:0/1763144405 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e83e7b7320 con 0x55e83e894ac0 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.578+0000 7f141ed9d640 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e893810 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1418037040 tx=0x7f14180323d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10df69a533a44ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.578+0000 7f140cff9640 1 -- 172.21.15.79:0/1763144405 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418041020 con 0x55e83e894ac0 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.578+0000 7f140cff9640 1 -- 172.21.15.79:0/1763144405 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14180098e0 con 0x55e83e894ac0 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.579+0000 7f140f7fe640 1 -- 172.21.15.79:0/2100824535 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1410017070 con 0x7f1420101fd0 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.579+0000 7f140cff9640 1 -- 172.21.15.79:0/1763144405 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418039040 con 0x55e83e894ac0 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.579+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 msgr2=0x55e83e893810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:29.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.579+0000 7f14277f0740 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e893810 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1418037040 tx=0x7f14180323d0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 shutdown_connections 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 0x55e83e89f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e893d50 0x55e83e8941d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 --2- 172.21.15.79:0/1763144405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e893810 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f140f7fe640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14080a3e60 0x7f14080a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 >> 172.21.15.79:0/1763144405 conn(0x55e83e85ac90 msgr2=0x55e83e88ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f140f7fe640 1 -- 172.21.15.79:0/2100824535 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7f14100cf710 con 0x7f1420101fd0 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f141ed9d640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14080a3e60 0x7f14080a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 shutdown_connections 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.580+0000 7f14277f0740 1 -- 172.21.15.79:0/1763144405 wait complete. 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 -- start start 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 Processor -- start 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e893d50 0x55e83e89cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e899fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e83e8a0040 con 0x55e83e894ac0 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e83e780eb0 con 0x55e83e893d50 2024-04-22T06:51:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.581+0000 7f14277f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e83e8a01c0 con 0x55e83e89d2d0 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.582+0000 7f14277f0740 10 client.0 _ll_register_callbacks cb 0x55e83e7adb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.582+0000 7f141dd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e899fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.582+0000 7f141dd9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e899fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54782/0 (socket says 172.21.15.79:54782) 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.582+0000 7f141dd9b640 1 -- 172.21.15.79:0/4090390736 learned_addr learned my addr 172.21.15.79:0/4090390736 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.582+0000 7f141ed9d640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14080a3e60 0x7f14080a6320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1418009ca0 tx=0x7f1418005d80 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.582+0000 7f141e59c640 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e893d50 0x55e83e89cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f141dd9b640 1 -- 172.21.15.79:0/4090390736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 msgr2=0x55e83e89a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f141dd9b640 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f141dd9b640 1 -- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e893d50 msgr2=0x55e83e89cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f141dd9b640 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e893d50 0x55e83e89cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f141dd9b640 1 -- 172.21.15.79:0/4090390736 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e83e50b210 con 0x55e83e894ac0 2024-04-22T06:51:29.585 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f141dd9b640 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e899fe0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1414009d50 tx=0x7f141400b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1b2bc93b443c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:29.585 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.583+0000 7f1405ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:29.585 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.584+0000 7f1405ffb640 1 -- 172.21.15.79:0/4090390736 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414011420 con 0x55e83e894ac0 2024-04-22T06:51:29.585 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.584+0000 7f1405ffb640 1 -- 172.21.15.79:0/4090390736 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7f14140115c0 con 0x55e83e894ac0 2024-04-22T06:51:29.585 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.584+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14180098e0 con 0x55e83e894ac0 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.584+0000 7f1405ffb640 1 -- 172.21.15.79:0/4090390736 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f141401f050 con 0x55e83e894ac0 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.584+0000 7f14277f0740 10 client.17085 fetch_fsmap learned FSMap version 60 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.584+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55e83e5d7620 con 0x55e83e894ac0 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f1405ffb640 1 -- 172.21.15.79:0/4090390736 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14180098e0 con 0x55e83e894ac0 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f1405ffb640 1 -- 172.21.15.79:0/4090390736 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 60) ==== 8338+0+0 (secure 0 0 0) 0x7f141401a8f0 con 0x55e83e894ac0 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f14277f0740 10 client.17085 fetch_fsmap finished waiting for FSMap version 60 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f14277f0740 10 client.17085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f14277f0740 20 client.17085 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f14277f0740 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e8b5bf0 0x55e83e8b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f14277f0740 4 client.17085 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.585+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e83e80c880 con 0x55e83e8b5bf0 2024-04-22T06:51:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.587+0000 7f141e59c640 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e8b5bf0 0x55e83e8b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.589+0000 7f141e59c640 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e8b5bf0 0x55e83e8b8000 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f7dfe8253aa243e server_cookie=ab3c98d981bf808 in_seq=0 out_seq=0 2024-04-22T06:51:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.589+0000 7f1405ffb640 10 client.17085 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.592+0000 7f1405ffb640 1 -- 172.21.15.79:0/4090390736 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e83e80c880 con 0x55e83e8b5bf0 2024-04-22T06:51:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.592+0000 7f1405ffb640 10 client.17085 handle_command_reply: tid=0 2024-04-22T06:51:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.592+0000 7f14277f0740 1 client.17085 shutdown 2024-04-22T06:51:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.593+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 msgr2=0x55e83e899fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.593+0000 7f14277f0740 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e899fe0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1414009d50 tx=0x7f141400b4b0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.595+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 shutdown_connections 2024-04-22T06:51:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.595+0000 7f14277f0740 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e8b5bf0 0x55e83e8b8000 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.595+0000 7f14277f0740 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.595+0000 7f14277f0740 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e894ac0 0x55e83e899fe0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.595+0000 7f14277f0740 1 --2- 172.21.15.79:0/4090390736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e893d50 0x55e83e89cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.595+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 >> 172.21.15.79:0/4090390736 conn(0x55e83e85ac90 msgr2=0x55e83e890790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.596+0000 7f1405ffb640 0 client.17085 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.598+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 shutdown_connections 2024-04-22T06:51:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.599+0000 7f14277f0740 1 -- 172.21.15.79:0/4090390736 wait complete. 2024-04-22T06:51:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.600+0000 7f14277f0740 20 client.17085 trim_cache size 0 max 16384 2024-04-22T06:51:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.628+0000 7f14277f0740 1 Processor -- start 2024-04-22T06:51:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.628+0000 7f14277f0740 1 -- start start 2024-04-22T06:51:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.628+0000 7f14277f0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e8a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f14277f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e894ac0 0x55e83e8b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f14277f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f14277f0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f14201ae7c0 con 0x55e83e893d50 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f141e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e8a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f14277f0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f14201ac090 con 0x55e83e894ac0 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f14277f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14201a9f40 con 0x55e83e89d2d0 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f141e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e8a8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54784/0 (socket says 172.21.15.79:54784) 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.629+0000 7f141e59c640 1 -- 172.21.15.79:0/4056970163 learned_addr learned my addr 172.21.15.79:0/4056970163 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141ed9d640 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141e59c640 1 -- 172.21.15.79:0/4056970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 msgr2=0x55e83e89c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141e59c640 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141e59c640 1 -- 172.21.15.79:0/4056970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e894ac0 msgr2=0x55e83e8b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141e59c640 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e894ac0 0x55e83e8b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141e59c640 1 -- 172.21.15.79:0/4056970163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e83e8b4790 con 0x55e83e893d50 2024-04-22T06:51:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.630+0000 7f141ed9d640 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.631+0000 7f141e59c640 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e8a8f70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f141009b400 tx=0x7f14100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7afd523f2fd8dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:29.632 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.631+0000 7f1407fff640 1 -- 172.21.15.79:0/4056970163 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141001e040 con 0x55e83e893d50 2024-04-22T06:51:29.632 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.631+0000 7f1407fff640 1 -- 172.21.15.79:0/4056970163 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1410099510 con 0x55e83e893d50 2024-04-22T06:51:29.632 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.631+0000 7f1407fff640 1 -- 172.21.15.79:0/4056970163 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14100ccf20 con 0x55e83e893d50 2024-04-22T06:51:29.632 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.633+0000 7f14277f0740 1 -- 172.21.15.79:0/4056970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 msgr2=0x55e83e8a8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:29.632 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.633+0000 7f14277f0740 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e8a8f70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f141009b400 tx=0x7f14100046c0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.634+0000 7f14277f0740 1 -- 172.21.15.79:0/4056970163 shutdown_connections 2024-04-22T06:51:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.634+0000 7f14277f0740 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e894ac0 0x55e83e8b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.635+0000 7f14277f0740 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e89d2d0 0x55e83e89c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.635+0000 7f14277f0740 1 --2- 172.21.15.79:0/4056970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e8a8f70 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.635+0000 7f14277f0740 1 -- 172.21.15.79:0/4056970163 >> 172.21.15.79:0/4056970163 conn(0x55e83e85ac90 msgr2=0x55e83e8988f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.635+0000 7f14277f0740 1 -- 172.21.15.79:0/4056970163 shutdown_connections 2024-04-22T06:51:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.635+0000 7f14277f0740 1 -- 172.21.15.79:0/4056970163 wait complete. 2024-04-22T06:51:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.636+0000 7f14277f0740 1 -- start start 2024-04-22T06:51:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.637+0000 7f14277f0740 1 Processor -- start 2024-04-22T06:51:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.637+0000 7f14277f0740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e896c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.637+0000 7f14277f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e894ac0 0x55e83e897190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f14277f0740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 0x55e83e8976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f14277f0740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f14201aa590 con 0x55e83e893d50 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f14277f0740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f14201ab9c0 con 0x55e83e89d2d0 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f14277f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14201aac20 con 0x55e83e894ac0 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f141e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e896c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f14277f0740 10 client.0 _ll_register_callbacks cb 0x7f1414006e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f141e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e896c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54792/0 (socket says 172.21.15.79:54792) 2024-04-22T06:51:29.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.638+0000 7f141e59c640 1 -- 172.21.15.79:0/1750567765 learned_addr learned my addr 172.21.15.79:0/1750567765 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:29.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.639+0000 7f141e59c640 1 -- 172.21.15.79:0/1750567765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e894ac0 msgr2=0x55e83e897190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:29.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.639+0000 7f141e59c640 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e894ac0 0x55e83e897190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.639+0000 7f141e59c640 1 -- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 msgr2=0x55e83e8976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:29.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.639+0000 7f141e59c640 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 0x55e83e8976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:29.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.639+0000 7f141e59c640 1 -- 172.21.15.79:0/1750567765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e83e92c190 con 0x55e83e893d50 2024-04-22T06:51:29.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.639+0000 7f141e59c640 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e896c50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f141009b170 tx=0x7f14100092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ad5af0adaa217 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:29.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.640+0000 7f1405ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:29.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.640+0000 7f1405ffb640 1 -- 172.21.15.79:0/1750567765 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14100ae070 con 0x55e83e893d50 2024-04-22T06:51:29.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.640+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1410099510 con 0x55e83e893d50 2024-04-22T06:51:29.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.641+0000 7f1405ffb640 1 -- 172.21.15.79:0/1750567765 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7f1410097ae0 con 0x55e83e893d50 2024-04-22T06:51:29.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.641+0000 7f1405ffb640 1 -- 172.21.15.79:0/1750567765 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f14100ccf00 con 0x55e83e893d50 2024-04-22T06:51:29.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.642+0000 7f14277f0740 10 client.17091 fetch_fsmap learned FSMap version 60 2024-04-22T06:51:29.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.642+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55e83e92d370 con 0x55e83e893d50 2024-04-22T06:51:29.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f1405ffb640 1 -- 172.21.15.79:0/1750567765 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1410099510 con 0x55e83e893d50 2024-04-22T06:51:29.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f1405ffb640 1 -- 172.21.15.79:0/1750567765 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 60) ==== 8338+0+0 (secure 0 0 0) 0x7f141009a8c0 con 0x55e83e893d50 2024-04-22T06:51:29.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f14277f0740 10 client.17091 fetch_fsmap finished waiting for FSMap version 60 2024-04-22T06:51:29.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f14277f0740 10 client.17091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:51:29.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f14277f0740 20 client.17091 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:29.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f14277f0740 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e92da30 0x55e83e92fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:29.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f14277f0740 4 client.17091 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:29.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.643+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e83e7983d0 con 0x55e83e92da30 2024-04-22T06:51:29.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.654+0000 7f141dd9b640 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e92da30 0x55e83e92fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.661+0000 7f141dd9b640 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e92da30 0x55e83e92fe90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c81fb9da1549a4c8 server_cookie=fa6973fed8dad7b5 in_seq=0 out_seq=0 2024-04-22T06:51:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:29.661+0000 7f1405ffb640 10 client.17091 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:31.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:30 smithi193 ceph-mon[31815]: pgmap v226: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 744 MiB/s wr, 2.80k op/s 2024-04-22T06:51:31.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:30 smithi190 ceph-mon[31421]: pgmap v226: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 744 MiB/s wr, 2.80k op/s 2024-04-22T06:51:31.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:30 smithi079 ceph-mon[27946]: pgmap v226: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 744 MiB/s wr, 2.80k op/s 2024-04-22T06:51:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.105+0000 7f1405ffb640 1 -- 172.21.15.79:0/1750567765 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e83e7983d0 con 0x55e83e92da30 2024-04-22T06:51:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f1405ffb640 10 client.17091 handle_command_reply: tid=0 2024-04-22T06:51:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 client.17091 shutdown 2024-04-22T06:51:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 msgr2=0x55e83e896c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e896c50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f141009b170 tx=0x7f14100092b0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 shutdown_connections 2024-04-22T06:51:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e83e92da30 0x55e83e92fe90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e83e89d2d0 0x55e83e8976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e83e894ac0 0x55e83e897190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 --2- 172.21.15.79:0/1750567765 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e83e893d50 0x55e83e896c50 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 >> 172.21.15.79:0/1750567765 conn(0x55e83e85ac90 msgr2=0x55e83e898d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.106+0000 7f1405ffb640 0 client.17091 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.107+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 shutdown_connections 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.107+0000 7f14277f0740 1 -- 172.21.15.79:0/1750567765 wait complete. 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.107+0000 7f14277f0740 20 client.17091 trim_cache size 0 max 16384 2024-04-22T06:51:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:51:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:51:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:51:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14080a3e60 msgr2=0x7f14080a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14080a3e60 0x7f14080a6320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1418009ca0 tx=0x7f1418005d80 comp rx=0 tx=0).stop 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 msgr2=0x7f142019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 0x7f142019a6f0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f141000b4a0 tx=0x7f141000b960 comp rx=0 tx=0).stop 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 shutdown_connections 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f14080a3e60 0x7f14080a6320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.110+0000 7f141f59e640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f142019ef60 0x7f142019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.111+0000 7f141f59e640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14201070a0 0x7f142019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.111+0000 7f141f59e640 1 --2- 172.21.15.79:0/2100824535 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1420101fd0 0x7f142019a6f0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.111+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 >> 172.21.15.79:0/2100824535 conn(0x7f14200fd890 msgr2=0x7f14200ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.112+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 shutdown_connections 2024-04-22T06:51:33.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:33.112+0000 7f141f59e640 1 -- 172.21.15.79:0/2100824535 wait complete. 2024-04-22T06:51:33.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:32 smithi193 ceph-mon[31815]: pgmap v227: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 788 MiB/s wr, 2.42k op/s 2024-04-22T06:51:33.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:32 smithi190 ceph-mon[31421]: pgmap v227: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 788 MiB/s wr, 2.42k op/s 2024-04-22T06:51:33.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:32 smithi079 ceph-mon[27946]: pgmap v227: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 788 MiB/s wr, 2.42k op/s 2024-04-22T06:51:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:34 smithi193 ceph-mon[31815]: pgmap v228: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 MiB/s wr, 2.21k op/s 2024-04-22T06:51:35.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:34 smithi190 ceph-mon[31421]: pgmap v228: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 MiB/s wr, 2.21k op/s 2024-04-22T06:51:35.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:34 smithi079 ceph-mon[27946]: pgmap v228: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 MiB/s wr, 2.21k op/s 2024-04-22T06:51:37.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:36 smithi190 ceph-mon[31421]: pgmap v229: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 704 MiB/s wr, 2.04k op/s 2024-04-22T06:51:37.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:36 smithi079 ceph-mon[27946]: pgmap v229: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 704 MiB/s wr, 2.04k op/s 2024-04-22T06:51:37.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:36 smithi193 ceph-mon[31815]: pgmap v229: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 704 MiB/s wr, 2.04k op/s 2024-04-22T06:51:39.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:38 smithi190 ceph-mon[31421]: pgmap v230: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 835 MiB/s wr, 2.60k op/s 2024-04-22T06:51:39.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:38 smithi079 ceph-mon[27946]: pgmap v230: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 835 MiB/s wr, 2.60k op/s 2024-04-22T06:51:39.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:38 smithi193 ceph-mon[31815]: pgmap v230: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 835 MiB/s wr, 2.60k op/s 2024-04-22T06:51:41.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:40 smithi079 ceph-mon[27946]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 731 MiB/s wr, 2.33k op/s 2024-04-22T06:51:41.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:40 smithi193 ceph-mon[31815]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 731 MiB/s wr, 2.33k op/s 2024-04-22T06:51:41.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:40 smithi190 ceph-mon[31421]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 731 MiB/s wr, 2.33k op/s 2024-04-22T06:51:43.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:42 smithi079 ceph-mon[27946]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 752 MiB/s wr, 2.29k op/s 2024-04-22T06:51:43.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:42 smithi193 ceph-mon[31815]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 752 MiB/s wr, 2.29k op/s 2024-04-22T06:51:43.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:42 smithi190 ceph-mon[31421]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 752 MiB/s wr, 2.29k op/s 2024-04-22T06:51:44.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:43 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:51:44.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:43 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:51:44.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:43 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:51:45.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:45 smithi193 ceph-mon[31815]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 683 MiB/s wr, 1.98k op/s 2024-04-22T06:51:45.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:45 smithi190 ceph-mon[31421]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 683 MiB/s wr, 1.98k op/s 2024-04-22T06:51:45.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:45 smithi079 ceph-mon[27946]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 683 MiB/s wr, 1.98k op/s 2024-04-22T06:51:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 -- 172.21.15.79:0/2015438442 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e741021d0 msgr2=0x7f1e741025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 --2- 172.21.15.79:0/2015438442 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e741021d0 0x7f1e741025b0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009900 tx=0x7f1e6402bc30 comp rx=0 tx=0).stop 2024-04-22T06:51:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 -- 172.21.15.79:0/2015438442 shutdown_connections 2024-04-22T06:51:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 --2- 172.21.15.79:0/2015438442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e74102b80 0x7f1e74106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 --2- 172.21.15.79:0/2015438442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e741072a0 0x7f1e7410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 --2- 172.21.15.79:0/2015438442 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e741021d0 0x7f1e741025b0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 -- 172.21.15.79:0/2015438442 >> 172.21.15.79:0/2015438442 conn(0x7f1e740793e0 msgr2=0x7f1e740797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 -- 172.21.15.79:0/2015438442 shutdown_connections 2024-04-22T06:51:46.290 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.291+0000 7f1e79c81640 1 -- 172.21.15.79:0/2015438442 wait complete. 2024-04-22T06:51:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.292+0000 7f1e79c81640 1 Processor -- start 2024-04-22T06:51:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 -- start start 2024-04-22T06:51:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e741021d0 0x7f1e74196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e741072a0 0x7f1e74196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 0x7f1e7419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741100d0 con 0x7f1e7419aea0 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7410ff50 con 0x7f1e741021d0 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.293+0000 7f1e79c81640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1e74110250 con 0x7f1e741072a0 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.294+0000 7f1e73fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 0x7f1e7419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.294+0000 7f1e73fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 0x7f1e7419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37158/0 (socket says 172.21.15.79:37158) 2024-04-22T06:51:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.294+0000 7f1e73fff640 1 -- 172.21.15.79:0/406423707 learned_addr learned my addr 172.21.15.79:0/406423707 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.295+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e741072a0 0x7f1e74196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e737fe640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e741021d0 0x7f1e74196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e73fff640 1 -- 172.21.15.79:0/406423707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e741072a0 msgr2=0x7f1e74196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e73fff640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e741072a0 0x7f1e74196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e73fff640 1 -- 172.21.15.79:0/406423707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e741021d0 msgr2=0x7f1e74196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e73fff640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e741021d0 0x7f1e74196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e73fff640 1 -- 172.21.15.79:0/406423707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7419b990 con 0x7f1e7419aea0 2024-04-22T06:51:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.296+0000 7f1e73fff640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 0x7f1e7419b350 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00b9e0 tx=0x7f1e6c00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.297+0000 7f1e70ff9640 1 -- 172.21.15.79:0/406423707 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c019070 con 0x7f1e7419aea0 2024-04-22T06:51:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.297+0000 7f1e70ff9640 1 -- 172.21.15.79:0/406423707 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e6c014440 con 0x7f1e7419aea0 2024-04-22T06:51:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.297+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7419bc80 con 0x7f1e7419aea0 2024-04-22T06:51:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.297+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7406bce0 con 0x7f1e7419aea0 2024-04-22T06:51:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.298+0000 7f1e70ff9640 1 -- 172.21.15.79:0/406423707 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0075d0 con 0x7f1e7419aea0 2024-04-22T06:51:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.301+0000 7f1e70ff9640 1 -- 172.21.15.79:0/406423707 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e6c01c070 con 0x7f1e7419aea0 2024-04-22T06:51:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 Processor -- start 2024-04-22T06:51:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 -- start start 2024-04-22T06:51:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af91195620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af91195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55af910fa8d0 con 0x55af911968d0 2024-04-22T06:51:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55af91097e80 con 0x55af911a78f0 2024-04-22T06:51:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.302+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55af910f6890 con 0x55af91195b60 2024-04-22T06:51:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.303+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af91195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.303+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af91195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37172/0 (socket says 172.21.15.79:37172) 2024-04-22T06:51:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.303+0000 7f1e72ffd640 1 -- 172.21.15.79:0/2291579814 learned_addr learned my addr 172.21.15.79:0/2291579814 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.303+0000 7f1e70ff9640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1e5c0a3e50 0x7f1e5c0a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.304+0000 7f1e73fff640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1e5c0a3e50 0x7f1e5c0a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.304+0000 7f1e737fe640 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af91195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e72ffd640 1 -- 172.21.15.79:0/2291579814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 msgr2=0x55af91195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af91195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e72ffd640 1 -- 172.21.15.79:0/2291579814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 msgr2=0x55af911a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e72ffd640 1 -- 172.21.15.79:0/2291579814 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af910b9320 con 0x55af911968d0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e70ff9640 1 -- 172.21.15.79:0/406423707 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7f1e6c0cfe30 con 0x7f1e7419aea0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e73fff640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1e5c0a3e50 0x7f1e5c0a6310 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0984c0 tx=0x7f1e6c0c6c00 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.305+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af91195620 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009ab0 tx=0x7f1e68009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0a825f6fc1e4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.306+0000 7f1e59ffb640 1 -- 172.21.15.79:0/2291579814 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68018020 con 0x55af911968d0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.306+0000 7f1e59ffb640 1 -- 172.21.15.79:0/2291579814 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e68010dc0 con 0x55af911968d0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.307+0000 7f1e59ffb640 1 -- 172.21.15.79:0/2291579814 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e680136d0 con 0x55af911968d0 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.307+0000 7f1e7c87c740 1 -- 172.21.15.79:0/2291579814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 msgr2=0x55af91195620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.307+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af91195620 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009ab0 tx=0x7f1e68009f70 comp rx=0 tx=0).stop 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.307+0000 7f1e7c87c740 1 -- 172.21.15.79:0/2291579814 shutdown_connections 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.307+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.308+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af91195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.308+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/2291579814 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af91195620 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.308+0000 7f1e7c87c740 1 -- 172.21.15.79:0/2291579814 >> 172.21.15.79:0/2291579814 conn(0x55af9115cc90 msgr2=0x55af91190850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.308+0000 7f1e7c87c740 1 -- 172.21.15.79:0/2291579814 shutdown_connections 2024-04-22T06:51:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.308+0000 7f1e7c87c740 1 -- 172.21.15.79:0/2291579814 wait complete. 2024-04-22T06:51:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.309+0000 7f1e7c87c740 1 -- start start 2024-04-22T06:51:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.310+0000 7f1e7c87c740 1 Processor -- start 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.310+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911a4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911a4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37174/0 (socket says 172.21.15.79:37174) 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e737fe640 1 -- 172.21.15.79:0/770999214 learned_addr learned my addr 172.21.15.79:0/770999214 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55af911aa670 con 0x55af911968d0 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55af91082eb0 con 0x55af911a78f0 2024-04-22T06:51:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.311+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55af911aa7f0 con 0x55af91195b60 2024-04-22T06:51:46.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e737fe640 1 -- 172.21.15.79:0/770999214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 msgr2=0x55af911a70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e737fe640 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.311 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e737fe640 1 -- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 msgr2=0x55af911a49a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e737fe640 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e737fe640 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af90e0d210 con 0x55af911968d0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e7c87c740 10 client.0 _ll_register_callbacks cb 0x55af910afb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e737fe640 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911a4460 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f1e64006010 tx=0x7f1e64005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529fa77794d9ea64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.312+0000 7f1e52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e52ffd640 1 -- 172.21.15.79:0/770999214 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64004350 con 0x55af911968d0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e52ffd640 1 -- 172.21.15.79:0/770999214 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7f1e64005030 con 0x55af911968d0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e68010dc0 con 0x55af911968d0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e52ffd640 1 -- 172.21.15.79:0/770999214 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f1e6404e050 con 0x55af911968d0 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e7c87c740 10 client.17103 fetch_fsmap learned FSMap version 60 2024-04-22T06:51:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({fsmap=60}) -- 0x55af90ed9620 con 0x55af911968d0 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.313+0000 7f1e52ffd640 1 -- 172.21.15.79:0/770999214 <== mon.0 v2:172.21.15.79:3300/0 4 ==== fsmap(e 60) ==== 8338+0+0 (secure 0 0 0) 0x7f1e64005230 con 0x55af911968d0 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.314+0000 7f1e7c87c740 10 client.17103 fetch_fsmap finished waiting for FSMap version 60 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.314+0000 7f1e7c87c740 10 client.17103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.314+0000 7f1e7c87c740 20 client.17103 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.314+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af911b76c0 0x55af911b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.314+0000 7f1e7c87c740 4 client.17103 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.314+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1e68010dc0 con 0x55af911b76c0 2024-04-22T06:51:46.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.321+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af911b76c0 0x55af911b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.326+0000 7f1e52ffd640 10 client.17103 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.326+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af911b76c0 0x55af911b9b80 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933fda5dbd359bc2 server_cookie=39c449bda47a497a in_seq=0 out_seq=0 2024-04-22T06:51:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.330+0000 7f1e52ffd640 1 -- 172.21.15.79:0/770999214 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1e68010dc0 con 0x55af911b76c0 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.330+0000 7f1e52ffd640 10 client.17103 handle_command_reply: tid=0 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.330+0000 7f1e7c87c740 1 client.17103 shutdown 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.330+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 msgr2=0x55af911a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911a4460 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f1e64006010 tx=0x7f1e64005dc0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 shutdown_connections 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af911b76c0 0x55af911b9b80 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911a4460 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/770999214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911a70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 >> 172.21.15.79:0/770999214 conn(0x55af9115cc90 msgr2=0x55af911925a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e52ffd640 0 client.17103 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 shutdown_connections 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 1 -- 172.21.15.79:0/770999214 wait complete. 2024-04-22T06:51:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.331+0000 7f1e7c87c740 20 client.17103 trim_cache size 0 max 16384 2024-04-22T06:51:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.372+0000 7f1e7c87c740 1 Processor -- start 2024-04-22T06:51:46.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 -- start start 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 0x55af911b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af911968d0 0x55af911a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1e600117b0 con 0x55af91195b60 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6000f080 con 0x55af911a78f0 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.380+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6000cef0 con 0x55af911968d0 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.382+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 0x55af911b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.382+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af911968d0 0x55af911a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.382+0000 7f1e72ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 0x55af911b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37180/0 (socket says 172.21.15.79:37180) 2024-04-22T06:51:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.382+0000 7f1e72ffd640 1 -- 172.21.15.79:0/921902387 learned_addr learned my addr 172.21.15.79:0/921902387 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.382+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.384+0000 7f1e72ffd640 1 -- 172.21.15.79:0/921902387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af911968d0 msgr2=0x55af911a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.384+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af911968d0 0x55af911a6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.384+0000 7f1e72ffd640 1 -- 172.21.15.79:0/921902387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 msgr2=0x55af911a70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.384+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.384+0000 7f1e72ffd640 1 -- 172.21.15.79:0/921902387 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af9118ace0 con 0x55af91195b60 2024-04-22T06:51:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.398+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 0x55af911b3650 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1e680133e0 tx=0x7f1e68014f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2607d98302c7e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.398+0000 7f1e597fa640 1 -- 172.21.15.79:0/921902387 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68018040 con 0x55af91195b60 2024-04-22T06:51:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.399+0000 7f1e597fa640 1 -- 172.21.15.79:0/921902387 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e68026760 con 0x55af91195b60 2024-04-22T06:51:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 -- 172.21.15.79:0/921902387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 msgr2=0x55af911b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 0x55af911b3650 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1e680133e0 tx=0x7f1e68014f60 comp rx=0 tx=0).stop 2024-04-22T06:51:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 -- 172.21.15.79:0/921902387 shutdown_connections 2024-04-22T06:51:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af911a70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af911968d0 0x55af911a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/921902387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af91195b60 0x55af911b3650 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 -- 172.21.15.79:0/921902387 >> 172.21.15.79:0/921902387 conn(0x55af9115cc90 msgr2=0x55af911a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:46.399 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.400+0000 7f1e7c87c740 1 -- 172.21.15.79:0/921902387 shutdown_connections 2024-04-22T06:51:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.408+0000 7f1e7c87c740 1 -- 172.21.15.79:0/921902387 wait complete. 2024-04-22T06:51:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.408+0000 7f1e7c87c740 1 -- start start 2024-04-22T06:51:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.409+0000 7f1e7c87c740 1 Processor -- start 2024-04-22T06:51:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af912191f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6000d540 con 0x55af911968d0 2024-04-22T06:51:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6000e9b0 con 0x55af911a78f0 2024-04-22T06:51:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6000dc10 con 0x55af91195b60 2024-04-22T06:51:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.410+0000 7f1e7c87c740 10 client.0 _ll_register_callbacks cb 0x7f1e64009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.413+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.413+0000 7f1e737fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37190/0 (socket says 172.21.15.79:37190) 2024-04-22T06:51:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.413+0000 7f1e737fe640 1 -- 172.21.15.79:0/3543980612 learned_addr learned my addr 172.21.15.79:0/3543980612 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:51:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.414+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.415+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af912191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.417+0000 7f1e72ffd640 1 -- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 msgr2=0x55af912191f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.417+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af912191f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.417+0000 7f1e72ffd640 1 -- 172.21.15.79:0/3543980612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 msgr2=0x55af911b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.417+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.417+0000 7f1e72ffd640 1 -- 172.21.15.79:0/3543980612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af912199f0 con 0x55af91195b60 2024-04-22T06:51:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.422+0000 7f1e72ffd640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911afb60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1e68001eb0 tx=0x7f1e6801f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d0be0aff9696252 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.422+0000 7f1e52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-22T06:51:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.424+0000 7f1e52ffd640 1 -- 172.21.15.79:0/3543980612 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68033070 con 0x55af91195b60 2024-04-22T06:51:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.424+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e68026760 con 0x55af91195b60 2024-04-22T06:51:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.425+0000 7f1e52ffd640 1 -- 172.21.15.79:0/3543980612 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7f1e680264b0 con 0x55af91195b60 2024-04-22T06:51:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.427+0000 7f1e52ffd640 1 -- 172.21.15.79:0/3543980612 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f1e68042050 con 0x55af91195b60 2024-04-22T06:51:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.429+0000 7f1e7c87c740 10 client.25939 fetch_fsmap learned FSMap version 60 2024-04-22T06:51:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.429+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55af9119b100 con 0x55af91195b60 2024-04-22T06:51:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.434+0000 7f1e52ffd640 1 -- 172.21.15.79:0/3543980612 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e68026760 con 0x55af91195b60 2024-04-22T06:51:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.436+0000 7f1e52ffd640 1 -- 172.21.15.79:0/3543980612 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 60) ==== 8338+0+0 (secure 0 0 0) 0x7f1e6802d020 con 0x55af91195b60 2024-04-22T06:51:46.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:46 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:51:46.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:46 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:51:46.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:46 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:51:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.437+0000 7f1e7c87c740 10 client.25939 fetch_fsmap finished waiting for FSMap version 60 2024-04-22T06:51:46.439 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.437+0000 7f1e7c87c740 10 client.25939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:51:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.437+0000 7f1e7c87c740 20 client.25939 populate_metadata read hostname 'smithi079' 2024-04-22T06:51:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.437+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af9119b520 0x55af9119d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.437+0000 7f1e7c87c740 4 client.25939 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.437+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af9110e880 con 0x55af9119b520 2024-04-22T06:51:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.440+0000 7f1e737fe640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af9119b520 0x55af9119d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:46.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:46 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:51:46.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:46 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:51:46.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:46 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:51:46.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.459+0000 7f1e737fe640 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af9119b520 0x55af9119d940 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a48a7412697ee95 server_cookie=debab31a6ba1ca1d in_seq=0 out_seq=0 2024-04-22T06:51:46.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:46.462+0000 7f1e52ffd640 10 client.25939 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:46.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:46 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:51:46.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:46 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:51:46.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:46 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:51:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:47 smithi193 ceph-mon[31815]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 628 MiB/s wr, 1.78k op/s 2024-04-22T06:51:47.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:47 smithi190 ceph-mon[31421]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 628 MiB/s wr, 1.78k op/s 2024-04-22T06:51:47.494 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:47 smithi079 ceph-mon[27946]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 628 MiB/s wr, 1.78k op/s 2024-04-22T06:51:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.034+0000 7f1e52ffd640 1 -- 172.21.15.79:0/3543980612 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af9110e880 con 0x55af9119b520 2024-04-22T06:51:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.034+0000 7f1e52ffd640 10 client.25939 handle_command_reply: tid=0 2024-04-22T06:51:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.034+0000 7f1e7c87c740 1 client.25939 shutdown 2024-04-22T06:51:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.034+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 msgr2=0x55af911afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.034+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911afb60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1e68001eb0 tx=0x7f1e6801f040 comp rx=0 tx=0).stop 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 shutdown_connections 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55af9119b520 0x55af9119d940 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af911a78f0 0x55af912191f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55af911968d0 0x55af911b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e7c87c740 1 --2- 172.21.15.79:0/3543980612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af91195b60 0x55af911afb60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 >> 172.21.15.79:0/3543980612 conn(0x55af9115cc90 msgr2=0x55af911a2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.036+0000 7f1e52ffd640 0 client.25939 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:51:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.037+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 shutdown_connections 2024-04-22T06:51:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.037+0000 7f1e7c87c740 1 -- 172.21.15.79:0/3543980612 wait complete. 2024-04-22T06:51:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.037+0000 7f1e7c87c740 20 client.25939 trim_cache size 0 max 16384 2024-04-22T06:51:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:51:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:51:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:51:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1e5c0a3e50 msgr2=0x7f1e5c0a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1e5c0a3e50 0x7f1e5c0a6310 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0984c0 tx=0x7f1e6c0c6c00 comp rx=0 tx=0).stop 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 msgr2=0x7f1e7419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 0x7f1e7419b350 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00b9e0 tx=0x7f1e6c00bde0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 shutdown_connections 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e7419aea0 0x7f1e7419b350 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e741072a0 0x7f1e74196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1e5c0a3e50 0x7f1e5c0a6310 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 --2- 172.21.15.79:0/406423707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e741021d0 0x7f1e74196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.039+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 >> 172.21.15.79:0/406423707 conn(0x7f1e740793e0 msgr2=0x7f1e740fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.040+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 shutdown_connections 2024-04-22T06:51:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:51:49.040+0000 7f1e79c81640 1 -- 172.21.15.79:0/406423707 wait complete. 2024-04-22T06:51:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:49 smithi193 ceph-mon[31815]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 799 MiB/s wr, 2.10k op/s 2024-04-22T06:51:49.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:49 smithi190 ceph-mon[31421]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 799 MiB/s wr, 2.10k op/s 2024-04-22T06:51:49.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:49 smithi079 ceph-mon[27946]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 799 MiB/s wr, 2.10k op/s 2024-04-22T06:51:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:51 smithi193 ceph-mon[31815]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 707 MiB/s wr, 1.52k op/s 2024-04-22T06:51:51.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:51 smithi190 ceph-mon[31421]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 707 MiB/s wr, 1.52k op/s 2024-04-22T06:51:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:51 smithi079 ceph-mon[27946]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 707 MiB/s wr, 1.52k op/s 2024-04-22T06:51:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:53 smithi193 ceph-mon[31815]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 763 MiB/s wr, 1.32k op/s 2024-04-22T06:51:53.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:53 smithi190 ceph-mon[31421]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 763 MiB/s wr, 1.32k op/s 2024-04-22T06:51:53.500 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:53 smithi079 ceph-mon[27946]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 763 MiB/s wr, 1.32k op/s 2024-04-22T06:51:55.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:55 smithi193 ceph-mon[31815]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 728 MiB/s wr, 1.12k op/s 2024-04-22T06:51:55.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:55 smithi190 ceph-mon[31421]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 728 MiB/s wr, 1.12k op/s 2024-04-22T06:51:55.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:55 smithi079 ceph-mon[27946]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 728 MiB/s wr, 1.12k op/s 2024-04-22T06:51:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:57 smithi193 ceph-mon[31815]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 661 MiB/s wr, 1.06k op/s 2024-04-22T06:51:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:57 smithi190 ceph-mon[31421]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 661 MiB/s wr, 1.06k op/s 2024-04-22T06:51:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:57 smithi079 ceph-mon[27946]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 661 MiB/s wr, 1.06k op/s 2024-04-22T06:51:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:51:59 smithi193 ceph-mon[31815]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 813 MiB/s wr, 1.30k op/s 2024-04-22T06:51:59.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:51:59 smithi190 ceph-mon[31421]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 813 MiB/s wr, 1.30k op/s 2024-04-22T06:51:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:51:59 smithi079 ceph-mon[27946]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 813 MiB/s wr, 1.30k op/s 2024-04-22T06:52:01.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:01 smithi193 ceph-mon[31815]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 705 MiB/s wr, 948 op/s 2024-04-22T06:52:01.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:01 smithi190 ceph-mon[31421]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 705 MiB/s wr, 948 op/s 2024-04-22T06:52:01.494 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:01 smithi079 ceph-mon[27946]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 705 MiB/s wr, 948 op/s 2024-04-22T06:52:02.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.368+0000 7ffb1fe32640 1 -- 172.21.15.79:0/1705389536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb18106170 msgr2=0x7ffb1810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.368+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/1705389536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb18106170 0x7ffb1810dd40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009ca0 tx=0x7ffb1002c120 comp rx=0 tx=0).stop 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 -- 172.21.15.79:0/1705389536 shutdown_connections 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/1705389536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb18101a50 0x7ffb18105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/1705389536 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb18106170 0x7ffb1810dd40 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/1705389536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb181010a0 0x7ffb18101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 -- 172.21.15.79:0/1705389536 >> 172.21.15.79:0/1705389536 conn(0x7ffb180fc920 msgr2=0x7ffb180fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 -- 172.21.15.79:0/1705389536 shutdown_connections 2024-04-22T06:52:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.369+0000 7ffb1fe32640 1 -- 172.21.15.79:0/1705389536 wait complete. 2024-04-22T06:52:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1fe32640 1 Processor -- start 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1fe32640 1 -- start start 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1fe32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 0x7ffb18073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1fe32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18106170 0x7ffb1806bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 0x7ffb18073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 0x7ffb18073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35910/0 (socket says 172.21.15.79:35910) 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1dbce640 1 -- 172.21.15.79:0/2756695316 learned_addr learned my addr 172.21.15.79:0/2756695316 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.371+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1806c1b0 0x7ffb1806c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1810ef30 con 0x7ffb181010a0 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1810edb0 con 0x7ffb1806c1b0 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1810f0b0 con 0x7ffb18106170 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1dbce640 1 -- 172.21.15.79:0/2756695316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18106170 msgr2=0x7ffb1806bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18106170 0x7ffb1806bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1dbce640 1 -- 172.21.15.79:0/2756695316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1806c1b0 msgr2=0x7ffb1806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1806c1b0 0x7ffb1806c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.372+0000 7ffb1dbce640 1 -- 172.21.15.79:0/2756695316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb181af940 con 0x7ffb181010a0 2024-04-22T06:52:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.373+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 0x7ffb18073ae0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400b450 tx=0x7ffb1400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.373+0000 7ffb0effd640 1 -- 172.21.15.79:0/2756695316 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb14015020 con 0x7ffb181010a0 2024-04-22T06:52:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.373+0000 7ffb0effd640 1 -- 172.21.15.79:0/2756695316 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb14011ce0 con 0x7ffb181010a0 2024-04-22T06:52:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.373+0000 7ffb0effd640 1 -- 172.21.15.79:0/2756695316 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb14010e20 con 0x7ffb181010a0 2024-04-22T06:52:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.373+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb181afc30 con 0x7ffb181010a0 2024-04-22T06:52:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.374+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb181b0130 con 0x7ffb181010a0 2024-04-22T06:52:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.377+0000 7ffb0effd640 1 -- 172.21.15.79:0/2756695316 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffb14018020 con 0x7ffb181010a0 2024-04-22T06:52:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.379+0000 7ffb22a1b740 1 Processor -- start 2024-04-22T06:52:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.379+0000 7ffb22a1b740 1 -- start start 2024-04-22T06:52:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.379+0000 7ffb0effd640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffb040a3fb0 0x7ffb040a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1e3cf640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffb040a3fb0 0x7ffb040a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 0x5621257c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 0x5621257cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5621257208d0 con 0x5621257c50e0 2024-04-22T06:52:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5621256bde80 con 0x5621257cd8f0 2024-04-22T06:52:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56212571c890 con 0x5621257c4370 2024-04-22T06:52:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb0effd640 1 -- 172.21.15.79:0/2756695316 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7ffb140d0260 con 0x7ffb181010a0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35918/0 (socket says 172.21.15.79:35918) 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1106860575 learned_addr learned my addr 172.21.15.79:0/1106860575 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1106860575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 msgr2=0x5621257c47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 0x5621257c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1106860575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 msgr2=0x5621257cfce0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 0x5621257cfce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1106860575 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621256df320 con 0x5621257c50e0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1e3cf640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffb040a3fb0 0x7ffb040a6470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffb10006010 tx=0x7ffb10005fa0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.380+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257c3e30 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ffb14098930 tx=0x7ffb1409b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3025c53d411756 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.381+0000 7ffb03fff640 1 -- 172.21.15.79:0/1106860575 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb140a0310 con 0x5621257c50e0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.381+0000 7ffb03fff640 1 -- 172.21.15.79:0/1106860575 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb140b5070 con 0x5621257c50e0 2024-04-22T06:52:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.381+0000 7ffb03fff640 1 -- 172.21.15.79:0/1106860575 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb140b5220 con 0x5621257c50e0 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.381+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1106860575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 msgr2=0x5621257c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.381+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257c3e30 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ffb14098930 tx=0x7ffb1409b980 comp rx=0 tx=0).stop 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1106860575 shutdown_connections 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 0x5621257cfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 0x5621257c47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1106860575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257c3e30 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1106860575 >> 172.21.15.79:0/1106860575 conn(0x562125782c90 msgr2=0x5621257bf060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1106860575 shutdown_connections 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1106860575 wait complete. 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 -- start start 2024-04-22T06:52:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.382+0000 7ffb22a1b740 1 Processor -- start 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 0x5621257cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 0x5621257caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5621257d0610 con 0x5621257c50e0 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5621256a8eb0 con 0x5621257cd8f0 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621257d0790 con 0x5621257c4370 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb22a1b740 10 client.0 _ll_register_callbacks cb 0x5621256d5b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35928/0 (socket says 172.21.15.79:35928) 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 -- 172.21.15.79:0/1759605654 learned_addr learned my addr 172.21.15.79:0/1759605654 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 -- 172.21.15.79:0/1759605654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 msgr2=0x5621257cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 0x5621257cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 -- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 msgr2=0x5621257caab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 0x5621257caab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.383+0000 7ffb1d3cd640 1 -- 172.21.15.79:0/1759605654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562125433210 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb1d3cd640 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257ca570 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009dc0 tx=0x7ffb0800b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5373b364543fa4c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb017fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb017fa640 1 -- 172.21.15.79:0/1759605654 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb08015020 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb140b5070 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb017fa640 1 -- 172.21.15.79:0/1759605654 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7ffb08011420 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb017fa640 1 -- 172.21.15.79:0/1759605654 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7ffb08020050 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 10 client.17121 fetch_fsmap learned FSMap version 60 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x5621254ff620 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb017fa640 1 -- 172.21.15.79:0/1759605654 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb140b5070 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb017fa640 1 -- 172.21.15.79:0/1759605654 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 60) ==== 8338+0+0 (secure 0 0 0) 0x7ffb08019840 con 0x5621257c50e0 2024-04-22T06:52:02.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 10 client.17121 fetch_fsmap finished waiting for FSMap version 60 2024-04-22T06:52:02.387 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 10 client.17121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:02.387 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 20 client.17121 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:02.387 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5621257dd950 0x5621257dfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.387 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 4 client.17121 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:02.387 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.384+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562125734880 con 0x5621257dd950 2024-04-22T06:52:02.387 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.387+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5621257dd950 0x5621257dfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.389 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.391+0000 7ffb017fa640 10 client.17121 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:02.390 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.391+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5621257dd950 0x5621257dfd80 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2430067f09358621 server_cookie=6b21bcf13b7811 in_seq=0 out_seq=0 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.394+0000 7ffb017fa640 1 -- 172.21.15.79:0/1759605654 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562125734880 con 0x5621257dd950 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.394+0000 7ffb017fa640 10 client.17121 handle_command_reply: tid=0 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.394+0000 7ffb22a1b740 1 client.17121 shutdown 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.394+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 msgr2=0x5621257ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.394+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257ca570 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009dc0 tx=0x7ffb0800b4b0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 shutdown_connections 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5621257dd950 0x5621257dfd80 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257cd8f0 0x5621257caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c50e0 0x5621257ca570 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1759605654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257c4370 0x5621257cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 >> 172.21.15.79:0/1759605654 conn(0x562125782c90 msgr2=0x5621257c0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb017fa640 0 client.17121 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 shutdown_connections 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1759605654 wait complete. 2024-04-22T06:52:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.395+0000 7ffb22a1b740 20 client.17121 trim_cache size 0 max 16384 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 Processor -- start 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 -- start start 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257c3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b72e0 con 0x5621257c4370 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b4bb0 con 0x5621257c50e0 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b2a40 con 0x5621257cd8f0 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35936/0 (socket says 172.21.15.79:35936) 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1750657910 learned_addr learned my addr 172.21.15.79:0/1750657910 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.415+0000 7ffb1d3cd640 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1e3cf640 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1750657910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 msgr2=0x5621257ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1750657910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 msgr2=0x5621257c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1750657910 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621257dc4f0 con 0x5621257c4370 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d9560 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400be10 tx=0x7ffb140106a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fa8585b4be8709 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb037fe640 1 -- 172.21.15.79:0/1750657910 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb140a9070 con 0x5621257c4370 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb037fe640 1 -- 172.21.15.79:0/1750657910 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb140b5040 con 0x5621257c4370 2024-04-22T06:52:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.416+0000 7ffb037fe640 1 -- 172.21.15.79:0/1750657910 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb140b5450 con 0x5621257c4370 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1750657910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 msgr2=0x5621257d9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d9560 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400be10 tx=0x7ffb140106a0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1750657910 shutdown_connections 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1750657910 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d9560 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1750657910 >> 172.21.15.79:0/1750657910 conn(0x562125782c90 msgr2=0x5621257c8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1750657910 shutdown_connections 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.417+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1750657910 wait complete. 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 -- start start 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 Processor -- start 2024-04-22T06:52:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b3090 con 0x5621257c4370 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b44e0 con 0x5621257c50e0 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb181b3740 con 0x5621257cd8f0 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.418+0000 7ffb22a1b740 10 client.0 _ll_register_callbacks cb 0x7ffb08006e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35948/0 (socket says 172.21.15.79:35948) 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1824494006 learned_addr learned my addr 172.21.15.79:0/1824494006 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1e3cf640 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1824494006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 msgr2=0x5621257d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 msgr2=0x5621257d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 -- 172.21.15.79:0/1824494006 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621257b1f30 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb1dbce640 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d5af0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ffb140983c0 tx=0x7ffb140a0390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be440b695185195a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb017fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb017fa640 1 -- 172.21.15.79:0/1824494006 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb140a9070 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb017fa640 1 -- 172.21.15.79:0/1824494006 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9373+0+0 (secure 0 0 0) 0x7ffb140985d0 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.419+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb140b5040 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb017fa640 1 -- 172.21.15.79:0/1824494006 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7ffb140cda40 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 10 client.17127 fetch_fsmap learned FSMap version 60 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x562125843970 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb017fa640 1 -- 172.21.15.79:0/1824494006 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb140b5040 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb017fa640 1 -- 172.21.15.79:0/1824494006 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 60) ==== 8338+0+0 (secure 0 0 0) 0x7ffb14097cb0 con 0x5621257c4370 2024-04-22T06:52:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 10 client.17127 fetch_fsmap finished waiting for FSMap version 60 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 10 client.17127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 20 client.17127 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562125843bf0 0x562125845fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 4 client.17127 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.420+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621256c03d0 con 0x562125843bf0 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.421+0000 7ffb1d3cd640 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562125843bf0 0x562125845fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.421+0000 7ffb1d3cd640 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562125843bf0 0x562125845fe0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75d6385eaec6a994 server_cookie=d3e68836b85aeebf in_seq=0 out_seq=0 2024-04-22T06:52:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.421+0000 7ffb017fa640 10 client.17127 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.476+0000 7ffb017fa640 1 -- 172.21.15.79:0/1824494006 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621256c03d0 con 0x562125843bf0 2024-04-22T06:52:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.476+0000 7ffb017fa640 10 client.17127 handle_command_reply: tid=0 2024-04-22T06:52:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.476+0000 7ffb22a1b740 1 client.17127 shutdown 2024-04-22T06:52:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.477+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 msgr2=0x5621257d5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.477+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d5af0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7ffb140983c0 tx=0x7ffb140a0390 comp rx=0 tx=0).stop 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 shutdown_connections 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562125843bf0 0x562125845fe0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb017fa640 0 client.17127 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621257c50e0 0x5621257d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621257cd8f0 0x5621257d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 --2- 172.21.15.79:0/1824494006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5621257c4370 0x5621257d5af0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 >> 172.21.15.79:0/1824494006 conn(0x562125782c90 msgr2=0x5621257b58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 shutdown_connections 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.478+0000 7ffb22a1b740 1 -- 172.21.15.79:0/1824494006 wait complete. 2024-04-22T06:52:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.479+0000 7ffb22a1b740 20 client.17127 trim_cache size 0 max 16384 2024-04-22T06:52:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffb040a3fb0 msgr2=0x7ffb040a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffb040a3fb0 0x7ffb040a6470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffb10006010 tx=0x7ffb10005fa0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 msgr2=0x7ffb18073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 0x7ffb18073ae0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400b450 tx=0x7ffb1400b910 comp rx=0 tx=0).stop 2024-04-22T06:52:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 shutdown_connections 2024-04-22T06:52:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffb040a3fb0 0x7ffb040a6470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb1806c1b0 0x7ffb1806c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18106170 0x7ffb1806bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 --2- 172.21.15.79:0/2756695316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb181010a0 0x7ffb18073ae0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.481+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 >> 172.21.15.79:0/2756695316 conn(0x7ffb180fc920 msgr2=0x7ffb180fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.482+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 shutdown_connections 2024-04-22T06:52:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:02.483+0000 7ffb1fe32640 1 -- 172.21.15.79:0/2756695316 wait complete. 2024-04-22T06:52:03.134 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:52:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: ignoring --setuser ceph since I am not root 2024-04-22T06:52:03.134 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:52:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: ignoring --setgroup ceph since I am not root 2024-04-22T06:52:03.134 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:52:02 smithi193 ceph-mds[62955]: -- 172.21.15.193:0/2049375829 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d59c045a40 con 0x55d59c019000 2024-04-22T06:52:03.134 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:52:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: starting mds.c at 2024-04-22T06:52:03.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:03 smithi193 ceph-mon[31815]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 752 MiB/s wr, 888 op/s 2024-04-22T06:52:03.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:03 smithi193 ceph-mon[31815]: daemon mds.c restarted 2024-04-22T06:52:03.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:03 smithi190 ceph-mon[31421]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 752 MiB/s wr, 888 op/s 2024-04-22T06:52:03.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:03 smithi190 ceph-mon[31421]: daemon mds.c restarted 2024-04-22T06:52:03.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:03 smithi079 ceph-mon[27946]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 752 MiB/s wr, 888 op/s 2024-04-22T06:52:03.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:03 smithi079 ceph-mon[27946]: daemon mds.c restarted 2024-04-22T06:52:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:04 smithi079 ceph-mon[27946]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:52:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:04 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6836/30505216,v1:172.21.15.193:6837/30505216] up:boot 2024-04-22T06:52:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:04 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:52:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:04 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:52:04.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:04 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:52:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:04 smithi193 ceph-mon[31815]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:52:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:04 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6836/30505216,v1:172.21.15.193:6837/30505216] up:boot 2024-04-22T06:52:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:04 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:52:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:04 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:52:04.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:04 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:52:04.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:04 smithi190 ceph-mon[31421]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:52:04.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:04 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6836/30505216,v1:172.21.15.193:6837/30505216] up:boot 2024-04-22T06:52:04.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:04 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:52:04.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:04 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:52:04.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:04 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:52:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:05 smithi193 ceph-mon[31815]: pgmap v244: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 686 MiB/s wr, 831 op/s 2024-04-22T06:52:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:05 smithi193 ceph-mon[31815]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:52:05.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:05 smithi190 ceph-mon[31421]: pgmap v244: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 686 MiB/s wr, 831 op/s 2024-04-22T06:52:05.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:05 smithi190 ceph-mon[31421]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:52:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:05 smithi079 ceph-mon[27946]: pgmap v244: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 686 MiB/s wr, 831 op/s 2024-04-22T06:52:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:05 smithi079 ceph-mon[27946]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:52:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:07 smithi193 ceph-mon[31815]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 567 MiB/s wr, 497 op/s 2024-04-22T06:52:07.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:07 smithi190 ceph-mon[31421]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 567 MiB/s wr, 497 op/s 2024-04-22T06:52:07.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:07 smithi079 ceph-mon[27946]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 567 MiB/s wr, 497 op/s 2024-04-22T06:52:09.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:09 smithi193 ceph-mon[31815]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 678 MiB/s wr, 869 op/s 2024-04-22T06:52:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:09 smithi190 ceph-mon[31421]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 678 MiB/s wr, 869 op/s 2024-04-22T06:52:09.473 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.7 2024-04-22T06:52:09.473 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:52:09.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:09 smithi079 ceph-mon[27946]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 678 MiB/s wr, 869 op/s 2024-04-22T06:52:09.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 -- 172.21.15.79:0/3172027206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8069aa0 msgr2=0x7f7ec810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:09.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 --2- 172.21.15.79:0/3172027206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8069aa0 0x7f7ec810ede0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009ca0 tx=0x7f7ec002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 -- 172.21.15.79:0/3172027206 shutdown_connections 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 --2- 172.21.15.79:0/3172027206 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8069aa0 0x7f7ec810ede0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 --2- 172.21.15.79:0/3172027206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec80690e0 0x7f7ec8069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 --2- 172.21.15.79:0/3172027206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec8106f50 0x7f7ec8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 -- 172.21.15.79:0/3172027206 >> 172.21.15.79:0/3172027206 conn(0x7f7ec8077e30 msgr2=0x7f7ec8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 -- 172.21.15.79:0/3172027206 shutdown_connections 2024-04-22T06:52:09.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.672+0000 7f7eccf32640 1 -- 172.21.15.79:0/3172027206 wait complete. 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 Processor -- start 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 -- start start 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec80690e0 0x7f7ec8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec8069aa0 0x7f7ec8198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 0x7f7ec819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec8112170 con 0x7f7ec8106f50 2024-04-22T06:52:09.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec8111ff0 con 0x7f7ec80690e0 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7eccf32640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec81122f0 con 0x7f7ec8069aa0 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 0x7f7ec819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec6d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 0x7f7ec819cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42868/0 (socket says 172.21.15.79:42868) 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec6d9d640 1 -- 172.21.15.79:0/4104605509 learned_addr learned my addr 172.21.15.79:0/4104605509 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec659c640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec80690e0 0x7f7ec8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec5d9b640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec8069aa0 0x7f7ec8198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:09.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec6d9d640 1 -- 172.21.15.79:0/4104605509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec8069aa0 msgr2=0x7f7ec8198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.673+0000 7f7ec6d9d640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec8069aa0 0x7f7ec8198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7ec6d9d640 1 -- 172.21.15.79:0/4104605509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec80690e0 msgr2=0x7f7ec8198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7ec6d9d640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec80690e0 0x7f7ec8198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7ec6d9d640 1 -- 172.21.15.79:0/4104605509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec819d5d0 con 0x7f7ec8106f50 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7ec5d9b640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec8069aa0 0x7f7ec8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7ec659c640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec80690e0 0x7f7ec8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7ec6d9d640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 0x7f7ec819cea0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7ec002c700 tx=0x7f7ec00346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:09.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0041020 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec0032960 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0039040 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec819d860 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.674+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec81a5150 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.675+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec8103650 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.676+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7ec0044020 con 0x7f7ec8106f50 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.677+0000 7f7eb77fe640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7eb00a3d20 0x7f7eb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.677+0000 7f7ec659c640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7eb00a3d20 0x7f7eb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:09.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.677+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9417+0+0 (secure 0 0 0) 0x7f7ec00f2c90 con 0x7f7ec8106f50 2024-04-22T06:52:09.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.677+0000 7f7ec659c640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7eb00a3d20 0x7f7eb00a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80058c0 tx=0x7f7eb8005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:09.681 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.680+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ec004b020 con 0x7f7ec8106f50 2024-04-22T06:52:09.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.896+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ec8068de0 con 0x7f7ec8106f50 2024-04-22T06:52:09.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.897+0000 7f7eb77fe640 1 -- 172.21.15.79:0/4104605509 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11874 (secure 0 0 0) 0x7f7ec00bf0c0 con 0x7f7ec8106f50 2024-04-22T06:52:09.896 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:52:09.896 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":62,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:52:03.164497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25865,"mds_2":25855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25855":{"gid":25855,"name":"e","rank":2,"incarnation":55,"state":"up:active","state_seq":26,"addr":"172.21.15.190:6835/1167513219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1167513219},{"type":"v1","addr":"172.21.15.190:6835","nonce":1167513219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25865":{"gid":25865,"name":"f","rank":1,"incarnation":55,"state":"up:active","state_seq":26,"addr":"172.21.15.193:6835/99277862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":99277862},{"type":"v1","addr":"172.21.15.193:6835","nonce":99277862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25927":{"gid":25927,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/3285898346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3285898346},{"type":"v1","addr":"172.21.15.193:6839","nonce":3285898346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25937":{"gid":25937,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1832563615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1832563615},{"type":"v1","addr":"172.21.15.190:6837","nonce":1832563615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25951":{"gid":25951,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/30505216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":30505216},{"type":"v1","addr":"172.21.15.193:6837","nonce":30505216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25855,25865,24467]},"id":1}]} 2024-04-22T06:52:09.896 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 62 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7eb00a3d20 msgr2=0x7f7eb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7eb00a3d20 0x7f7eb00a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80058c0 tx=0x7f7eb8005850 comp rx=0 tx=0).stop 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 msgr2=0x7f7ec819cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 0x7f7ec819cea0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7ec002c700 tx=0x7f7ec00346d0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 shutdown_connections 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ec8106f50 0x7f7ec819cea0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ec8069aa0 0x7f7ec8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7eb00a3d20 0x7f7eb00a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 --2- 172.21.15.79:0/4104605509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ec80690e0 0x7f7ec8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 >> 172.21.15.79:0/4104605509 conn(0x7f7ec8077e30 msgr2=0x7f7ec80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:09.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 shutdown_connections 2024-04-22T06:52:09.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:09.899+0000 7f7eccf32640 1 -- 172.21.15.79:0/4104605509 wait complete. 2024-04-22T06:52:09.924 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 85.7 --await' 2024-04-22T06:52:09.924 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 85.7 --await 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 -- 172.21.15.79:0/962536603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc107260 msgr2=0x7f1fcc10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 --2- 172.21.15.79:0/962536603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc107260 0x7f1fcc10ee30 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009ca0 tx=0x7f1fc402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 -- 172.21.15.79:0/962536603 shutdown_connections 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 --2- 172.21.15.79:0/962536603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1fcc102b40 0x7f1fcc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 --2- 172.21.15.79:0/962536603 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc107260 0x7f1fcc10ee30 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 --2- 172.21.15.79:0/962536603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fcc102190 0x7f1fcc102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 -- 172.21.15.79:0/962536603 >> 172.21.15.79:0/962536603 conn(0x7f1fcc079310 msgr2=0x7f1fcc079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 -- 172.21.15.79:0/962536603 shutdown_connections 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 -- 172.21.15.79:0/962536603 wait complete. 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.073+0000 7f1fd309a640 1 Processor -- start 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 -- start start 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1fcc102190 0x7f1fcc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fcc107260 0x7f1fcc198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 0x7f1fcc19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc112220 con 0x7f1fcc19d000 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1120a0 con 0x7f1fcc102190 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1123a0 con 0x7f1fcc107260 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 0x7f1fcc19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 0x7f1fcc19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42890/0 (socket says 172.21.15.79:42890) 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 -- 172.21.15.79:0/2746035583 learned_addr learned my addr 172.21.15.79:0/2746035583 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:10.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1fcc102190 0x7f1fcc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 -- 172.21.15.79:0/2746035583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fcc107260 msgr2=0x7f1fcc198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fcc107260 0x7f1fcc198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 -- 172.21.15.79:0/2746035583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1fcc102190 msgr2=0x7f1fcc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1fcc102190 0x7f1fcc198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 -- 172.21.15.79:0/2746035583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc19daf0 con 0x7f1fcc19d000 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd1637640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 0x7f1fcc19d4b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40098d0 tx=0x7f1fc4035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fc9ffb640 1 -- 172.21.15.79:0/2746035583 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4039040 con 0x7f1fcc19d000 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fc9ffb640 1 -- 172.21.15.79:0/2746035583 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc4044070 con 0x7f1fcc19d000 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fc9ffb640 1 -- 172.21.15.79:0/2746035583 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40042a0 con 0x7f1fcc19d000 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.074+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fcc19dd80 con 0x7f1fcc19d000 2024-04-22T06:52:10.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.075+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fcc1a5540 con 0x7f1fcc19d000 2024-04-22T06:52:10.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fc9ffb640 1 -- 172.21.15.79:0/2746035583 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1fc4004440 con 0x7f1fcc19d000 2024-04-22T06:52:10.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 Processor -- start 2024-04-22T06:52:10.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 -- start start 2024-04-22T06:52:10.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 0x55e0b6e14da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e0b6d750b0 con 0x55e0b6e149c0 2024-04-22T06:52:10.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e0b6d26680 con 0x55e0b6e1dec0 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e0b6d32c60 con 0x55e0b6e152e0 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fc9ffb640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1fbc0a3d20 0x7f1fbc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 0x55e0b6e14da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 0x55e0b6e14da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42898/0 (socket says 172.21.15.79:42898) 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fcbfff640 1 -- 172.21.15.79:0/3892403509 learned_addr learned my addr 172.21.15.79:0/3892403509 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.077+0000 7f1fd1637640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1fbc0a3d20 0x7f1fbc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fcbfff640 1 -- 172.21.15.79:0/3892403509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 msgr2=0x55e0b6e1d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fc9ffb640 1 -- 172.21.15.79:0/2746035583 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9417+0+0 (secure 0 0 0) 0x7f1fc40f3090 con 0x7f1fcc19d000 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fcbfff640 1 -- 172.21.15.79:0/3892403509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e1dec0 msgr2=0x55e0b6e202b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fcbfff640 1 -- 172.21.15.79:0/3892403509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0b6b57c90 con 0x55e0b6e149c0 2024-04-22T06:52:10.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 0x55e0b6e14da0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800aa60 tx=0x7f1fb800af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db4cfb97c914e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fd1637640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1fbc0a3d20 0x7f1fbc0a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40beac0 tx=0x7f1fc4090800 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fb6ffd640 1 -- 172.21.15.79:0/3892403509 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb800ee60 con 0x55e0b6e149c0 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fb6ffd640 1 -- 172.21.15.79:0/3892403509 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb80035f0 con 0x55e0b6e149c0 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.078+0000 7f1fb6ffd640 1 -- 172.21.15.79:0/3892403509 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8010660 con 0x55e0b6e149c0 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3892403509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 msgr2=0x55e0b6e14da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 0x55e0b6e14da0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800aa60 tx=0x7f1fb800af20 comp rx=0 tx=0).stop 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3892403509 shutdown_connections 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e202b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3892403509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e149c0 0x55e0b6e14da0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3892403509 >> 172.21.15.79:0/3892403509 conn(0x55e0b6e0f0f0 msgr2=0x55e0b6e0f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3892403509 shutdown_connections 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3892403509 wait complete. 2024-04-22T06:52:10.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- start start 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 Processor -- start 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e1afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e1ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e0b6d78400 con 0x55e0b6e152e0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e0b6d34840 con 0x55e0b6e149c0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.079+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e0b6d7d720 con 0x55e0b6e1dec0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd5d6f740 10 client.0 _ll_register_callbacks cb 0x55e0b6d85ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42906/0 (socket says 172.21.15.79:42906) 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 -- 172.21.15.79:0/3270454263 learned_addr learned my addr 172.21.15.79:0/3270454263 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e1ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e1afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 -- 172.21.15.79:0/3270454263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e1dec0 msgr2=0x55e0b6e1ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e1ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 -- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 msgr2=0x55e0b6e1afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e1afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 -- 172.21.15.79:0/3270454263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0b6bb8420 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b4f0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000e2d0 tx=0x7f1fc000e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8186b3605ebc4a27 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fabfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fabfff640 1 -- 172.21.15.79:0/3270454263 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0018020 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fabfff640 1 -- 172.21.15.79:0/3270454263 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9417+0+0 (secure 0 0 0) 0x7f1fc0013660 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fb80035f0 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fabfff640 1 -- 172.21.15.79:0/3270454263 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f1fc0023050 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd5d6f740 10 client.17145 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.080+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55e0b6a4b580 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fabfff640 1 -- 172.21.15.79:0/3270454263 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb80035f0 con 0x55e0b6e152e0 2024-04-22T06:52:10.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fabfff640 1 -- 172.21.15.79:0/3270454263 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f1fc0014990 con 0x55e0b6e152e0 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fd5d6f740 10 client.17145 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fd5d6f740 10 client.17145 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fd5d6f740 20 client.17145 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e2dd00 0x55e0b6e30150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fd5d6f740 4 client.17145 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0b6d86b10 con 0x55e0b6e2dd00 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e2dd00 0x55e0b6e30150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fabfff640 10 client.17145 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.081+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e2dd00 0x55e0b6e30150 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2395ad2b60e1b73a server_cookie=9ad4d8deac8337d3 in_seq=0 out_seq=0 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fabfff640 1 -- 172.21.15.79:0/3270454263 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0b6d86b10 con 0x55e0b6e2dd00 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fabfff640 10 client.17145 handle_command_reply: tid=0 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 client.17145 shutdown 2024-04-22T06:52:10.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 msgr2=0x55e0b6e1b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b4f0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000e2d0 tx=0x7f1fc000e790 comp rx=0 tx=0).stop 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 shutdown_connections 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e2dd00 0x55e0b6e30150 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e1ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fabfff640 0 client.17145 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b4f0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/3270454263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e1afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.082+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 >> 172.21.15.79:0/3270454263 conn(0x55e0b6e0f0f0 msgr2=0x55e0b6e10e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.083+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 shutdown_connections 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.083+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/3270454263 wait complete. 2024-04-22T06:52:10.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.083+0000 7f1fd5d6f740 20 client.17145 trim_cache size 0 max 16384 2024-04-22T06:52:10.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 Processor -- start 2024-04-22T06:52:10.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 -- start start 2024-04-22T06:52:10.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e298a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e35a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1ae610 con 0x55e0b6e1dec0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1ac480 con 0x55e0b6e149c0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1adf40 con 0x55e0b6e152e0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e35a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e35a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42918/0 (socket says 172.21.15.79:42918) 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 -- 172.21.15.79:0/1303574749 learned_addr learned my addr 172.21.15.79:0/1303574749 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 -- 172.21.15.79:0/1303574749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 msgr2=0x55e0b6e1b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 -- 172.21.15.79:0/1303574749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 msgr2=0x55e0b6e298a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e298a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 -- 172.21.15.79:0/1303574749 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0b6e0e840 con 0x55e0b6e1dec0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e35a60 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8005890 tx=0x7f1fb800a3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102c8632758af457 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fb67fc640 1 -- 172.21.15.79:0/1303574749 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb80043f0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fb67fc640 1 -- 172.21.15.79:0/1303574749 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb80288c0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.100+0000 7f1fb67fc640 1 -- 172.21.15.79:0/1303574749 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8010400 con 0x55e0b6e1dec0 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/1303574749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 msgr2=0x55e0b6e35a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e35a60 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8005890 tx=0x7f1fb800a3b0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/1303574749 shutdown_connections 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e35a60 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e1b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/1303574749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e298a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/1303574749 >> 172.21.15.79:0/1303574749 conn(0x55e0b6e0f0f0 msgr2=0x55e0b6e1af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/1303574749 shutdown_connections 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/1303574749 wait complete. 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 -- start start 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.101+0000 7f1fd5d6f740 1 Processor -- start 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e22800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e22d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e23280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1ad1a0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1acad0 con 0x55e0b6e149c0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1ad870 con 0x55e0b6e152e0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 10 client.0 _ll_register_callbacks cb 0x7f1fc000c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e23280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e23280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42930/0 (socket says 172.21.15.79:42930) 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 -- 172.21.15.79:0/4105550710 learned_addr learned my addr 172.21.15.79:0/4105550710 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e22d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 -- 172.21.15.79:0/4105550710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 msgr2=0x55e0b6e22d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e22d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 -- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 msgr2=0x55e0b6e22800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e22800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 -- 172.21.15.79:0/4105550710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0b6e94310 con 0x55e0b6e1dec0 2024-04-22T06:52:10.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e22d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fcbfff640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e23280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8005f60 tx=0x7f1fb800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49cce0ec2c531a84 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fabfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fabfff640 1 -- 172.21.15.79:0/4105550710 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003ab0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fabfff640 1 -- 172.21.15.79:0/4105550710 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9417+0+0 (secure 0 0 0) 0x7f1fb8003f40 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.102+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fb80288c0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fabfff640 1 -- 172.21.15.79:0/4105550710 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f1fb803d050 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 10 client.17151 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55e0b6e955d0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fabfff640 1 -- 172.21.15.79:0/4105550710 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fb80288c0 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fabfff640 1 -- 172.21.15.79:0/4105550710 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f1fb800f040 con 0x55e0b6e1dec0 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 10 client.17151 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 10 client.17151 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 20 client.17151 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e95ac0 0x55e0b6e97eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:10.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 4 client.17151 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.7, "await": true}] 2024-04-22T06:52:10.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.7, "await": true}) -- 0x55e0b6d80980 con 0x55e0b6e95ac0 2024-04-22T06:52:10.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.103+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e95ac0 0x55e0b6e97eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:10.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.104+0000 7f1fabfff640 10 client.17151 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:10.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.104+0000 7f1fd0e36640 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e95ac0 0x55e0b6e97eb0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9b44d5871b6775a server_cookie=2f0b560a664ef217 in_seq=0 out_seq=0 2024-04-22T06:52:10.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.129+0000 7f1fabfff640 1 -- 172.21.15.79:0/4105550710 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e0b6d80980 con 0x55e0b6e95ac0 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.129+0000 7f1fabfff640 10 client.17151 handle_command_reply: tid=0 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.129+0000 7f1fd5d6f740 1 client.17151 shutdown 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 msgr2=0x55e0b6e23280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e23280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8005f60 tx=0x7f1fb800a620 comp rx=0 tx=0).stop 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 shutdown_connections 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e0b6e95ac0 0x55e0b6e97eb0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e0b6e1dec0 0x55e0b6e23280 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0b6e152e0 0x55e0b6e22d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 --2- 172.21.15.79:0/4105550710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0b6e149c0 0x55e0b6e22800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fabfff640 0 client.17151 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 >> 172.21.15.79:0/4105550710 conn(0x55e0b6e0f0f0 msgr2=0x55e0b6e1b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:10.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 shutdown_connections 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 1 -- 172.21.15.79:0/4105550710 wait complete. 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd5d6f740 20 client.17151 trim_cache size 0 max 16384 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 62, 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 24, 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "3537766f": { 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 24, 2024-04-22T06:52:10.130 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 85.7, 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:10.131 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:10.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:10.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:10.132 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:10.132 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1fbc0a3d20 msgr2=0x7f1fbc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1fbc0a3d20 0x7f1fbc0a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40beac0 tx=0x7f1fc4090800 comp rx=0 tx=0).stop 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 msgr2=0x7f1fcc19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 0x7f1fcc19d4b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40098d0 tx=0x7f1fc4035d60 comp rx=0 tx=0).stop 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 shutdown_connections 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fcc19d000 0x7f1fcc19d4b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1fcc107260 0x7f1fcc198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1fbc0a3d20 0x7f1fbc0a61e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 --2- 172.21.15.79:0/2746035583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1fcc102190 0x7f1fcc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.130+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 >> 172.21.15.79:0/2746035583 conn(0x7f1fcc079310 msgr2=0x7f1fcc100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.131+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 shutdown_connections 2024-04-22T06:52:10.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:10.131+0000 7f1fd309a640 1 -- 172.21.15.79:0/2746035583 wait complete. 2024-04-22T06:52:10.140 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 3537766f, quiesce took 0.7 seconds. Will release after: 25.7 2024-04-22T06:52:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:10 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4104605509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:10.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:10 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4104605509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4104605509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:11.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:11 smithi193 ceph-mon[31815]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 541 MiB/s wr, 927 op/s 2024-04-22T06:52:11.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:11 smithi190 ceph-mon[31421]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 541 MiB/s wr, 927 op/s 2024-04-22T06:52:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:11 smithi079 ceph-mon[27946]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 541 MiB/s wr, 927 op/s 2024-04-22T06:52:13.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:13 smithi193 ceph-mon[31815]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 404 MiB/s wr, 829 op/s 2024-04-22T06:52:13.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:13 smithi190 ceph-mon[31421]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 404 MiB/s wr, 829 op/s 2024-04-22T06:52:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:13 smithi079 ceph-mon[27946]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 404 MiB/s wr, 829 op/s 2024-04-22T06:52:14.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:14 smithi079 ceph-mon[27946]: pgmap v250: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 328 MiB/s wr, 674 op/s 2024-04-22T06:52:14.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:14 smithi193 ceph-mon[31815]: pgmap v250: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 328 MiB/s wr, 674 op/s 2024-04-22T06:52:14.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:14 smithi190 ceph-mon[31421]: pgmap v250: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 328 MiB/s wr, 674 op/s 2024-04-22T06:52:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 -- 172.21.15.79:0/744379663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4778102180 msgr2=0x7f4778105ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 --2- 172.21.15.79:0/744379663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4778102180 0x7f4778105ca0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f476c009900 tx=0x7f476c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 -- 172.21.15.79:0/744379663 shutdown_connections 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 --2- 172.21.15.79:0/744379663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4778106380 0x7f477810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 --2- 172.21.15.79:0/744379663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4778102180 0x7f4778105ca0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 --2- 172.21.15.79:0/744379663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47781017d0 0x7f4778101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 -- 172.21.15.79:0/744379663 >> 172.21.15.79:0/744379663 conn(0x7f47780773e0 msgr2=0x7f47780fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 -- 172.21.15.79:0/744379663 shutdown_connections 2024-04-22T06:52:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.609+0000 7f477e625640 1 -- 172.21.15.79:0/744379663 wait complete. 2024-04-22T06:52:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.610+0000 7f477e625640 1 Processor -- start 2024-04-22T06:52:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.610+0000 7f477e625640 1 -- start start 2024-04-22T06:52:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.610+0000 7f477e625640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 0x7f477810e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.610+0000 7f477e625640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4778106380 0x7f477810ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.610+0000 7f477e625640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f477810f360 0x7f47781a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f477e625640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f47781124e0 con 0x7f47781017d0 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f477e625640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4778112360 con 0x7f477810f360 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f477e625640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4778112660 con 0x7f4778106380 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 0x7f477810e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 0x7f477810e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46930/0 (socket says 172.21.15.79:46930) 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 -- 172.21.15.79:0/3260032112 learned_addr learned my addr 172.21.15.79:0/3260032112 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 -- 172.21.15.79:0/3260032112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4778106380 msgr2=0x7f477810ee20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4778106380 0x7f477810ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 -- 172.21.15.79:0/3260032112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f477810f360 msgr2=0x7f47781a6c30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f477810f360 0x7f47781a6c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 -- 172.21.15.79:0/3260032112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47781a7330 con 0x7f47781017d0 2024-04-22T06:52:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.611+0000 7f4777fff640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 0x7f477810e8e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f476800b450 tx=0x7f476800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.612+0000 7f47757fa640 1 -- 172.21.15.79:0/3260032112 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768015020 con 0x7f47781017d0 2024-04-22T06:52:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.612+0000 7f47757fa640 1 -- 172.21.15.79:0/3260032112 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4768011d40 con 0x7f47781017d0 2024-04-22T06:52:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.612+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47781a7620 con 0x7f47781017d0 2024-04-22T06:52:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.612+0000 7f47757fa640 1 -- 172.21.15.79:0/3260032112 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476800f340 con 0x7f47781017d0 2024-04-22T06:52:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.612+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47781a7bc0 con 0x7f47781017d0 2024-04-22T06:52:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.615+0000 7f47757fa640 1 -- 172.21.15.79:0/3260032112 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4768018020 con 0x7f47781017d0 2024-04-22T06:52:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.615+0000 7f4781214740 1 Processor -- start 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.615+0000 7f47757fa640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47600a3ef0 0x7f47600a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.615+0000 7f477cbc2640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47600a3ef0 0x7f47600a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.615+0000 7f47757fa640 1 -- 172.21.15.79:0/3260032112 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f47680d0550 con 0x7f47781017d0 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.615+0000 7f4781214740 1 -- start start 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4781214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bea8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4781214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea8b60 0x563a0bea8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4781214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0beb44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4781214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563a0be0d8d0 con 0x563a0bea98d0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4781214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563a0bdaae80 con 0x563a0beb20e0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4781214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563a0be09890 con 0x563a0bea8b60 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f477cbc2640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47600a3ef0 0x7f47600a63b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f477000ab30 tx=0x7f4770009250 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bea8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bea8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46946/0 (socket says 172.21.15.79:46946) 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 -- 172.21.15.79:0/1530899433 learned_addr learned my addr 172.21.15.79:0/1530899433 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 -- 172.21.15.79:0/1530899433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea8b60 msgr2=0x563a0bea8fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea8b60 0x563a0bea8fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 -- 172.21.15.79:0/1530899433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 msgr2=0x563a0beb44d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0beb44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.616+0000 7f4777fff640 1 -- 172.21.15.79:0/1530899433 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a0bdcc320 con 0x563a0bea98d0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.617+0000 7f4777fff640 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bea8620 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f47680986b0 tx=0x7f4768010ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabead302dc10054 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.617+0000 7f475e7fc640 1 -- 172.21.15.79:0/1530899433 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47680b5070 con 0x563a0bea98d0 2024-04-22T06:52:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.617+0000 7f475e7fc640 1 -- 172.21.15.79:0/1530899433 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47680a0600 con 0x563a0bea98d0 2024-04-22T06:52:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.617+0000 7f475e7fc640 1 -- 172.21.15.79:0/1530899433 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47680cdb70 con 0x563a0bea98d0 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 -- 172.21.15.79:0/1530899433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 msgr2=0x563a0bea8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bea8620 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f47680986b0 tx=0x7f4768010ff0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 -- 172.21.15.79:0/1530899433 shutdown_connections 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0beb44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea8b60 0x563a0bea8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 --2- 172.21.15.79:0/1530899433 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bea8620 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 -- 172.21.15.79:0/1530899433 >> 172.21.15.79:0/1530899433 conn(0x563a0be6fc90 msgr2=0x563a0bea3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:15.620 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 -- 172.21.15.79:0/1530899433 shutdown_connections 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 -- 172.21.15.79:0/1530899433 wait complete. 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 -- start start 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.618+0000 7f4781214740 1 Processor -- start 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 0x563a0beaee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0beaf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563a0beb4e10 con 0x563a0bea8b60 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563a0bd95eb0 con 0x563a0beb20e0 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563a0beb4f90 con 0x563a0bea98d0 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 10 client.0 _ll_register_callbacks cb 0x563a0bdc2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46950/0 (socket says 172.21.15.79:46950) 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 -- 172.21.15.79:0/1689053336 learned_addr learned my addr 172.21.15.79:0/1689053336 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:15.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 -- 172.21.15.79:0/1689053336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 msgr2=0x563a0beaee70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 0x563a0beaee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 -- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 msgr2=0x563a0beaf480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0beaf480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 -- 172.21.15.79:0/1689053336 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a0bb20210 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4777fff640 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb1ae0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f476809c410 tx=0x7f4768010730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1904bf4cac7e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f47577fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f47577fe640 1 -- 172.21.15.79:0/1689053336 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47680b9020 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f47577fe640 1 -- 172.21.15.79:0/1689053336 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f47680cdb70 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.619+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47680a0600 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f47577fe640 1 -- 172.21.15.79:0/1689053336 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f4768084050 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 10 client.17163 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x563a0bbec620 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f47577fe640 1 -- 172.21.15.79:0/1689053336 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47680a0600 con 0x563a0bea8b60 2024-04-22T06:52:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f47577fe640 1 -- 172.21.15.79:0/1689053336 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f47680a4a30 con 0x563a0bea8b60 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 10 client.17163 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 10 client.17163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 20 client.17163 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bec43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 4 client.17163 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.620+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a0be21880 con 0x563a0bec1fc0 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.622+0000 7f47777fe640 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bec43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.624+0000 7f47577fe640 10 client.17163 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.624+0000 7f47777fe640 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bec43b0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e71b48d9c4452ef server_cookie=34302c02bc72fab3 in_seq=0 out_seq=0 2024-04-22T06:52:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.626+0000 7f47577fe640 1 -- 172.21.15.79:0/1689053336 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a0be21880 con 0x563a0bec1fc0 2024-04-22T06:52:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.626+0000 7f47577fe640 10 client.17163 handle_command_reply: tid=0 2024-04-22T06:52:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.626+0000 7f4781214740 1 client.17163 shutdown 2024-04-22T06:52:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.626+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 msgr2=0x563a0beb1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.626+0000 7f4781214740 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb1ae0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f476809c410 tx=0x7f4768010730 comp rx=0 tx=0).stop 2024-04-22T06:52:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 shutdown_connections 2024-04-22T06:52:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bec43b0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f47577fe640 0 client.17163 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0beaf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 0x563a0beaee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 --2- 172.21.15.79:0/1689053336 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb1ae0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 >> 172.21.15.79:0/1689053336 conn(0x563a0be6fc90 msgr2=0x563a0bea55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 shutdown_connections 2024-04-22T06:52:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 1 -- 172.21.15.79:0/1689053336 wait complete. 2024-04-22T06:52:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.627+0000 7f4781214740 20 client.17163 trim_cache size 0 max 16384 2024-04-22T06:52:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 Processor -- start 2024-04-22T06:52:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 -- start start 2024-04-22T06:52:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 0x563a0beb1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0bec73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f47781b3000 con 0x563a0bea8b60 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47781b08d0 con 0x563a0beb20e0 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4781214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47781ae740 con 0x563a0bea98d0 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46952/0 (socket says 172.21.15.79:46952) 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 -- 172.21.15.79:0/4050282475 learned_addr learned my addr 172.21.15.79:0/4050282475 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 -- 172.21.15.79:0/4050282475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 msgr2=0x563a0beb1ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 0x563a0beb1ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 -- 172.21.15.79:0/4050282475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 msgr2=0x563a0bec73d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0bec73d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.648+0000 7f4777fff640 1 -- 172.21.15.79:0/4050282475 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a0beaf7f0 con 0x563a0bea8b60 2024-04-22T06:52:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.649+0000 7f4777fff640 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb15a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f476809a500 tx=0x7f4768010400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae3082932d16bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.649+0000 7f475dffb640 1 -- 172.21.15.79:0/4050282475 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47680b9070 con 0x563a0bea8b60 2024-04-22T06:52:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.649+0000 7f475dffb640 1 -- 172.21.15.79:0/4050282475 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47680a0600 con 0x563a0bea8b60 2024-04-22T06:52:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.649+0000 7f475dffb640 1 -- 172.21.15.79:0/4050282475 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47680b5040 con 0x563a0bea8b60 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.650+0000 7f4781214740 1 -- 172.21.15.79:0/4050282475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 msgr2=0x563a0beb15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.650+0000 7f4781214740 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb15a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f476809a500 tx=0x7f4768010400 comp rx=0 tx=0).stop 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 -- 172.21.15.79:0/4050282475 shutdown_connections 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0beb20e0 0x563a0bec73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0bea98d0 0x563a0beb1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 --2- 172.21.15.79:0/4050282475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea8b60 0x563a0beb15a0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 -- 172.21.15.79:0/4050282475 >> 172.21.15.79:0/4050282475 conn(0x563a0be6fc90 msgr2=0x563a0bead740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 -- 172.21.15.79:0/4050282475 shutdown_connections 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 -- 172.21.15.79:0/4050282475 wait complete. 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.651+0000 7f4781214740 1 -- start start 2024-04-22T06:52:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 Processor -- start 2024-04-22T06:52:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0bea8b60 0x563a0bec78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bec4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0beb20e0 0x563a0bec5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f47781aed90 con 0x563a0bea98d0 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47781b0200 con 0x563a0bea8b60 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f47777fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bec4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f4781214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47781af460 con 0x563a0beb20e0 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f47777fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bec4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46968/0 (socket says 172.21.15.79:46968) 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.652+0000 7f47777fe640 1 -- 172.21.15.79:0/3131952517 learned_addr learned my addr 172.21.15.79:0/3131952517 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f4781214740 10 client.0 _ll_register_callbacks cb 0x7f47680b3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47777fe640 1 -- 172.21.15.79:0/3131952517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0beb20e0 msgr2=0x563a0bec5400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47777fe640 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0beb20e0 0x563a0bec5400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47777fe640 1 -- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0bea8b60 msgr2=0x563a0bec78e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47777fe640 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0bea8b60 0x563a0bec78e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47777fe640 1 -- 172.21.15.79:0/3131952517 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a0bf271a0 con 0x563a0bea98d0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47777fe640 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bec4cc0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f476c009230 tx=0x7f476c005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f43c371f6ce663 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47577fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47577fe640 1 -- 172.21.15.79:0/3131952517 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0044d0 con 0x563a0bea98d0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f47577fe640 1 -- 172.21.15.79:0/3131952517 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f476c02c830 con 0x563a0bea98d0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.653+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47680a0600 con 0x563a0bea98d0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.654+0000 7f47577fe640 1 -- 172.21.15.79:0/3131952517 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f476c053050 con 0x563a0bea98d0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.654+0000 7f4781214740 10 client.17169 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.654+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x563a0bf28340 con 0x563a0bea98d0 2024-04-22T06:52:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.654+0000 7f47577fe640 1 -- 172.21.15.79:0/3131952517 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f476c047030 con 0x563a0bea98d0 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f47577fe640 1 -- 172.21.15.79:0/3131952517 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f476c0049c0 con 0x563a0bea98d0 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f4781214740 10 client.17169 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f4781214740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f4781214740 20 client.17169 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f4781214740 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bf2aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f4781214740 4 client.17169 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.655+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f47680a0600 con 0x563a0bec1fc0 2024-04-22T06:52:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.656+0000 7f4777fff640 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bf2aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.658+0000 7f4777fff640 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bf2aae0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b1ccd5ab3695b2e server_cookie=b8b303b9feaa0c69 in_seq=0 out_seq=0 2024-04-22T06:52:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.658+0000 7f47577fe640 10 client.17169 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.687+0000 7f47577fe640 1 -- 172.21.15.79:0/3131952517 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f47680a0600 con 0x563a0bec1fc0 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f47577fe640 10 client.17169 handle_command_reply: tid=0 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 client.17169 shutdown 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 msgr2=0x563a0bec4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bec4cc0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f476c009230 tx=0x7f476c005e50 comp rx=0 tx=0).stop 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 shutdown_connections 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563a0bec1fc0 0x563a0bf2aae0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a0beb20e0 0x563a0bec5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563a0bea98d0 0x563a0bec4cc0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 --2- 172.21.15.79:0/3131952517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563a0bea8b60 0x563a0bec78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 >> 172.21.15.79:0/3131952517 conn(0x563a0be6fc90 msgr2=0x563a0bea5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:15.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.688+0000 7f47577fe640 0 client.17169 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.689+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 shutdown_connections 2024-04-22T06:52:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.689+0000 7f4781214740 1 -- 172.21.15.79:0/3131952517 wait complete. 2024-04-22T06:52:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.689+0000 7f4781214740 20 client.17169 trim_cache size 0 max 16384 2024-04-22T06:52:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:52:15.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:52:15.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:52:15.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:52:15.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47600a3ef0 msgr2=0x7f47600a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47600a3ef0 0x7f47600a63b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f477000ab30 tx=0x7f4770009250 comp rx=0 tx=0).stop 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 msgr2=0x7f477810e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 0x7f477810e8e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f476800b450 tx=0x7f476800b910 comp rx=0 tx=0).stop 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 shutdown_connections 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f47600a3ef0 0x7f47600a63b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f477810f360 0x7f47781a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4778106380 0x7f477810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 --2- 172.21.15.79:0/3260032112 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f47781017d0 0x7f477810e8e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 >> 172.21.15.79:0/3260032112 conn(0x7f47780773e0 msgr2=0x7f47780fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 shutdown_connections 2024-04-22T06:52:15.690 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:15.690+0000 7f477e625640 1 -- 172.21.15.79:0/3260032112 wait complete. 2024-04-22T06:52:15.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:15 smithi193 ceph-mon[31815]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:52:15.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:15 smithi190 ceph-mon[31421]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:52:15.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:15 smithi079 ceph-mon[27946]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:52:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:16 smithi193 ceph-mon[31815]: pgmap v252: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 287 MiB/s wr, 642 op/s 2024-04-22T06:52:16.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:16 smithi190 ceph-mon[31421]: pgmap v252: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 287 MiB/s wr, 642 op/s 2024-04-22T06:52:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:16 smithi079 ceph-mon[27946]: pgmap v252: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 287 MiB/s wr, 642 op/s 2024-04-22T06:52:18.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:18 smithi193 ceph-mon[31815]: pgmap v253: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 226 MiB/s wr, 1.23k op/s 2024-04-22T06:52:18.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:18 smithi190 ceph-mon[31421]: pgmap v253: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 226 MiB/s wr, 1.23k op/s 2024-04-22T06:52:18.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:18 smithi079 ceph-mon[27946]: pgmap v253: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 226 MiB/s wr, 1.23k op/s 2024-04-22T06:52:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:20 smithi193 ceph-mon[31815]: pgmap v254: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 238 MiB/s wr, 1.59k op/s 2024-04-22T06:52:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:20 smithi190 ceph-mon[31421]: pgmap v254: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 238 MiB/s wr, 1.59k op/s 2024-04-22T06:52:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:20 smithi079 ceph-mon[27946]: pgmap v254: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 238 MiB/s wr, 1.59k op/s 2024-04-22T06:52:22.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:22 smithi193 ceph-mon[31815]: pgmap v255: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 MiB/s wr, 1.68k op/s 2024-04-22T06:52:22.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:22 smithi190 ceph-mon[31421]: pgmap v255: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 MiB/s wr, 1.68k op/s 2024-04-22T06:52:22.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:22 smithi079 ceph-mon[27946]: pgmap v255: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 MiB/s wr, 1.68k op/s 2024-04-22T06:52:24.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:24 smithi193 ceph-mon[31815]: pgmap v256: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 249 MiB/s wr, 1.68k op/s 2024-04-22T06:52:24.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:24 smithi190 ceph-mon[31421]: pgmap v256: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 249 MiB/s wr, 1.68k op/s 2024-04-22T06:52:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:24 smithi079 ceph-mon[27946]: pgmap v256: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 249 MiB/s wr, 1.68k op/s 2024-04-22T06:52:26.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:26 smithi193 ceph-mon[31815]: pgmap v257: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 229 MiB/s wr, 1.54k op/s 2024-04-22T06:52:26.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:26 smithi190 ceph-mon[31421]: pgmap v257: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 229 MiB/s wr, 1.54k op/s 2024-04-22T06:52:26.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:26 smithi079 ceph-mon[27946]: pgmap v257: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 229 MiB/s wr, 1.54k op/s 2024-04-22T06:52:28.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:28 smithi193 ceph-mon[31815]: pgmap v258: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 207 MiB/s wr, 1.39k op/s 2024-04-22T06:52:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:28 smithi190 ceph-mon[31421]: pgmap v258: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 207 MiB/s wr, 1.39k op/s 2024-04-22T06:52:28.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:28 smithi079 ceph-mon[27946]: pgmap v258: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 207 MiB/s wr, 1.39k op/s 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 -- 172.21.15.79:0/2374852925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1072a0 msgr2=0x7fb1ac10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 --2- 172.21.15.79:0/2374852925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac10ee70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009ca0 tx=0x7fb1a402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 -- 172.21.15.79:0/2374852925 shutdown_connections 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 --2- 172.21.15.79:0/2374852925 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac10ee70 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 --2- 172.21.15.79:0/2374852925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac102b80 0x7fb1ac106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 --2- 172.21.15.79:0/2374852925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 -- 172.21.15.79:0/2374852925 >> 172.21.15.79:0/2374852925 conn(0x7fb1ac0793e0 msgr2=0x7fb1ac0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 -- 172.21.15.79:0/2374852925 shutdown_connections 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.794+0000 7fb1b4211640 1 -- 172.21.15.79:0/2374852925 wait complete. 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 Processor -- start 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 -- start start 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac074cd0 0x7fb1ac1a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac112210 con 0x7fb1ac1021d0 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac112090 con 0x7fb1ac1072a0 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b4211640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac112390 con 0x7fb1ac074cd0 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b1fad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b1fad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43084/0 (socket says 172.21.15.79:43084) 2024-04-22T06:52:28.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.795+0000 7fb1b1fad640 1 -- 172.21.15.79:0/57574366 learned_addr learned my addr 172.21.15.79:0/57574366 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b1fad640 1 -- 172.21.15.79:0/57574366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac074cd0 msgr2=0x7fb1ac1a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac074cd0 0x7fb1ac1a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac074cd0 0x7fb1ac1a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b1fad640 1 -- 172.21.15.79:0/57574366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1072a0 msgr2=0x7fb1ac074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b1fad640 1 -- 172.21.15.79:0/57574366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ac1a6f20 con 0x7fb1ac1021d0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac074cd0 0x7fb1ac1a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac074250 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800e9f0 tx=0x7fb1a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1a2ffd640 1 -- 172.21.15.79:0/57574366 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a800cd80 con 0x7fb1ac1021d0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1a2ffd640 1 -- 172.21.15.79:0/57574366 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a80108e0 con 0x7fb1ac1021d0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ac1a7210 con 0x7fb1ac1021d0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1a2ffd640 1 -- 172.21.15.79:0/57574366 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8018980 con 0x7fb1ac1021d0 2024-04-22T06:52:28.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.796+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ac1a7760 con 0x7fb1ac1021d0 2024-04-22T06:52:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.798+0000 7fb1a2ffd640 1 -- 172.21.15.79:0/57574366 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb1a8010430 con 0x7fb1ac1021d0 2024-04-22T06:52:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 Processor -- start 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 -- start start 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db9849620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9849fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db98530e0 0x564db98554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564db97ae8d0 con 0x564db984a8d0 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564db974be80 con 0x564db98530e0 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.799+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564db97aa890 con 0x564db9849b60 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1a2ffd640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb1980a3df0 0x7fb1980a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db9849620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db9849620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43098/0 (socket says 172.21.15.79:43098) 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2538673627 learned_addr learned my addr 172.21.15.79:0/2538673627 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb1980a3df0 0x7fb1980a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db98530e0 0x564db98554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9849fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1a2ffd640 1 -- 172.21.15.79:0/57574366 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7fb1a80d0fe0 con 0x7fb1ac1021d0 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2538673627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 msgr2=0x564db9849fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9849fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2538673627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db98530e0 msgr2=0x564db98554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db98530e0 0x564db98554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2538673627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db976d320 con 0x564db984a8d0 2024-04-22T06:52:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db98530e0 0x564db98554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9849fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db9849620 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4032830 tx=0x7fb1a4032860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a10692b2eb92e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.800+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb1980a3df0 0x7fb1980a62b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4005e30 tx=0x7fb1a4040000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb197fff640 1 -- 172.21.15.79:0/2538673627 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4050070 con 0x564db984a8d0 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb197fff640 1 -- 172.21.15.79:0/2538673627 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a403a040 con 0x564db984a8d0 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb197fff640 1 -- 172.21.15.79:0/2538673627 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4039040 con 0x564db984a8d0 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2538673627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 msgr2=0x564db9849620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db9849620 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4032830 tx=0x7fb1a4032860 comp rx=0 tx=0).stop 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2538673627 shutdown_connections 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db98530e0 0x564db98554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9849fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2538673627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db9849620 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2538673627 >> 172.21.15.79:0/2538673627 conn(0x564db9810c90 msgr2=0x564db9844850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2538673627 shutdown_connections 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.801+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2538673627 wait complete. 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 -- start start 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 Processor -- start 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9852b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db984fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db98504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564db9855db0 con 0x564db98530e0 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564db9736eb0 con 0x564db984a8d0 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564db9855f30 con 0x564db9849b60 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b6e00740 10 client.0 _ll_register_callbacks cb 0x564db9763b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b27ae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db98504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b27ae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db98504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43108/0 (socket says 172.21.15.79:43108) 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.802+0000 7fb1b27ae640 1 -- 172.21.15.79:0/3458635455 learned_addr learned my addr 172.21.15.79:0/3458635455 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:28.809 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db984fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9852b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b27ae640 1 -- 172.21.15.79:0/3458635455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 msgr2=0x564db9852b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9852b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b27ae640 1 -- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 msgr2=0x564db984fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db984fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b27ae640 1 -- 172.21.15.79:0/3458635455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db94c1210 con 0x564db98530e0 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db984fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9852b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db98504a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb1a402f020 tx=0x7fb1a4032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c44b3dc73f3566 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1957fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:28.810 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1957fa640 1 -- 172.21.15.79:0/3458635455 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4050070 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1957fa640 1 -- 172.21.15.79:0/3458635455 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7fb1a4004450 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a403a040 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1957fa640 1 -- 172.21.15.79:0/3458635455 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fb1a405e050 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b6e00740 10 client.17181 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x564db958d620 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.803+0000 7fb1957fa640 1 -- 172.21.15.79:0/3458635455 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1a403a040 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1957fa640 1 -- 172.21.15.79:0/3458635455 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fb1a4055c40 con 0x564db98530e0 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b6e00740 10 client.17181 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b6e00740 10 client.17181 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b6e00740 20 client.17181 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db9862f10 0x564db9865300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.811 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b6e00740 4 client.17181 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564db97c2880 con 0x564db9862f10 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db9862f10 0x564db9865300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db9862f10 0x564db9865300 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68df895cb3e9be0e server_cookie=ea3203634a34246 in_seq=0 out_seq=0 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.804+0000 7fb1957fa640 10 client.17181 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1957fa640 1 -- 172.21.15.79:0/3458635455 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564db97c2880 con 0x564db9862f10 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1957fa640 10 client.17181 handle_command_reply: tid=0 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 client.17181 shutdown 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 msgr2=0x564db98504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db98504a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb1a402f020 tx=0x7fb1a4032d50 comp rx=0 tx=0).stop 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 shutdown_connections 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db9862f10 0x564db9865300 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db984fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.812 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db98504a0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1957fa640 0 client.17181 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/3458635455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db9852b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.805+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 >> 172.21.15.79:0/3458635455 conn(0x564db9810c90 msgr2=0x564db98465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.806+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 shutdown_connections 2024-04-22T06:52:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.806+0000 7fb1b6e00740 1 -- 172.21.15.79:0/3458635455 wait complete. 2024-04-22T06:52:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.806+0000 7fb1b6e00740 20 client.17181 trim_cache size 0 max 16384 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.820+0000 7fb1b6e00740 1 Processor -- start 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.820+0000 7fb1b6e00740 1 -- start start 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db9849b60 0x564db985ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db984b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db98530e0 0x564db98525c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac1b2dd0 con 0x564db984a8d0 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac1b06a0 con 0x564db9849b60 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac1ae510 con 0x564db98530e0 2024-04-22T06:52:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db984b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db984b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43118/0 (socket says 172.21.15.79:43118) 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 -- 172.21.15.79:0/2284115961 learned_addr learned my addr 172.21.15.79:0/2284115961 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db9849b60 0x564db985ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 -- 172.21.15.79:0/2284115961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db98530e0 msgr2=0x564db98525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db98530e0 0x564db98525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 -- 172.21.15.79:0/2284115961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db9849b60 msgr2=0x564db985ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db9849b60 0x564db985ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 -- 172.21.15.79:0/2284115961 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db9848e30 con 0x564db984a8d0 2024-04-22T06:52:28.825 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db9849b60 0x564db985ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db984b650 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0058c0 tx=0x7fb19c00be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ae66efe319aafd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1977fe640 1 -- 172.21.15.79:0/2284115961 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c011a30 con 0x564db984a8d0 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1977fe640 1 -- 172.21.15.79:0/2284115961 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb19c011bd0 con 0x564db984a8d0 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.821+0000 7fb1977fe640 1 -- 172.21.15.79:0/2284115961 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c00fe60 con 0x564db984a8d0 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2284115961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 msgr2=0x564db984b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db984b650 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0058c0 tx=0x7fb19c00be40 comp rx=0 tx=0).stop 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2284115961 shutdown_connections 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db98530e0 0x564db98525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db984a8d0 0x564db984b650 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2284115961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db9849b60 0x564db985ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2284115961 >> 172.21.15.79:0/2284115961 conn(0x564db9810c90 msgr2=0x564db984af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2284115961 shutdown_connections 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2284115961 wait complete. 2024-04-22T06:52:28.826 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 -- start start 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.822+0000 7fb1b6e00740 1 Processor -- start 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db985b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db985b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db9856180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac1aeb60 con 0x564db98530e0 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac1affd0 con 0x564db984a8d0 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac1af230 con 0x564db9849b60 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 10 client.0 _ll_register_callbacks cb 0x7fb1a404e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db9856180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db9856180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43132/0 (socket says 172.21.15.79:43132) 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2953664017 learned_addr learned my addr 172.21.15.79:0/2953664017 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b17ac640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db985b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db985b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2953664017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 msgr2=0x564db985b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db985b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 msgr2=0x564db985b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db985b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 -- 172.21.15.79:0/2953664017 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db98567f0 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b27ae640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db9856180 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4002230 tx=0x7fb1a4009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e64e7804bccd06 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1957fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1957fa640 1 -- 172.21.15.79:0/2953664017 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4050070 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1957fa640 1 -- 172.21.15.79:0/2953664017 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7fb1a4032cc0 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb19c011bd0 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.823+0000 7fb1957fa640 1 -- 172.21.15.79:0/2953664017 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fb1a406a050 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 10 client.17187 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x564db98c9b40 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1957fa640 1 -- 172.21.15.79:0/2953664017 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb19c011bd0 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1957fa640 1 -- 172.21.15.79:0/2953664017 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fb1a4004580 con 0x564db98530e0 2024-04-22T06:52:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 10 client.17187 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 10 client.17187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 20 client.17187 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db98ca290 0x564db98cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 4 client.17187 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564db974e3d0 con 0x564db98ca290 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.824+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db98ca290 0x564db98cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.825+0000 7fb1957fa640 10 client.17187 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:28.829 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.825+0000 7fb1b1fad640 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db98ca290 0x564db98cc6b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ccebbfe5df654bd server_cookie=43efc555554ad5da in_seq=0 out_seq=0 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1957fa640 1 -- 172.21.15.79:0/2953664017 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564db974e3d0 con 0x564db98ca290 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1957fa640 10 client.17187 handle_command_reply: tid=0 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 client.17187 shutdown 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 msgr2=0x564db9856180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db9856180 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4002230 tx=0x7fb1a4009880 comp rx=0 tx=0).stop 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 shutdown_connections 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db98ca290 0x564db98cc6b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db984a8d0 0x564db985b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db98530e0 0x564db9856180 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1957fa640 0 client.17187 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 --2- 172.21.15.79:0/2953664017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db9849b60 0x564db985b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 >> 172.21.15.79:0/2953664017 conn(0x564db9810c90 msgr2=0x564db9845c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 shutdown_connections 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 1 -- 172.21.15.79:0/2953664017 wait complete. 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.842+0000 7fb1b6e00740 20 client.17187 trim_cache size 0 max 16384 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:52:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:52:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb1980a3df0 msgr2=0x7fb1980a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb1980a3df0 0x7fb1980a62b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4005e30 tx=0x7fb1a4040000 comp rx=0 tx=0).stop 2024-04-22T06:52:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 msgr2=0x7fb1ac074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac074250 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800e9f0 tx=0x7fb1a800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 shutdown_connections 2024-04-22T06:52:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb1980a3df0 0x7fb1980a62b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1ac1072a0 0x7fb1ac074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1ac074cd0 0x7fb1ac1a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 --2- 172.21.15.79:0/57574366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1ac1021d0 0x7fb1ac074250 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 >> 172.21.15.79:0/57574366 conn(0x7fb1ac0793e0 msgr2=0x7fb1ac0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 shutdown_connections 2024-04-22T06:52:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:28.843+0000 7fb1b4211640 1 -- 172.21.15.79:0/57574366 wait complete. 2024-04-22T06:52:30.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:30 smithi193 ceph-mon[31815]: pgmap v259: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 87 MiB/s wr, 565 op/s 2024-04-22T06:52:30.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:30 smithi190 ceph-mon[31421]: pgmap v259: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 87 MiB/s wr, 565 op/s 2024-04-22T06:52:30.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:30 smithi079 ceph-mon[27946]: pgmap v259: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 87 MiB/s wr, 565 op/s 2024-04-22T06:52:32.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:32 smithi193 ceph-mon[31815]: pgmap v260: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 16 MiB/s wr, 115 op/s 2024-04-22T06:52:32.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:32 smithi190 ceph-mon[31421]: pgmap v260: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 16 MiB/s wr, 115 op/s 2024-04-22T06:52:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:32 smithi079 ceph-mon[27946]: pgmap v260: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 16 MiB/s wr, 115 op/s 2024-04-22T06:52:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:34 smithi193 ceph-mon[31815]: pgmap v261: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.3 MiB/s wr, 25 op/s 2024-04-22T06:52:34.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:34 smithi190 ceph-mon[31421]: pgmap v261: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.3 MiB/s wr, 25 op/s 2024-04-22T06:52:34.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:34 smithi079 ceph-mon[27946]: pgmap v261: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.3 MiB/s wr, 25 op/s 2024-04-22T06:52:35.840 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:52:35.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 -- 172.21.15.79:0/561414798 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514102190 msgr2=0x7f3514102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:35.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 --2- 172.21.15.79:0/561414798 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514102190 0x7f3514102570 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3510009900 tx=0x7f351002bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:35.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 -- 172.21.15.79:0/561414798 shutdown_connections 2024-04-22T06:52:35.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 --2- 172.21.15.79:0/561414798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3514102b40 0x7f3514106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:35.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 --2- 172.21.15.79:0/561414798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3514107260 0x7f351410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:35.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 --2- 172.21.15.79:0/561414798 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514102190 0x7f3514102570 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 -- 172.21.15.79:0/561414798 >> 172.21.15.79:0/561414798 conn(0x7f35140793e0 msgr2=0x7f35140797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 -- 172.21.15.79:0/561414798 shutdown_connections 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.955+0000 7f351bf75640 1 -- 172.21.15.79:0/561414798 wait complete. 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 Processor -- start 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 -- start start 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3514102190 0x7f35141986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 0x7f3514198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f351419cf60 0x7f351419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f35141121a0 con 0x7f3514107260 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3514112020 con 0x7f3514102190 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351bf75640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3514112320 con 0x7f351419cf60 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 0x7f3514198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 0x7f3514198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60138/0 (socket says 172.21.15.79:60138) 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 -- 172.21.15.79:0/241229884 learned_addr learned my addr 172.21.15.79:0/241229884 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:35.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519d11640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3514102190 0x7f35141986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f351a512640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f351419cf60 0x7f351419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 -- 172.21.15.79:0/241229884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f351419cf60 msgr2=0x7f351419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f351419cf60 0x7f351419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 -- 172.21.15.79:0/241229884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3514102190 msgr2=0x7f35141986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3514102190 0x7f35141986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.956+0000 7f3519510640 1 -- 172.21.15.79:0/241229884 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351419da50 con 0x7f3514107260 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.957+0000 7f3519510640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 0x7f3514198be0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f350400e9c0 tx=0x7f350400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.957+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350400cd80 con 0x7f3514107260 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.957+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f351419dd40 con 0x7f3514107260 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.957+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35040054d0 con 0x7f3514107260 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.957+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3504010430 con 0x7f3514107260 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.957+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35141a5530 con 0x7f3514107260 2024-04-22T06:52:35.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.958+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3514069230 con 0x7f3514107260 2024-04-22T06:52:35.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.959+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3504004fb0 con 0x7f3514107260 2024-04-22T06:52:35.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.959+0000 7f350affd640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f35000a3d20 0x7f35000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:35.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.960+0000 7f3519d11640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f35000a3d20 0x7f35000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:35.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.960+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f35040d11f0 con 0x7f3514107260 2024-04-22T06:52:35.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.960+0000 7f3519d11640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f35000a3d20 0x7f35000a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3510006010 tx=0x7f3510005e40 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:35.961 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:35.963+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3504053ff0 con 0x7f3514107260 2024-04-22T06:52:36.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.185+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35141999f0 con 0x7f3514107260 2024-04-22T06:52:36.184 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.186+0000 7f350affd640 1 -- 172.21.15.79:0/241229884 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11874 (secure 0 0 0) 0x7f35141999f0 con 0x7f3514107260 2024-04-22T06:52:36.184 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:52:36.184 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":62,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:52:03.164497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25865,"mds_2":25855},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25855":{"gid":25855,"name":"e","rank":2,"incarnation":55,"state":"up:active","state_seq":26,"addr":"172.21.15.190:6835/1167513219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1167513219},{"type":"v1","addr":"172.21.15.190:6835","nonce":1167513219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25865":{"gid":25865,"name":"f","rank":1,"incarnation":55,"state":"up:active","state_seq":26,"addr":"172.21.15.193:6835/99277862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":99277862},{"type":"v1","addr":"172.21.15.193:6835","nonce":99277862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25927":{"gid":25927,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/3285898346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3285898346},{"type":"v1","addr":"172.21.15.193:6839","nonce":3285898346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25937":{"gid":25937,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/1832563615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1832563615},{"type":"v1","addr":"172.21.15.190:6837","nonce":1832563615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25951":{"gid":25951,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/30505216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":30505216},{"type":"v1","addr":"172.21.15.193:6837","nonce":30505216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25855,25865,24467]},"id":1}]} 2024-04-22T06:52:36.184 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 62 2024-04-22T06:52:36.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.187+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f35000a3d20 msgr2=0x7f35000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.187+0000 7f351bf75640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f35000a3d20 0x7f35000a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3510006010 tx=0x7f3510005e40 comp rx=0 tx=0).stop 2024-04-22T06:52:36.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.187+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 msgr2=0x7f3514198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.187+0000 7f351bf75640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 0x7f3514198be0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f350400e9c0 tx=0x7f350400ee80 comp rx=0 tx=0).stop 2024-04-22T06:52:36.186 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 shutdown_connections 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f351419cf60 0x7f351419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3514107260 0x7f3514198be0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f35000a3d20 0x7f35000a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 --2- 172.21.15.79:0/241229884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3514102190 0x7f35141986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 >> 172.21.15.79:0/241229884 conn(0x7f35140793e0 msgr2=0x7f35140ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 shutdown_connections 2024-04-22T06:52:36.187 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.188+0000 7f351bf75640 1 -- 172.21.15.79:0/241229884 wait complete. 2024-04-22T06:52:36.198 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id 3537766f --release --await' 2024-04-22T06:52:36.198 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id 3537766f --release --await 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 -- 172.21.15.79:0/2037653668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc1011a0 msgr2=0x7f48bc101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/2037653668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc1011a0 0x7f48bc101600 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009930 tx=0x7f48b002bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 -- 172.21.15.79:0/2037653668 shutdown_connections 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/2037653668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48bc101b40 0x7f48bc10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/2037653668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc1011a0 0x7f48bc101600 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/2037653668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48bc107030 0x7f48bc107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 -- 172.21.15.79:0/2037653668 >> 172.21.15.79:0/2037653668 conn(0x7f48bc076c50 msgr2=0x7f48bc077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 -- 172.21.15.79:0/2037653668 shutdown_connections 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.305+0000 7f48c1d7e640 1 -- 172.21.15.79:0/2037653668 wait complete. 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 Processor -- start 2024-04-22T06:52:36.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 -- start start 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48bc1011a0 0x7f48bc197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48bc101b40 0x7f48bc197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 0x7f48bc19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1110c0 con 0x7f48bc107030 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc110f40 con 0x7f48bc1011a0 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48c1d7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc111240 con 0x7f48bc101b40 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 0x7f48bc19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 0x7f48bc19bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60162/0 (socket says 172.21.15.79:60162) 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 -- 172.21.15.79:0/3811736328 learned_addr learned my addr 172.21.15.79:0/3811736328 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48bc1011a0 0x7f48bc197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 -- 172.21.15.79:0/3811736328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48bc101b40 msgr2=0x7f48bc197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48bc101b40 0x7f48bc197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 -- 172.21.15.79:0/3811736328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48bc1011a0 msgr2=0x7f48bc197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.307 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48bc1011a0 0x7f48bc197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.306+0000 7f48bbfff640 1 -- 172.21.15.79:0/3811736328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc19c610 con 0x7f48bc107030 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.307+0000 7f48bbfff640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 0x7f48bc19bee0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f48ac007900 tx=0x7f48ac007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.307+0000 7f48b8ff9640 1 -- 172.21.15.79:0/3811736328 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00e6f0 con 0x7f48bc107030 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.307+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc19c900 con 0x7f48bc107030 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.307+0000 7f48b8ff9640 1 -- 172.21.15.79:0/3811736328 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48ac00ed40 con 0x7f48bc107030 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.307+0000 7f48b8ff9640 1 -- 172.21.15.79:0/3811736328 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00ac40 con 0x7f48bc107030 2024-04-22T06:52:36.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.307+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48bc06a550 con 0x7f48bc107030 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48b8ff9640 1 -- 172.21.15.79:0/3811736328 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48ac00e890 con 0x7f48bc107030 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 Processor -- start 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 -- start start 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9310 0x5606620c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9c30 0x5606620d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620d4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x56066202e500 con 0x5606620d2810 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560661fd46a0 con 0x5606620c9310 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.309+0000 7f48c496d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56066202a4c0 con 0x5606620c9c30 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48b8ff9640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f48a40a3d20 0x7f48a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620d4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620d4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60176/0 (socket says 172.21.15.79:60176) 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 -- 172.21.15.79:0/1363392120 learned_addr learned my addr 172.21.15.79:0/1363392120 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48bbfff640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f48a40a3d20 0x7f48a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9310 0x5606620c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9c30 0x5606620d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48b8ff9640 1 -- 172.21.15.79:0/3811736328 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f48ac0d25a0 con 0x7f48bc107030 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 -- 172.21.15.79:0/1363392120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9c30 msgr2=0x5606620d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9c30 0x5606620d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 -- 172.21.15.79:0/1363392120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9310 msgr2=0x5606620c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9310 0x5606620c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 -- 172.21.15.79:0/1363392120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560661ff35e0 con 0x5606620d2810 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48bbfff640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f48a40a3d20 0x7f48a40a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0cfa50 tx=0x7f48ac00a660 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48baffd640 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620d4c00 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f48b0036040 tx=0x7f48b0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4466414c1c9646 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48a1ffb640 1 -- 172.21.15.79:0/1363392120 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0004340 con 0x5606620d2810 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48a1ffb640 1 -- 172.21.15.79:0/1363392120 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b0046070 con 0x5606620d2810 2024-04-22T06:52:36.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.310+0000 7f48a1ffb640 1 -- 172.21.15.79:0/1363392120 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0003ca0 con 0x5606620d2810 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 -- 172.21.15.79:0/1363392120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 msgr2=0x5606620d4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620d4c00 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f48b0036040 tx=0x7f48b0005a20 comp rx=0 tx=0).stop 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 -- 172.21.15.79:0/1363392120 shutdown_connections 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620d4c00 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9c30 0x5606620d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 --2- 172.21.15.79:0/1363392120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9310 0x5606620c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 -- 172.21.15.79:0/1363392120 >> 172.21.15.79:0/1363392120 conn(0x5606620c3a40 msgr2=0x5606620c3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 -- 172.21.15.79:0/1363392120 shutdown_connections 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 -- 172.21.15.79:0/1363392120 wait complete. 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.311+0000 7f48c496d740 1 -- start start 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 Processor -- start 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 0x5606620cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620d2810 0x5606620d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5606620d55a0 con 0x5606620c9310 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560661fe7a60 con 0x5606620c9c30 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5606620d5720 con 0x5606620d2810 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 10 client.0 _ll_register_callbacks cb 0x560661d40110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 0x5606620cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 0x5606620cf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60182/0 (socket says 172.21.15.79:60182) 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 -- 172.21.15.79:0/1287542644 learned_addr learned my addr 172.21.15.79:0/1287542644 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 -- 172.21.15.79:0/1287542644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620d2810 msgr2=0x5606620d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620d2810 0x5606620d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 -- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 msgr2=0x5606620cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 -- 172.21.15.79:0/1287542644 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560661d40920 con 0x5606620c9310 2024-04-22T06:52:36.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48baffd640 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 0x5606620cf6a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f48b00045b0 tx=0x7f48b002c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ea8d7768348551 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f489affd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f489affd640 1 -- 172.21.15.79:0/1287542644 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b004f020 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f489affd640 1 -- 172.21.15.79:0/1287542644 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f48b002c9e0 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.312+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b0046070 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f489affd640 1 -- 172.21.15.79:0/1287542644 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f48b005a050 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 10 client.17205 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x560661e0cd30 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f489affd640 1 -- 172.21.15.79:0/1287542644 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b0046070 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f489affd640 1 -- 172.21.15.79:0/1287542644 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f48b0034440 con 0x5606620c9310 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 10 client.17205 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 10 client.17205 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 20 client.17205 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5606620e2820 0x5606620e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 4 client.17205 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560662001a10 con 0x5606620e2820 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.313+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5606620e2820 0x5606620e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.314+0000 7f489affd640 10 client.17205 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.314+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5606620e2820 0x5606620e4c10 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67c74648f895adb server_cookie=6b6c9a17a0435b9 in_seq=0 out_seq=0 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.314+0000 7f489affd640 1 -- 172.21.15.79:0/1287542644 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560662001a10 con 0x5606620e2820 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.314+0000 7f489affd640 10 client.17205 handle_command_reply: tid=0 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.314+0000 7f48c496d740 1 client.17205 shutdown 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 msgr2=0x5606620cf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 0x5606620cf6a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f48b00045b0 tx=0x7f48b002c330 comp rx=0 tx=0).stop 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 shutdown_connections 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5606620e2820 0x5606620e4c10 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620d2810 0x5606620d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.320 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 --2- 172.21.15.79:0/1287542644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9310 0x5606620cf6a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f489affd640 0 client.17205 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:36.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 >> 172.21.15.79:0/1287542644 conn(0x5606620c3a40 msgr2=0x5606620c5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 shutdown_connections 2024-04-22T06:52:36.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 1 -- 172.21.15.79:0/1287542644 wait complete. 2024-04-22T06:52:36.321 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.315+0000 7f48c496d740 20 client.17205 trim_cache size 0 max 16384 2024-04-22T06:52:36.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.331+0000 7f48c496d740 1 Processor -- start 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 -- start start 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620de460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1ae8b0 con 0x5606620d2810 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1ac180 con 0x5606620c9c30 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48c496d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1a9ff0 con 0x5606620c9310 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60194/0 (socket says 172.21.15.79:60194) 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 -- 172.21.15.79:0/1725313589 learned_addr learned my addr 172.21.15.79:0/1725313589 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 -- 172.21.15.79:0/1725313589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 msgr2=0x5606620d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620de460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 -- 172.21.15.79:0/1725313589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 msgr2=0x5606620de460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620de460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 -- 172.21.15.79:0/1725313589 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606620bdc50 con 0x5606620d2810 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620de460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48baffd640 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620cfd70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f48b0005ec0 tx=0x7f48b002efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8d11c490df614c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48a17fa640 1 -- 172.21.15.79:0/1725313589 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b004f070 con 0x5606620d2810 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48a17fa640 1 -- 172.21.15.79:0/1725313589 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b0046030 con 0x5606620d2810 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.332+0000 7f48a17fa640 1 -- 172.21.15.79:0/1725313589 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0034c40 con 0x5606620d2810 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 -- 172.21.15.79:0/1725313589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 msgr2=0x5606620cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620cfd70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f48b0005ec0 tx=0x7f48b002efd0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 -- 172.21.15.79:0/1725313589 shutdown_connections 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620c9c30 0x5606620de460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620d2810 0x5606620cfd70 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 --2- 172.21.15.79:0/1725313589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 -- 172.21.15.79:0/1725313589 >> 172.21.15.79:0/1725313589 conn(0x5606620c3a40 msgr2=0x5606620cf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 -- 172.21.15.79:0/1725313589 shutdown_connections 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 -- 172.21.15.79:0/1725313589 wait complete. 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.333+0000 7f48c496d740 1 -- start start 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 Processor -- start 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 0x5606620d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620d2810 0x5606620d8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1aa640 con 0x5606620c9c30 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1abab0 con 0x5606620d2810 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1aad10 con 0x5606620c9310 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 10 client.0 _ll_register_callbacks cb 0x7f48b002c5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:36.337 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 0x5606620d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 0x5606620d3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60198/0 (socket says 172.21.15.79:60198) 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 -- 172.21.15.79:0/3228454421 learned_addr learned my addr 172.21.15.79:0/3228454421 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48baffd640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48baffd640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620d2810 0x5606620d8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 -- 172.21.15.79:0/3228454421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 msgr2=0x5606620d3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 -- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620d2810 msgr2=0x5606620d8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620d2810 0x5606620d8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 -- 172.21.15.79:0/3228454421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606620d9430 con 0x5606620c9c30 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48baffd640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620d2810 0x5606620d8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f489affd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48bb7fe640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 0x5606620d3f90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009470 tx=0x7f48b4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94228344fb28a470 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:36.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f489affd640 1 -- 172.21.15.79:0/3228454421 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b401e020 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f489affd640 1 -- 172.21.15.79:0/3228454421 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7f48b401adc0 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.334+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b0046030 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f489affd640 1 -- 172.21.15.79:0/3228454421 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f48b4029050 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 10 client.17211 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5606621491b0 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f489affd640 1 -- 172.21.15.79:0/3228454421 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b0046030 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f489affd640 1 -- 172.21.15.79:0/3228454421 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f48b4012850 con 0x5606620c9c30 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 10 client.17211 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 10 client.17211 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 20 client.17211 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560662149960 0x56066214bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 4 client.17211 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3537766f", "release": true, "await": true}] 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3537766f", "release": true, "await": true}) -- 0x56066203bf20 con 0x560662149960 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.335+0000 7f48baffd640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560662149960 0x56066214bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:36.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.336+0000 7f48baffd640 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560662149960 0x56066214bd80 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b15cfd90daa54209 server_cookie=e84a7be9c2f40e67 in_seq=0 out_seq=0 2024-04-22T06:52:36.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.336+0000 7f489affd640 10 client.17211 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.338+0000 7f489affd640 1 -- 172.21.15.79:0/3228454421 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56066203bf20 con 0x560662149960 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.338+0000 7f489affd640 10 client.17211 handle_command_reply: tid=0 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.338+0000 7f48c496d740 1 client.17211 shutdown 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 msgr2=0x5606620d3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 0x5606620d3f90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f48b4009470 tx=0x7f48b4005040 comp rx=0 tx=0).stop 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 shutdown_connections 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x560662149960 0x56066214bd80 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5606620d2810 0x5606620d8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5606620c9c30 0x5606620d3f90 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f489affd640 0 client.17211 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 --2- 172.21.15.79:0/3228454421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606620c9310 0x5606620d3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 >> 172.21.15.79:0/3228454421 conn(0x5606620c3a40 msgr2=0x5606620c2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 shutdown_connections 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 1 -- 172.21.15.79:0/3228454421 wait complete. 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c496d740 20 client.17211 trim_cache size 0 max 16384 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 62, 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:52:36.342 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 26, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "3537766f": { 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 26, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 85.7, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:52:36.343 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:52:36.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:36.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:36.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:36.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:36.344 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:52:36.344 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f48a40a3d20 msgr2=0x7f48a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f48a40a3d20 0x7f48a40a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0cfa50 tx=0x7f48ac00a660 comp rx=0 tx=0).stop 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 msgr2=0x7f48bc19bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 0x7f48bc19bee0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f48ac007900 tx=0x7f48ac007dc0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 shutdown_connections 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f48bc107030 0x7f48bc19bee0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48bc101b40 0x7f48bc197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f48a40a3d20 0x7f48a40a61e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 --2- 172.21.15.79:0/3811736328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48bc1011a0 0x7f48bc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 >> 172.21.15.79:0/3811736328 conn(0x7f48bc076c50 msgr2=0x7f48bc0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:36.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 shutdown_connections 2024-04-22T06:52:36.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:52:36.339+0000 7f48c1d7e640 1 -- 172.21.15.79:0/3811736328 wait complete. 2024-04-22T06:52:36.349 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 3537766f, seconds elapsed: 26.9 2024-04-22T06:52:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:36 smithi079 ceph-mon[27946]: pgmap v262: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.3 MiB/s wr, 29 op/s 2024-04-22T06:52:36.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:36 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/241229884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-mon[31815]: pgmap v262: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.3 MiB/s wr, 29 op/s 2024-04-22T06:52:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/241229884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:36.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fee97f20700 time 2024-04-22T06:52:36.337942+0000 2024-04-22T06:52:36.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feeab55b3dd] 2024-04-22T06:52:36.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.690 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: clone() 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.337+0000 7fee97f20700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fee97f20700 time 2024-04-22T06:52:36.337942+0000 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feeab55b3dd] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.691 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: clone() 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: *** Caught signal (Aborted) ** 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: in thread 7fee97f20700 thread_name:quiesce.agt 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feeaa2e3d20] 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: gsignal() 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: abort() 2024-04-22T06:52:36.692 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feeab55b437] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 13: clone() 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.339+0000 7fee97f20700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:36.693 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: in thread 7fee97f20700 thread_name:quiesce.agt 2024-04-22T06:52:36.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feeaa2e3d20] 2024-04-22T06:52:36.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: gsignal() 2024-04-22T06:52:36.694 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: abort() 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feeab55b437] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 13: clone() 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: -2> 2024-04-22T06:52:36.337+0000 7fee97f20700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fee97f20700 time 2024-04-22T06:52:36.337942+0000 2024-04-22T06:52:36.695 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feeab55b3dd] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: clone() 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.696 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 0> 2024-04-22T06:52:36.339+0000 7fee97f20700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: in thread 7fee97f20700 thread_name:quiesce.agt 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feeaa2e3d20] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: gsignal() 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: abort() 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feeab55b437] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.697 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 13: clone() 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: -2> 2024-04-22T06:52:36.337+0000 7fee97f20700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fee97f20700 time 2024-04-22T06:52:36.337942+0000 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7feeab55b3dd] 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.698 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: clone() 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 0> 2024-04-22T06:52:36.339+0000 7fee97f20700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: in thread 7fee97f20700 thread_name:quiesce.agt 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feeaa2e3d20] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2: gsignal() 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 3: abort() 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7feeab55b437] 2024-04-22T06:52:36.699 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7feeab55b5a3] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5600cf373e9c] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5600cf373f9f] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5600cf2a20d3] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5600cf2a203c] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5600cf521d3f] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5600cf58a560] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7feeaa2d91ca] 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 13: clone() 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[67582]: 2024-04-22T06:52:36.700 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:36 smithi193 podman[68174]: 2024-04-22 06:52:36.395377316 +0000 UTC m=+0.022570944 container died 6a31b953468a0696b78a68a3f013bb268fc6cc63d9d9268c68228fdadde3b95b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=) 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-mon[31421]: pgmap v262: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 1.3 MiB/s wr, 29 op/s 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/241229884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f9c4e4cc700 time 2024-04-22T06:52:36.337714+0000 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f9c61b073dd] 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.703 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: clone() 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.338+0000 7f9c4e4cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f9c4e4cc700 time 2024-04-22T06:52:36.337714+0000 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f9c61b073dd] 2024-04-22T06:52:36.704 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.705 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: clone() 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: *** Caught signal (Aborted) ** 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: in thread 7f9c4e4cc700 thread_name:quiesce.agt 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9c6088fd20] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: gsignal() 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: abort() 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f9c61b07437] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.706 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 13: clone() 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.340+0000 7f9c4e4cc700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: in thread 7f9c4e4cc700 thread_name:quiesce.agt 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9c6088fd20] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: gsignal() 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: abort() 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f9c61b07437] 2024-04-22T06:52:36.707 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 13: clone() 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: -2> 2024-04-22T06:52:36.338+0000 7f9c4e4cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f9c4e4cc700 time 2024-04-22T06:52:36.337714+0000 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.708 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f9c61b073dd] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: clone() 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 0> 2024-04-22T06:52:36.340+0000 7f9c4e4cc700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: in thread 7f9c4e4cc700 thread_name:quiesce.agt 2024-04-22T06:52:36.709 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9c6088fd20] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: gsignal() 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: abort() 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f9c61b07437] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.710 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 13: clone() 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: -2> 2024-04-22T06:52:36.338+0000 7f9c4e4cc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f9c4e4cc700 time 2024-04-22T06:52:36.337714+0000 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f9c61b073dd] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.711 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: clone() 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 0> 2024-04-22T06:52:36.340+0000 7f9c4e4cc700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: in thread 7f9c4e4cc700 thread_name:quiesce.agt 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9c6088fd20] 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2: gsignal() 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 3: abort() 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f9c61b07437] 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f9c61b075a3] 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5585b1fdbe9c] 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5585b1fdbf9f] 2024-04-22T06:52:36.712 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5585b1f0a0d3] 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5585b1f0a03c] 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5585b2189d3f] 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5585b21f2560] 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f9c608851ca] 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 13: clone() 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[66488]: 2024-04-22T06:52:36.713 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:36 smithi190 podman[66950]: 2024-04-22 06:52:36.39906872 +0000 UTC m=+0.021453697 container died 4561aeb1feeef4b27b523b92b4e71e681b9c4eff092c2a25bc4c11ecdab2821b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T06:52:37.439 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:37 smithi193 podman[68174]: 2024-04-22 06:52:37.177200604 +0000 UTC m=+0.804394215 container remove 6a31b953468a0696b78a68a3f013bb268fc6cc63d9d9268c68228fdadde3b95b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T06:52:37.439 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:37 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:52:37.439 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:37 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-22T06:52:37.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:37 smithi190 podman[66950]: 2024-04-22 06:52:37.191710764 +0000 UTC m=+0.814095786 container remove 4561aeb1feeef4b27b523b92b4e71e681b9c4eff092c2a25bc4c11ecdab2821b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T06:52:37.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:37 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:52:37.452 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:37 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-22T06:52:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:38 smithi193 ceph-mon[31815]: pgmap v263: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 3.1 MiB/s wr, 59 op/s 2024-04-22T06:52:38.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:38 smithi190 ceph-mon[31421]: pgmap v263: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 3.1 MiB/s wr, 59 op/s 2024-04-22T06:52:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:38 smithi079 ceph-mon[27946]: pgmap v263: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 3.1 MiB/s wr, 59 op/s 2024-04-22T06:52:40.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:40 smithi193 ceph-mon[31815]: pgmap v264: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.0 MiB/s wr, 56 op/s 2024-04-22T06:52:40.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:40 smithi190 ceph-mon[31421]: pgmap v264: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.0 MiB/s wr, 56 op/s 2024-04-22T06:52:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:40 smithi079 ceph-mon[27946]: pgmap v264: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.0 MiB/s wr, 56 op/s 2024-04-22T06:52:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 -- 172.21.15.79:0/350950889 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa520105e40 msgr2=0x7fa520106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 --2- 172.21.15.79:0/350950889 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa520105e40 0x7fa520106220 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009900 tx=0x7fa51c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 -- 172.21.15.79:0/350950889 shutdown_connections 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 --2- 172.21.15.79:0/350950889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5201015e0 0x7fa520101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 --2- 172.21.15.79:0/350950889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa520101f80 0x7fa52010dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 --2- 172.21.15.79:0/350950889 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa520105e40 0x7fa520106220 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 -- 172.21.15.79:0/350950889 >> 172.21.15.79:0/350950889 conn(0x7fa520077370 msgr2=0x7fa5200fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 -- 172.21.15.79:0/350950889 shutdown_connections 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.939+0000 7fa527cfd640 1 -- 172.21.15.79:0/350950889 wait complete. 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 Processor -- start 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 -- start start 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 0x7fa5201986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520101f80 0x7fa520198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa520105e40 0x7fa52019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa520111190 con 0x7fa5201015e0 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa520111010 con 0x7fa520101f80 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa527cfd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa520111310 con 0x7fa520105e40 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa525a99640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 0x7fa5201986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa525a99640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 0x7fa5201986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60214/0 (socket says 172.21.15.79:60214) 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.940+0000 7fa525a99640 1 -- 172.21.15.79:0/866018791 learned_addr learned my addr 172.21.15.79:0/866018791 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa52629a640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa520105e40 0x7fa52019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525298640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520101f80 0x7fa520198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525a99640 1 -- 172.21.15.79:0/866018791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa520105e40 msgr2=0x7fa52019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525a99640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa520105e40 0x7fa52019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525a99640 1 -- 172.21.15.79:0/866018791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520101f80 msgr2=0x7fa520198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525a99640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520101f80 0x7fa520198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525a99640 1 -- 172.21.15.79:0/866018791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52019d6d0 con 0x7fa5201015e0 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525298640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520101f80 0x7fa520198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa525a99640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 0x7fa5201986e0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005940 tx=0x7fa51c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa516ffd640 1 -- 172.21.15.79:0/866018791 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c043020 con 0x7fa5201015e0 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa516ffd640 1 -- 172.21.15.79:0/866018791 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c0054c0 con 0x7fa5201015e0 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52019d960 con 0x7fa5201015e0 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa516ffd640 1 -- 172.21.15.79:0/866018791 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c003d40 con 0x7fa5201015e0 2024-04-22T06:52:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.941+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5201a5250 con 0x7fa5201015e0 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.944+0000 7fa516ffd640 1 -- 172.21.15.79:0/866018791 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa51c046020 con 0x7fa5201015e0 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.944+0000 7fa52a9c9740 1 Processor -- start 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 -- start start 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 0x55d49540a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49540afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4954140e0 0x55d4954164d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d49536f8d0 con 0x55d49540b8d0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d49530ce80 con 0x55d4954140e0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d49536b890 con 0x55d49540ab60 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 0x55d49540a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 0x55d49540a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60228/0 (socket says 172.21.15.79:60228) 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 -- 172.21.15.79:0/1709507683 learned_addr learned my addr 172.21.15.79:0/1709507683 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa516ffd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa50c0a3dc0 0x7fa50c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa525298640 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4954140e0 0x55d4954164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa525a99640 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49540afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa50c0a3dc0 0x7fa50c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa516ffd640 1 -- 172.21.15.79:0/866018791 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7fa51c0f2ba0 con 0x7fa5201015e0 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 -- 172.21.15.79:0/1709507683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 msgr2=0x55d49540afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49540afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 -- 172.21.15.79:0/1709507683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4954140e0 msgr2=0x55d4954164d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4954140e0 0x55d4954164d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.945+0000 7fa52629a640 1 -- 172.21.15.79:0/1709507683 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d49532e320 con 0x55d49540b8d0 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.946+0000 7fa52629a640 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 0x55d49540a620 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fa518012890 tx=0x7fa518012d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69fad91d7f9e8ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.946+0000 7fa50bfff640 1 -- 172.21.15.79:0/1709507683 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518003d40 con 0x55d49540b8d0 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.946+0000 7fa50bfff640 1 -- 172.21.15.79:0/1709507683 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa518004b60 con 0x55d49540b8d0 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.946+0000 7fa50bfff640 1 -- 172.21.15.79:0/1709507683 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518005050 con 0x55d49540b8d0 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 -- 172.21.15.79:0/1709507683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 msgr2=0x55d49540a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52629a640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa50c0a3dc0 0x7fa50c0a6280 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa51801f000 tx=0x7fa51801e000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 0x55d49540a620 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fa518012890 tx=0x7fa518012d50 comp rx=0 tx=0).stop 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 -- 172.21.15.79:0/1709507683 shutdown_connections 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4954140e0 0x55d4954164d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49540afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/1709507683 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d49540b8d0 0x55d49540a620 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 -- 172.21.15.79:0/1709507683 >> 172.21.15.79:0/1709507683 conn(0x55d4953d1c90 msgr2=0x55d495405850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 -- 172.21.15.79:0/1709507683 shutdown_connections 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 -- 172.21.15.79:0/1709507683 wait complete. 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 -- start start 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.947+0000 7fa52a9c9740 1 Processor -- start 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540ab60 0x55d495413a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540b8d0 0x55d495410e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d4954113b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d495416e10 con 0x55d4954140e0 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d4952f7eb0 con 0x55d49540ab60 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d495416f90 con 0x55d49540b8d0 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 10 client.0 _ll_register_callbacks cb 0x55d495324b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d4954113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d4954113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60244/0 (socket says 172.21.15.79:60244) 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 -- 172.21.15.79:0/2832235600 learned_addr learned my addr 172.21.15.79:0/2832235600 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa525298640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540b8d0 0x55d495410e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa525a99640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540ab60 0x55d495413a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 -- 172.21.15.79:0/2832235600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540b8d0 msgr2=0x55d495410e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540b8d0 0x55d495410e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 -- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540ab60 msgr2=0x55d495413a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540ab60 0x55d495413a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 -- 172.21.15.79:0/2832235600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d495082210 con 0x55d4954140e0 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa525298640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540b8d0 0x55d495410e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52629a640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d4954113b0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa518008560 tx=0x7fa518012fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d36d5f4a18e8f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa5097fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa5097fa640 1 -- 172.21.15.79:0/2832235600 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518005050 con 0x55d4954140e0 2024-04-22T06:52:41.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa5097fa640 1 -- 172.21.15.79:0/2832235600 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7fa518004d10 con 0x55d4954140e0 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa518004b60 con 0x55d4954140e0 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.948+0000 7fa5097fa640 1 -- 172.21.15.79:0/2832235600 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fa51803a050 con 0x55d4954140e0 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 10 client.17223 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55d49514e620 con 0x55d4954140e0 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa5097fa640 1 -- 172.21.15.79:0/2832235600 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa518004b60 con 0x55d4954140e0 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa5097fa640 1 -- 172.21.15.79:0/2832235600 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fa518025470 con 0x55d4954140e0 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 10 client.17223 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 10 client.17223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 20 client.17223 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d495424090 0x55d495426480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 4 client.17223 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d495383880 con 0x55d495424090 2024-04-22T06:52:41.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.949+0000 7fa525a99640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d495424090 0x55d495426480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa525a99640 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d495424090 0x55d495426480 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a56e5d8f4bed52c server_cookie=24eb93beb8b3d9bc in_seq=0 out_seq=0 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa5097fa640 10 client.17223 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa5097fa640 1 -- 172.21.15.79:0/2832235600 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d495383880 con 0x55d495424090 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa5097fa640 10 client.17223 handle_command_reply: tid=0 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa52a9c9740 1 client.17223 shutdown 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 msgr2=0x55d4954113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.950+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d4954113b0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa518008560 tx=0x7fa518012fc0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 shutdown_connections 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d495424090 0x55d495426480 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d4954113b0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540b8d0 0x55d495410e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/2832235600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540ab60 0x55d495413a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa5097fa640 0 client.17223 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 >> 172.21.15.79:0/2832235600 conn(0x55d4953d1c90 msgr2=0x55d4954075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:41.956 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 shutdown_connections 2024-04-22T06:52:41.957 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 1 -- 172.21.15.79:0/2832235600 wait complete. 2024-04-22T06:52:41.957 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.951+0000 7fa52a9c9740 20 client.17223 trim_cache size 0 max 16384 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 Processor -- start 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 -- start start 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d49542bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d49542c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa5201b0b00 con 0x55d4954140e0 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa5201ae3d0 con 0x55d49540b8d0 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5201ac240 con 0x55d49540ab60 2024-04-22T06:52:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d49542c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d49542c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60260/0 (socket says 172.21.15.79:60260) 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 -- 172.21.15.79:0/3776531898 learned_addr learned my addr 172.21.15.79:0/3776531898 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 -- 172.21.15.79:0/3776531898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 msgr2=0x55d49541fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa525298640 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d49542bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 -- 172.21.15.79:0/3776531898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 msgr2=0x55d49542bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d49542bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 -- 172.21.15.79:0/3776531898 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d495409e30 con 0x55d4954140e0 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa525298640 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d49542bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.966+0000 7fa52629a640 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d49542c130 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa518002230 tx=0x7fa518008cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445c06bc5a0d730f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa50b7fe640 1 -- 172.21.15.79:0/3776531898 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518005050 con 0x55d4954140e0 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa50b7fe640 1 -- 172.21.15.79:0/3776531898 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa518008e10 con 0x55d4954140e0 2024-04-22T06:52:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa50b7fe640 1 -- 172.21.15.79:0/3776531898 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518025470 con 0x55d4954140e0 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 -- 172.21.15.79:0/3776531898 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 msgr2=0x55d49542c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d49542c130 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa518002230 tx=0x7fa518008cc0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 -- 172.21.15.79:0/3776531898 shutdown_connections 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d49542bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d49542c130 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/3776531898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 -- 172.21.15.79:0/3776531898 >> 172.21.15.79:0/3776531898 conn(0x55d4953d1c90 msgr2=0x55d49540c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 -- 172.21.15.79:0/3776531898 shutdown_connections 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 -- 172.21.15.79:0/3776531898 wait complete. 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.967+0000 7fa52a9c9740 1 -- start start 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 Processor -- start 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d495417a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d495418050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa5201ac890 con 0x55d4954140e0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa5201add00 con 0x55d49540b8d0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5201acf60 con 0x55d49540ab60 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52a9c9740 10 client.0 _ll_register_callbacks cb 0x7fa51000b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d495418050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d495418050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60262/0 (socket says 172.21.15.79:60262) 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 -- 172.21.15.79:0/122191210 learned_addr learned my addr 172.21.15.79:0/122191210 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa525298640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d495417a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa525a99640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 -- 172.21.15.79:0/122191210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 msgr2=0x55d49541c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 -- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 msgr2=0x55d495417a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d495417a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.968+0000 7fa52629a640 1 -- 172.21.15.79:0/122191210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d495407b50 con 0x55d4954140e0 2024-04-22T06:52:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52629a640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d495418050 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa5180083c0 tx=0x7fa518004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e366259880c7a2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa5097fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa5097fa640 1 -- 172.21.15.79:0/122191210 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51804c060 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa518008e10 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa5097fa640 1 -- 172.21.15.79:0/122191210 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9373+0+0 (secure 0 0 0) 0x7fa518025470 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa5097fa640 1 -- 172.21.15.79:0/122191210 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fa518053050 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 10 client.17229 fetch_fsmap learned FSMap version 62 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55d49548e720 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa5097fa640 1 -- 172.21.15.79:0/122191210 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa518008e10 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa5097fa640 1 -- 172.21.15.79:0/122191210 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fa5180053c0 con 0x55d4954140e0 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 10 client.17229 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 10 client.17229 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 20 client.17229 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d49548ed90 0x55d4954911b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 4 client.17229 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.969+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d49530f3d0 con 0x55d49548ed90 2024-04-22T06:52:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.970+0000 7fa525a99640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d49548ed90 0x55d4954911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.970+0000 7fa5097fa640 10 client.17229 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:41.974 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.970+0000 7fa525a99640 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d49548ed90 0x55d4954911b0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf402d543fc27b5b server_cookie=9cbf10c694d31ada in_seq=0 out_seq=0 2024-04-22T06:52:41.981 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.982+0000 7fa5097fa640 1 -- 172.21.15.79:0/122191210 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d49530f3d0 con 0x55d49548ed90 2024-04-22T06:52:41.981 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.982+0000 7fa5097fa640 10 client.17229 handle_command_reply: tid=0 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.982+0000 7fa52a9c9740 1 client.17229 shutdown 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 msgr2=0x55d495418050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d495418050 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa5180083c0 tx=0x7fa518004310 comp rx=0 tx=0).stop 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 shutdown_connections 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d49548ed90 0x55d4954911b0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d49540b8d0 0x55d495417a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d4954140e0 0x55d495418050 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 --2- 172.21.15.79:0/122191210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d49540ab60 0x55d49541c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa5097fa640 0 client.17229 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 >> 172.21.15.79:0/122191210 conn(0x55d4953d1c90 msgr2=0x55d495410350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 shutdown_connections 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 1 -- 172.21.15.79:0/122191210 wait complete. 2024-04-22T06:52:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa52a9c9740 20 client.17229 trim_cache size 0 max 16384 2024-04-22T06:52:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:52:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:52:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:52:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa50c0a3dc0 msgr2=0x7fa50c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa50c0a3dc0 0x7fa50c0a6280 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa51801f000 tx=0x7fa51801e000 comp rx=0 tx=0).stop 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 msgr2=0x7fa5201986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 0x7fa5201986e0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005940 tx=0x7fa51c0042f0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 shutdown_connections 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa50c0a3dc0 0x7fa50c0a6280 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa520101f80 0x7fa520198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa520105e40 0x7fa52019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 --2- 172.21.15.79:0/866018791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5201015e0 0x7fa5201986e0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.983+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 >> 172.21.15.79:0/866018791 conn(0x7fa520077370 msgr2=0x7fa5200fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.984+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 shutdown_connections 2024-04-22T06:52:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:41.984+0000 7fa527cfd640 1 -- 172.21.15.79:0/866018791 wait complete. 2024-04-22T06:52:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:42 smithi193 ceph-mon[31815]: pgmap v265: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.0 MiB/s wr, 58 op/s 2024-04-22T06:52:42.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:42 smithi190 ceph-mon[31421]: pgmap v265: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.0 MiB/s wr, 58 op/s 2024-04-22T06:52:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:42 smithi079 ceph-mon[27946]: pgmap v265: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.0 MiB/s wr, 58 op/s 2024-04-22T06:52:44.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:44 smithi193 ceph-mon[31815]: pgmap v266: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.0 MiB/s wr, 55 op/s 2024-04-22T06:52:44.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:44 smithi190 ceph-mon[31421]: pgmap v266: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.0 MiB/s wr, 55 op/s 2024-04-22T06:52:44.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:44 smithi079 ceph-mon[27946]: pgmap v266: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 3.0 MiB/s wr, 55 op/s 2024-04-22T06:52:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:46 smithi193 ceph-mon[31815]: pgmap v267: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 40 op/s 2024-04-22T06:52:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:46 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:52:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:46 smithi190 ceph-mon[31421]: pgmap v267: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 40 op/s 2024-04-22T06:52:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:46 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:52:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:46 smithi079 ceph-mon[27946]: pgmap v267: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 40 op/s 2024-04-22T06:52:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:46 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:52:47.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:52:47.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:52:47.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:52:47.678 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:52:47.678 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 systemd[1]: Stopped Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:52:47.678 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 systemd[1]: Starting Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:52:47.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:52:47.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:52:47.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:52:47.691 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:52:47.692 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 systemd[1]: Stopped Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:52:47.692 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 systemd[1]: Starting Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:52:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:52:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:52:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:52:47.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 podman[68401]: 2024-04-22 06:52:47.582471794 +0000 UTC m=+0.018306929 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:52:47.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 podman[68401]: 2024-04-22 06:52:47.719515773 +0000 UTC m=+0.155350948 container create 0f7958e8bb8ecb0f035dd7d232f4ffc11100caf23584cf8e39215787bdf723ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T06:52:47.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 podman[67177]: 2024-04-22 06:52:47.596889112 +0000 UTC m=+0.018269332 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:52:47.953 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 podman[67177]: 2024-04-22 06:52:47.72594222 +0000 UTC m=+0.147322480 container create 69e64f8300348a194a64c242960c1cb0603aa9c811d0bdbe1f67678ebaee382c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:52:48.209 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 podman[68401]: 2024-04-22 06:52:47.965693227 +0000 UTC m=+0.401528376 container init 0f7958e8bb8ecb0f035dd7d232f4ffc11100caf23584cf8e39215787bdf723ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T06:52:48.209 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 podman[68401]: 2024-04-22 06:52:47.968345111 +0000 UTC m=+0.404180250 container start 0f7958e8bb8ecb0f035dd7d232f4ffc11100caf23584cf8e39215787bdf723ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:52:48.209 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 ceph-mds[68421]: -- 172.21.15.193:0/1840532055 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562063277a40 con 0x56206324ac00 2024-04-22T06:52:48.210 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:47 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: starting mds.f at 2024-04-22T06:52:48.210 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:48 smithi193 bash[68401]: 0f7958e8bb8ecb0f035dd7d232f4ffc11100caf23584cf8e39215787bdf723ea 2024-04-22T06:52:48.210 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:52:48 smithi193 systemd[1]: Started Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:52:48.223 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 podman[67177]: 2024-04-22 06:52:47.961931892 +0000 UTC m=+0.383312096 container init 69e64f8300348a194a64c242960c1cb0603aa9c811d0bdbe1f67678ebaee382c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:52:48.223 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 podman[67177]: 2024-04-22 06:52:47.964525138 +0000 UTC m=+0.385905344 container start 69e64f8300348a194a64c242960c1cb0603aa9c811d0bdbe1f67678ebaee382c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:52:48.223 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 ceph-mds[67197]: -- 172.21.15.190:0/2280510650 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564bf04eda40 con 0x564bf04c1000 2024-04-22T06:52:48.223 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:47 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: starting mds.e at 2024-04-22T06:52:48.223 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:48 smithi190 bash[67177]: 69e64f8300348a194a64c242960c1cb0603aa9c811d0bdbe1f67678ebaee382c 2024-04-22T06:52:48.224 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:52:48 smithi190 systemd[1]: Started Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:52:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: pgmap v268: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-04-22T06:52:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: daemon mds.e restarted 2024-04-22T06:52:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:52:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:52:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:boot 2024-04-22T06:52:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,1=f=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: daemon mds.f restarted 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:boot 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:52:48.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:48 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:48.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: pgmap v268: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-04-22T06:52:48.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: daemon mds.e restarted 2024-04-22T06:52:48.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:boot 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,1=f=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: daemon mds.f restarted 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:boot 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:52:48.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:48 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:48.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: pgmap v268: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: daemon mds.e restarted 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:boot 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,1=f=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:active,2=i=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: daemon mds.f restarted 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:boot 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,2=i=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:52:48.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:48 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:50 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:resolve 2024-04-22T06:52:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:50 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:resolve 2024-04-22T06:52:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:50 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:50.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:50 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:resolve 2024-04-22T06:52:50.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:50 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:resolve 2024-04-22T06:52:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:50 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:50 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:resolve 2024-04-22T06:52:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:50 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:resolve 2024-04-22T06:52:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:50 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: pgmap v271: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-04-22T06:52:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:52:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.000999984 2024-04-22T06:52:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999973 2024-04-22T06:52:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:52:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:reconnect 2024-04-22T06:52:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:reconnect 2024-04-22T06:52:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:51 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:51.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: pgmap v271: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.000999984 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999973 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:reconnect 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:reconnect 2024-04-22T06:52:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:51 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: pgmap v271: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.000999984 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999973 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:reconnect 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:reconnect 2024-04-22T06:52:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:51 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:52.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:rejoin 2024-04-22T06:52:52.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:rejoin 2024-04-22T06:52:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:52:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:52:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:52:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:52:52.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:52 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:active 2024-04-22T06:52:52.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:rejoin 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:rejoin 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:52:52.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:52 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:active 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:rejoin 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:rejoin 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:52:52.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:52 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3285898346,v1:172.21.15.193:6839/3285898346] up:active 2024-04-22T06:52:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:53 smithi193 ceph-mon[31815]: pgmap v272: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.9 KiB/s wr, 10 op/s 2024-04-22T06:52:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:53 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:active 2024-04-22T06:52:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:53 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:53 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:52:53.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:53 smithi190 ceph-mon[31421]: pgmap v272: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.9 KiB/s wr, 10 op/s 2024-04-22T06:52:53.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:53 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:active 2024-04-22T06:52:53.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:53 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:53.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:53 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:52:53.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:53 smithi079 ceph-mon[27946]: pgmap v272: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.9 KiB/s wr, 10 op/s 2024-04-22T06:52:53.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:53 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/1832563615,v1:172.21.15.190:6837/1832563615] up:active 2024-04-22T06:52:53.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:53 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:52:53.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:53 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 -- 172.21.15.79:0/3315637360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98101aa0 msgr2=0x7f5b98105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 --2- 172.21.15.79:0/3315637360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98101aa0 0x7f5b98105a90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009900 tx=0x7f5b8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 -- 172.21.15.79:0/3315637360 shutdown_connections 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 --2- 172.21.15.79:0/3315637360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b98106170 0x7f5b9810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 --2- 172.21.15.79:0/3315637360 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98101aa0 0x7f5b98105a90 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 --2- 172.21.15.79:0/3315637360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b981010f0 0x7f5b981014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 -- 172.21.15.79:0/3315637360 >> 172.21.15.79:0/3315637360 conn(0x7f5b98077fd0 msgr2=0x7f5b980783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 -- 172.21.15.79:0/3315637360 shutdown_connections 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.082+0000 7f5b9e959640 1 -- 172.21.15.79:0/3315637360 wait complete. 2024-04-22T06:52:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 Processor -- start 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 -- start start 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b981010f0 0x7f5b98198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 0x7f5b98198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b9819cf30 0x7f5b9819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981121b0 con 0x7f5b98106170 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98112030 con 0x7f5b9819cf30 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9e959640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98112330 con 0x7f5b981010f0 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 0x7f5b98198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 0x7f5b98198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35748/0 (socket says 172.21.15.79:35748) 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b977fe640 1 -- 172.21.15.79:0/752928260 learned_addr learned my addr 172.21.15.79:0/752928260 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b9819cf30 0x7f5b9819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b97fff640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b981010f0 0x7f5b98198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.083+0000 7f5b977fe640 1 -- 172.21.15.79:0/752928260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b981010f0 msgr2=0x7f5b98198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b977fe640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b981010f0 0x7f5b98198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.084 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b977fe640 1 -- 172.21.15.79:0/752928260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b9819cf30 msgr2=0x7f5b9819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b977fe640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b9819cf30 0x7f5b9819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b977fe640 1 -- 172.21.15.79:0/752928260 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9819da20 con 0x7f5b98106170 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b9819cf30 0x7f5b9819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b977fe640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 0x7f5b98198bb0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c036040 tx=0x7f5b8c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b957fa640 1 -- 172.21.15.79:0/752928260 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c043020 con 0x7f5b98106170 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b957fa640 1 -- 172.21.15.79:0/752928260 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b8c02cbe0 con 0x7f5b98106170 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b957fa640 1 -- 172.21.15.79:0/752928260 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c003bf0 con 0x7f5b98106170 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9819dcb0 con 0x7f5b98106170 2024-04-22T06:52:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.084+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b981a5500 con 0x7f5b98106170 2024-04-22T06:52:55.091 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.086+0000 7f5b957fa640 1 -- 172.21.15.79:0/752928260 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b8c046020 con 0x7f5b98106170 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.086+0000 7f5ba162b740 1 Processor -- start 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.086+0000 7f5ba162b740 1 -- start start 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5ba162b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5626ae53a8d0 con 0x5626ae5d68d0 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5ba162b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5626ae4d7e80 con 0x5626ae5df0e0 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5ba162b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626ae536890 con 0x5626ae5d5b60 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b957fa640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5b800a3e40 0x7f5b800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35764/0 (socket says 172.21.15.79:35764) 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1489762832 learned_addr learned my addr 172.21.15.79:0/1489762832 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5b800a3e40 0x7f5b800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b957fa640 1 -- 172.21.15.79:0/752928260 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9501+0+0 (secure 0 0 0) 0x7f5b8c0f2d70 con 0x7f5b98106170 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b97fff640 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b977fe640 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1489762832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 msgr2=0x5626ae5d5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1489762832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 msgr2=0x5626ae5e14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5e14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.087+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1489762832 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626ae4f9320 con 0x5626ae5d68d0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d5620 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5b90013de0 tx=0x7f5b90010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720a0cc21b70b34c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5b800a3e40 0x7f5b800a6300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5b90010c40 tx=0x7f5b90015000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5b7e7fc640 1 -- 172.21.15.79:0/1489762832 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90010e50 con 0x5626ae5d68d0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5b7e7fc640 1 -- 172.21.15.79:0/1489762832 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b90013740 con 0x5626ae5d68d0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5b7e7fc640 1 -- 172.21.15.79:0/1489762832 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90005330 con 0x5626ae5d68d0 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 -- 172.21.15.79:0/1489762832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 msgr2=0x5626ae5d5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d5620 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5b90013de0 tx=0x7f5b90010740 comp rx=0 tx=0).stop 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 -- 172.21.15.79:0/1489762832 shutdown_connections 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1489762832 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d5620 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.088+0000 7f5ba162b740 1 -- 172.21.15.79:0/1489762832 >> 172.21.15.79:0/1489762832 conn(0x5626ae59cc90 msgr2=0x5626ae5d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 -- 172.21.15.79:0/1489762832 shutdown_connections 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 -- 172.21.15.79:0/1489762832 wait complete. 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 -- start start 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 Processor -- start 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d5b60 0x5626ae5deb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d68d0 0x5626ae5dbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5626ae5e1db0 con 0x5626ae5df0e0 2024-04-22T06:52:55.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5626ae4c2eb0 con 0x5626ae5d5b60 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626ae5e1f30 con 0x5626ae5d68d0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.089+0000 7f5ba162b740 10 client.0 _ll_register_callbacks cb 0x5626ae4efb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5dc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35766/0 (socket says 172.21.15.79:35766) 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 -- 172.21.15.79:0/860559385 learned_addr learned my addr 172.21.15.79:0/860559385 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b97fff640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d5b60 0x5626ae5deb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b977fe640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d68d0 0x5626ae5dbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 -- 172.21.15.79:0/860559385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d68d0 msgr2=0x5626ae5dbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d68d0 0x5626ae5dbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 -- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d5b60 msgr2=0x5626ae5deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d5b60 0x5626ae5deb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 -- 172.21.15.79:0/860559385 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626ae24d210 con 0x5626ae5df0e0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5dc5f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f5b9000bd40 tx=0x7f5b9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9ce2e5cd791dbd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b777fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b777fe640 1 -- 172.21.15.79:0/860559385 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9002d020 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b777fe640 1 -- 172.21.15.79:0/860559385 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9501+0+0 (secure 0 0 0) 0x7f5b90013260 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b90013740 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b777fe640 1 -- 172.21.15.79:0/860559385 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f5b90038050 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5ba162b740 10 client.17247 fetch_fsmap learned FSMap version 71 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x5626ae319620 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.090+0000 7f5b777fe640 1 -- 172.21.15.79:0/860559385 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b90013740 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5b777fe640 1 -- 172.21.15.79:0/860559385 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f5b9001c070 con 0x5626ae5df0e0 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5ba162b740 10 client.17247 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5ba162b740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5ba162b740 20 client.17247 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5ba162b740 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae5ef110 0x5626ae5f1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5ba162b740 4 client.17247 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626ae54e880 con 0x5626ae5ef110 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5b97fff640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae5ef110 0x5626ae5f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5b97fff640 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae5ef110 0x5626ae5f1500 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf154bde21fd9b03 server_cookie=ec7c1e0edf46220c in_seq=0 out_seq=0 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.091+0000 7f5b777fe640 10 client.17247 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5b777fe640 1 -- 172.21.15.79:0/860559385 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626ae54e880 con 0x5626ae5ef110 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5b777fe640 10 client.17247 handle_command_reply: tid=0 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 client.17247 shutdown 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 msgr2=0x5626ae5dc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5dc5f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f5b9000bd40 tx=0x7f5b9000bd70 comp rx=0 tx=0).stop 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 shutdown_connections 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae5ef110 0x5626ae5f1500 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5dc5f0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d68d0 0x5626ae5dbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 --2- 172.21.15.79:0/860559385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d5b60 0x5626ae5deb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 >> 172.21.15.79:0/860559385 conn(0x5626ae59cc90 msgr2=0x5626ae5d25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:55.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.092+0000 7f5b777fe640 0 client.17247 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:55.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.093+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 shutdown_connections 2024-04-22T06:52:55.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.093+0000 7f5ba162b740 1 -- 172.21.15.79:0/860559385 wait complete. 2024-04-22T06:52:55.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.093+0000 7f5ba162b740 20 client.17247 trim_cache size 0 max 16384 2024-04-22T06:52:55.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.107+0000 7f5ba162b740 1 Processor -- start 2024-04-22T06:52:55.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.107+0000 7f5ba162b740 1 -- start start 2024-04-22T06:52:55.110 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d68d0 0x5626ae5e75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5daeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5ba162b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981b0ca0 con 0x5626ae5df0e0 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5ba162b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981ae570 con 0x5626ae5d68d0 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5ba162b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981ac3e0 con 0x5626ae5d5b60 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5daeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5daeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35774/0 (socket says 172.21.15.79:35774) 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1846135870 learned_addr learned my addr 172.21.15.79:0/1846135870 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b977fe640 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d68d0 0x5626ae5e75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1846135870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 msgr2=0x5626ae5eacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.111 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1846135870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d68d0 msgr2=0x5626ae5e75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d68d0 0x5626ae5e75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 -- 172.21.15.79:0/1846135870 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626ae5d4e30 con 0x5626ae5df0e0 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b977fe640 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d68d0 0x5626ae5e75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5daeb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f5b90006010 tx=0x7f5b90010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea2abb8d8e925ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b7dffb640 1 -- 172.21.15.79:0/1846135870 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90041070 con 0x5626ae5df0e0 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b7dffb640 1 -- 172.21.15.79:0/1846135870 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9000bde0 con 0x5626ae5df0e0 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.108+0000 7f5b7dffb640 1 -- 172.21.15.79:0/1846135870 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9002d070 con 0x5626ae5df0e0 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 -- 172.21.15.79:0/1846135870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 msgr2=0x5626ae5daeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5daeb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f5b90006010 tx=0x7f5b90010740 comp rx=0 tx=0).stop 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 -- 172.21.15.79:0/1846135870 shutdown_connections 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5d68d0 0x5626ae5e75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5df0e0 0x5626ae5daeb0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.112 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 --2- 172.21.15.79:0/1846135870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5eacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 -- 172.21.15.79:0/1846135870 >> 172.21.15.79:0/1846135870 conn(0x5626ae59cc90 msgr2=0x5626ae5da7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 -- 172.21.15.79:0/1846135870 shutdown_connections 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 -- 172.21.15.79:0/1846135870 wait complete. 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.109+0000 7f5ba162b740 1 -- start start 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 Processor -- start 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5d9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981aca30 con 0x5626ae5d68d0 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981adea0 con 0x5626ae5df0e0 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981ad100 con 0x5626ae5d5b60 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5ba162b740 10 client.0 _ll_register_callbacks cb 0x7f5b90013be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:55.113 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35780/0 (socket says 172.21.15.79:35780) 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 -- 172.21.15.79:0/619943036 learned_addr learned my addr 172.21.15.79:0/619943036 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b9cef6640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5d9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b97fff640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 -- 172.21.15.79:0/619943036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 msgr2=0x5626ae5d9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 -- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 msgr2=0x5626ae5d9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5d9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 -- 172.21.15.79:0/619943036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626ae5ddef0 con 0x5626ae5d68d0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b977fe640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d9690 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0f0260 tx=0x7f5b8c0a9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c217bfb141f853be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.110+0000 7f5b777fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5b777fe640 1 -- 172.21.15.79:0/619943036 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0df0b0 con 0x5626ae5d68d0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5b777fe640 1 -- 172.21.15.79:0/619943036 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9501+0+0 (secure 0 0 0) 0x7f5b8c0bb0c0 con 0x5626ae5d68d0 2024-04-22T06:52:55.114 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9000bde0 con 0x5626ae5d68d0 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5b777fe640 1 -- 172.21.15.79:0/619943036 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f5b8c0bf150 con 0x5626ae5d68d0 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 10 client.17253 fetch_fsmap learned FSMap version 71 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x5626ae659690 con 0x5626ae5d68d0 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5b777fe640 1 -- 172.21.15.79:0/619943036 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9000bde0 con 0x5626ae5d68d0 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5b777fe640 1 -- 172.21.15.79:0/619943036 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f5b8c0c7230 con 0x5626ae5d68d0 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 10 client.17253 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 20 client.17253 populate_metadata read hostname 'smithi079' 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae659e90 0x5626ae65c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 4 client.17253 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626ae4da3d0 con 0x5626ae659e90 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.111+0000 7f5b97fff640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae659e90 0x5626ae65c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.112+0000 7f5b777fe640 10 client.17253 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.112+0000 7f5b97fff640 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae659e90 0x5626ae65c280 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7552c5e34564a20 server_cookie=3cc334ba74696f09 in_seq=0 out_seq=0 2024-04-22T06:52:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.119+0000 7f5b777fe640 1 -- 172.21.15.79:0/619943036 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626ae4da3d0 con 0x5626ae659e90 2024-04-22T06:52:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.119+0000 7f5b777fe640 10 client.17253 handle_command_reply: tid=0 2024-04-22T06:52:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.119+0000 7f5ba162b740 1 client.17253 shutdown 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 msgr2=0x5626ae5d9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d9690 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0f0260 tx=0x7f5b8c0a9040 comp rx=0 tx=0).stop 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 shutdown_connections 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5626ae659e90 0x5626ae65c280 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5626ae5df0e0 0x5626ae5d9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5626ae5d68d0 0x5626ae5d9690 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 --2- 172.21.15.79:0/619943036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626ae5d5b60 0x5626ae5d9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b777fe640 0 client.17253 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 >> 172.21.15.79:0/619943036 conn(0x5626ae59cc90 msgr2=0x5626ae5db030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 shutdown_connections 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 1 -- 172.21.15.79:0/619943036 wait complete. 2024-04-22T06:52:55.119 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5ba162b740 20 client.17253 trim_cache size 0 max 16384 2024-04-22T06:52:55.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:52:55.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:52:55.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:52:55.120 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5b800a3e40 msgr2=0x7f5b800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5b800a3e40 0x7f5b800a6300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5b90010c40 tx=0x7f5b90015000 comp rx=0 tx=0).stop 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 msgr2=0x7f5b98198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 0x7f5b98198bb0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c036040 tx=0x7f5b8c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 shutdown_connections 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f5b800a3e40 0x7f5b800a6300 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b9819cf30 0x7f5b9819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b98106170 0x7f5b98198bb0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 --2- 172.21.15.79:0/752928260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b981010f0 0x7f5b98198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 >> 172.21.15.79:0/752928260 conn(0x7f5b98077fd0 msgr2=0x7f5b980ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 shutdown_connections 2024-04-22T06:52:55.121 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:52:55.120+0000 7f5b9e959640 1 -- 172.21.15.79:0/752928260 wait complete. 2024-04-22T06:52:55.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:55 smithi193 ceph-mon[31815]: pgmap v273: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-04-22T06:52:55.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:55 smithi190 ceph-mon[31421]: pgmap v273: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-04-22T06:52:55.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:55 smithi079 ceph-mon[27946]: pgmap v273: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-04-22T06:52:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:57 smithi193 ceph-mon[31815]: pgmap v274: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T06:52:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:57 smithi190 ceph-mon[31421]: pgmap v274: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T06:52:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:57 smithi079 ceph-mon[27946]: pgmap v274: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T06:52:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:52:59 smithi193 ceph-mon[31815]: pgmap v275: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.5 KiB/s wr, 23 op/s 2024-04-22T06:52:59.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:52:59 smithi190 ceph-mon[31421]: pgmap v275: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.5 KiB/s wr, 23 op/s 2024-04-22T06:52:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:52:59 smithi079 ceph-mon[27946]: pgmap v275: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.5 KiB/s wr, 23 op/s 2024-04-22T06:53:01.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:01 smithi193 ceph-mon[31815]: pgmap v276: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.0 KiB/s wr, 19 op/s 2024-04-22T06:53:01.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:01 smithi190 ceph-mon[31421]: pgmap v276: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.0 KiB/s wr, 19 op/s 2024-04-22T06:53:01.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:01 smithi079 ceph-mon[27946]: pgmap v276: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.0 KiB/s wr, 19 op/s 2024-04-22T06:53:03.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:03 smithi193 ceph-mon[31815]: pgmap v277: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 KiB/s wr, 23 op/s 2024-04-22T06:53:03.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:03 smithi190 ceph-mon[31421]: pgmap v277: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 KiB/s wr, 23 op/s 2024-04-22T06:53:03.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:03 smithi079 ceph-mon[27946]: pgmap v277: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 KiB/s wr, 23 op/s 2024-04-22T06:53:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:05 smithi193 ceph-mon[31815]: pgmap v278: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 KiB/s wr, 35 op/s 2024-04-22T06:53:05.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:05 smithi190 ceph-mon[31421]: pgmap v278: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 KiB/s wr, 35 op/s 2024-04-22T06:53:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:05 smithi079 ceph-mon[27946]: pgmap v278: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 KiB/s wr, 35 op/s 2024-04-22T06:53:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:07 smithi193 ceph-mon[31815]: pgmap v279: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 46 KiB/s wr, 32 op/s 2024-04-22T06:53:07.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:07 smithi190 ceph-mon[31421]: pgmap v279: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 46 KiB/s wr, 32 op/s 2024-04-22T06:53:07.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:07 smithi079 ceph-mon[27946]: pgmap v279: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 46 KiB/s wr, 32 op/s 2024-04-22T06:53:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 -- 172.21.15.79:0/1113193142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8514102b70 msgr2=0x7f8514106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 --2- 172.21.15.79:0/1113193142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8514102b70 0x7f8514106bb0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8508009900 tx=0x7f850802bc30 comp rx=0 tx=0).stop 2024-04-22T06:53:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 -- 172.21.15.79:0/1113193142 shutdown_connections 2024-04-22T06:53:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 --2- 172.21.15.79:0/1113193142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 0x7f851410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 --2- 172.21.15.79:0/1113193142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8514102b70 0x7f8514106bb0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 --2- 172.21.15.79:0/1113193142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f85141021c0 0x7f85141025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 -- 172.21.15.79:0/1113193142 >> 172.21.15.79:0/1113193142 conn(0x7f85140fda20 msgr2=0x7f85140ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 -- 172.21.15.79:0/1113193142 shutdown_connections 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.217+0000 7f851a0ba640 1 -- 172.21.15.79:0/1113193142 wait complete. 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 Processor -- start 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 -- start start 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 0x7f8514198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 0x7f8514198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851419cf30 0x7f851419d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f85141121d0 con 0x7f85141021c0 2024-04-22T06:53:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8514112050 con 0x7f851419cf30 2024-04-22T06:53:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f851a0ba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8514112350 con 0x7f8514107290 2024-04-22T06:53:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f85137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 0x7f8514198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f85137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 0x7f8514198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42812/0 (socket says 172.21.15.79:42812) 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f85137fe640 1 -- 172.21.15.79:0/842096075 learned_addr learned my addr 172.21.15.79:0/842096075 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f8512ffd640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 0x7f8514198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.218+0000 7f8513fff640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851419cf30 0x7f851419d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f85137fe640 1 -- 172.21.15.79:0/842096075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 msgr2=0x7f8514198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f85137fe640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 0x7f8514198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f85137fe640 1 -- 172.21.15.79:0/842096075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851419cf30 msgr2=0x7f851419d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f85137fe640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851419cf30 0x7f851419d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f85137fe640 1 -- 172.21.15.79:0/842096075 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851419da20 con 0x7f85141021c0 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f8512ffd640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 0x7f8514198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f85137fe640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 0x7f8514198670 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f850400d950 tx=0x7f850400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f8510ff9640 1 -- 172.21.15.79:0/842096075 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504017030 con 0x7f85141021c0 2024-04-22T06:53:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f8510ff9640 1 -- 172.21.15.79:0/842096075 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8504004020 con 0x7f85141021c0 2024-04-22T06:53:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f851419dd10 con 0x7f85141021c0 2024-04-22T06:53:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f8510ff9640 1 -- 172.21.15.79:0/842096075 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85040047a0 con 0x7f85141021c0 2024-04-22T06:53:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.219+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85141a54d0 con 0x7f85141021c0 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.221+0000 7f8510ff9640 1 -- 172.21.15.79:0/842096075 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8504009a30 con 0x7f85141021c0 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 Processor -- start 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 -- start start 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbf4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a7ffb598d0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a7ffaf6e80 con 0x55a7ffbf58d0 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f851cd84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7ffb55890 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f8510ff9640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f84fc0a3dc0 0x7f84fc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f85137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbf4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f85137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbf4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42824/0 (socket says 172.21.15.79:42824) 2024-04-22T06:53:08.228 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f85137fe640 1 -- 172.21.15.79:0/2243614457 learned_addr learned my addr 172.21.15.79:0/2243614457 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f8513fff640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f84fc0a3dc0 0x7f84fc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f8513fff640 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.222+0000 7f8510ff9640 1 -- 172.21.15.79:0/842096075 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9501+0+0 (secure 0 0 0) 0x7f85040d06e0 con 0x7f85141021c0 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f85137fe640 1 -- 172.21.15.79:0/2243614457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbfe0e0 msgr2=0x55a7ffc004d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f85137fe640 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc004d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f85137fe640 1 -- 172.21.15.79:0/2243614457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 msgr2=0x55a7ffbf4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f85137fe640 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f85137fe640 1 -- 172.21.15.79:0/2243614457 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ffb18320 con 0x55a7ffbf4b60 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f8513fff640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f84fc0a3dc0 0x7f84fc0a6280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f850c005c40 tx=0x7f850c009c90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f85137fe640 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbf4fe0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f85040cd9d0 tx=0x7f850409bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1226c48215c3a7d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f84f9ffb640 1 -- 172.21.15.79:0/2243614457 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850409c9b0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f84f9ffb640 1 -- 172.21.15.79:0/2243614457 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f850409c2b0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.223+0000 7f84f9ffb640 1 -- 172.21.15.79:0/2243614457 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85040cdce0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 -- 172.21.15.79:0/2243614457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 msgr2=0x55a7ffbf4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbf4fe0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f85040cd9d0 tx=0x7f850409bcd0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 -- 172.21.15.79:0/2243614457 shutdown_connections 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbf4fe0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 --2- 172.21.15.79:0/2243614457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 -- 172.21.15.79:0/2243614457 >> 172.21.15.79:0/2243614457 conn(0x55a7ffbbbc90 msgr2=0x55a7ffbef850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 -- 172.21.15.79:0/2243614457 shutdown_connections 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 -- 172.21.15.79:0/2243614457 wait complete. 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 -- start start 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.224+0000 7f851cd84740 1 Processor -- start 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbfdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbfaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbfb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a7ffc00db0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a7ffae1eb0 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7ffc00f30 con 0x55a7ffbf58d0 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 10 client.0 _ll_register_callbacks cb 0x55a7ffb0eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbfdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbfdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42826/0 (socket says 172.21.15.79:42826) 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 -- 172.21.15.79:0/2998109559 learned_addr learned my addr 172.21.15.79:0/2998109559 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbfaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f8513fff640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbfb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 -- 172.21.15.79:0/2998109559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf58d0 msgr2=0x55a7ffbfaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbfaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 -- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 msgr2=0x55a7ffbfb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbfb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 -- 172.21.15.79:0/2998109559 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ff86c210 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f8513fff640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbfb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f85137fe640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbfdb40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f850409c240 tx=0x7f8504098550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ecc7238a58ca122 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f84f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f84f2ffd640 1 -- 172.21.15.79:0/2998109559 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85040a4f20 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f84f2ffd640 1 -- 172.21.15.79:0/2998109559 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9501+0+0 (secure 0 0 0) 0x7f85040cdce0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.225+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f850409c2b0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f84f2ffd640 1 -- 172.21.15.79:0/2998109559 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f850409c9b0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 10 client.17265 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a7ff938620 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f84f2ffd640 1 -- 172.21.15.79:0/2998109559 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f850409c2b0 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f84f2ffd640 1 -- 172.21.15.79:0/2998109559 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f85040d0b90 con 0x55a7ffbf4b60 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 10 client.17265 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 10 client.17265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 20 client.17265 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc0dfe0 0x55a7ffc103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 4 client.17265 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7ffb6d880 con 0x55a7ffc0dfe0 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.226+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc0dfe0 0x55a7ffc103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.227+0000 7f84f2ffd640 10 client.17265 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.227+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc0dfe0 0x55a7ffc103d0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13c2b1ebd7538824 server_cookie=18d7c59e987f9673 in_seq=0 out_seq=0 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.227+0000 7f84f2ffd640 1 -- 172.21.15.79:0/2998109559 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7ffb6d880 con 0x55a7ffc0dfe0 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.227+0000 7f84f2ffd640 10 client.17265 handle_command_reply: tid=0 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.227+0000 7f851cd84740 1 client.17265 shutdown 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 msgr2=0x55a7ffbfdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbfdb40 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f850409c240 tx=0x7f8504098550 comp rx=0 tx=0).stop 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 shutdown_connections 2024-04-22T06:53:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc0dfe0 0x55a7ffc103d0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbfb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbfaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 --2- 172.21.15.79:0/2998109559 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf4b60 0x55a7ffbfdb40 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f84f2ffd640 0 client.17265 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 >> 172.21.15.79:0/2998109559 conn(0x55a7ffbbbc90 msgr2=0x55a7ffbf15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 shutdown_connections 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 1 -- 172.21.15.79:0/2998109559 wait complete. 2024-04-22T06:53:08.234 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.228+0000 7f851cd84740 20 client.17265 trim_cache size 0 max 16384 2024-04-22T06:53:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.242+0000 7f851cd84740 1 Processor -- start 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.242+0000 7f851cd84740 1 -- start start 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f851cd84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc09cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f851cd84740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 0x55a7ffc15970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f851cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc15eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f851cd84740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f85141b0d10 con 0x55a7ffbf58d0 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f851cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f85141ae5e0 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f851cd84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85141ac450 con 0x55a7ffbf4b60 2024-04-22T06:53:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 0x55a7ffc15970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 0x55a7ffc15970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42830/0 (socket says 172.21.15.79:42830) 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 -- 172.21.15.79:0/2752607381 learned_addr learned my addr 172.21.15.79:0/2752607381 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 -- 172.21.15.79:0/2752607381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 msgr2=0x55a7ffc09cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc09cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 -- 172.21.15.79:0/2752607381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 msgr2=0x55a7ffc15eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc15eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 -- 172.21.15.79:0/2752607381 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ffbfa470 con 0x55a7ffbf58d0 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f8512ffd640 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 0x55a7ffc15970 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8508009900 tx=0x7f8508005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6684882b779b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f84f97fa640 1 -- 172.21.15.79:0/2752607381 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8508048050 con 0x55a7ffbf58d0 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f84f97fa640 1 -- 172.21.15.79:0/2752607381 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f850802cc90 con 0x55a7ffbf58d0 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.243+0000 7f84f97fa640 1 -- 172.21.15.79:0/2752607381 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8508005330 con 0x55a7ffbf58d0 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 -- 172.21.15.79:0/2752607381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 msgr2=0x55a7ffc15970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 0x55a7ffc15970 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8508009900 tx=0x7f8508005b50 comp rx=0 tx=0).stop 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 -- 172.21.15.79:0/2752607381 shutdown_connections 2024-04-22T06:53:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffc15eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbf58d0 0x55a7ffc15970 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 --2- 172.21.15.79:0/2752607381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc09cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 -- 172.21.15.79:0/2752607381 >> 172.21.15.79:0/2752607381 conn(0x55a7ffbbbc90 msgr2=0x55a7ffbf5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 -- 172.21.15.79:0/2752607381 shutdown_connections 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 -- 172.21.15.79:0/2752607381 wait complete. 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 -- start start 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.244+0000 7f851cd84740 1 Processor -- start 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbf86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f85141acaa0 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f85141adf10 con 0x55a7ffbf58d0 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85141ad170 con 0x55a7ffbf4b60 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 10 client.0 _ll_register_callbacks cb 0x7f850c0125d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbf86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbf86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42840/0 (socket says 172.21.15.79:42840) 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 -- 172.21.15.79:0/3619096298 learned_addr learned my addr 172.21.15.79:0/3619096298 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8512ffd640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f85137fe640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 -- 172.21.15.79:0/3619096298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 msgr2=0x55a7ffc063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 -- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 msgr2=0x55a7ffbf8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 -- 172.21.15.79:0/3619096298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ffbf8c90 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f85137fe640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8512ffd640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f8513fff640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbf86d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f850c009850 tx=0x7f850c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca74824f245f77fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f84f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f84f2ffd640 1 -- 172.21.15.79:0/3619096298 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c01eac0 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f84f2ffd640 1 -- 172.21.15.79:0/3619096298 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9501+0+0 (secure 0 0 0) 0x7f850c01ec60 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.245+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f850802cc90 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f84f2ffd640 1 -- 172.21.15.79:0/3619096298 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f850c033050 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 10 client.17271 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a7ffbff6d0 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f84f2ffd640 1 -- 172.21.15.79:0/3619096298 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f850802cc90 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f84f2ffd640 1 -- 172.21.15.79:0/3619096298 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f850c01f070 con 0x55a7ffbfe0e0 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 10 client.17271 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 10 client.17271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 20 client.17271 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc747f0 0x55a7ffc76c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 4 client.17271 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:08.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7ffaf93d0 con 0x55a7ffc747f0 2024-04-22T06:53:08.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.246+0000 7f85137fe640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc747f0 0x55a7ffc76c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:08.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.247+0000 7f84f2ffd640 10 client.17271 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:08.251 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.247+0000 7f85137fe640 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc747f0 0x55a7ffc76c10 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a56965b83f87460c server_cookie=ff1bd1ff7a844a2b in_seq=0 out_seq=0 2024-04-22T06:53:08.253 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f84f2ffd640 1 -- 172.21.15.79:0/3619096298 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7ffaf93d0 con 0x55a7ffc747f0 2024-04-22T06:53:08.253 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f84f2ffd640 10 client.17271 handle_command_reply: tid=0 2024-04-22T06:53:08.253 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 client.17271 shutdown 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 msgr2=0x55a7ffbf86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbf86d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f850c009850 tx=0x7f850c004a40 comp rx=0 tx=0).stop 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 shutdown_connections 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a7ffc747f0 0x55a7ffc76c10 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a7ffbf58d0 0x55a7ffbf8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a7ffbfe0e0 0x55a7ffbf86d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f84f2ffd640 0 client.17271 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:08.254 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 --2- 172.21.15.79:0/3619096298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7ffbf4b60 0x55a7ffc063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 >> 172.21.15.79:0/3619096298 conn(0x55a7ffbbbc90 msgr2=0x55a7ffbf9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 shutdown_connections 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 1 -- 172.21.15.79:0/3619096298 wait complete. 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.254+0000 7f851cd84740 20 client.17271 trim_cache size 0 max 16384 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:53:08.255 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:53:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f84fc0a3dc0 msgr2=0x7f84fc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f84fc0a3dc0 0x7f84fc0a6280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f850c005c40 tx=0x7f850c009c90 comp rx=0 tx=0).stop 2024-04-22T06:53:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 msgr2=0x7f8514198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:08.256 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 0x7f8514198670 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f850400d950 tx=0x7f850400de10 comp rx=0 tx=0).stop 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 shutdown_connections 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f84fc0a3dc0 0x7f84fc0a6280 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f851419cf30 0x7f851419d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8514107290 0x7f8514198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 --2- 172.21.15.79:0/842096075 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f85141021c0 0x7f8514198670 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 >> 172.21.15.79:0/842096075 conn(0x7f85140fda20 msgr2=0x7f85140ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 shutdown_connections 2024-04-22T06:53:08.257 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:08.255+0000 7f851a0ba640 1 -- 172.21.15.79:0/842096075 wait complete. 2024-04-22T06:53:09.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:09 smithi193 ceph-mon[31815]: pgmap v280: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 34 op/s 2024-04-22T06:53:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:09 smithi190 ceph-mon[31421]: pgmap v280: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 34 op/s 2024-04-22T06:53:09.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:09 smithi079 ceph-mon[27946]: pgmap v280: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 34 op/s 2024-04-22T06:53:10.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:10 smithi193 ceph-mon[31815]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:53:10.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:10 smithi190 ceph-mon[31421]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:53:10.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:10 smithi079 ceph-mon[27946]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:53:11.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:11 smithi193 ceph-mon[31815]: pgmap v282: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-22T06:53:11.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:11 smithi190 ceph-mon[31421]: pgmap v282: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-22T06:53:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:11 smithi079 ceph-mon[27946]: pgmap v282: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-22T06:53:13.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:13 smithi193 ceph-mon[31815]: pgmap v283: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:53:13.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:13 smithi190 ceph-mon[31421]: pgmap v283: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:53:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:13 smithi079 ceph-mon[27946]: pgmap v283: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:53:15.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:15 smithi193 ceph-mon[31815]: pgmap v284: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:53:15.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:15 smithi190 ceph-mon[31421]: pgmap v284: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:53:15.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:15 smithi079 ceph-mon[27946]: pgmap v284: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:53:17.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:17 smithi193 ceph-mon[31815]: pgmap v285: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:53:17.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:17 smithi190 ceph-mon[31421]: pgmap v285: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:53:17.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:17 smithi079 ceph-mon[27946]: pgmap v285: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:53:19.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:19 smithi193 ceph-mon[31815]: pgmap v286: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T06:53:19.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:19 smithi190 ceph-mon[31421]: pgmap v286: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T06:53:19.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:19 smithi079 ceph-mon[27946]: pgmap v286: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-22T06:53:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:20 smithi193 ceph-mon[31815]: pgmap v287: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1002 B/s wr, 6 op/s 2024-04-22T06:53:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:20 smithi193 ceph-mon[31815]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:53:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:20 smithi190 ceph-mon[31421]: pgmap v287: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1002 B/s wr, 6 op/s 2024-04-22T06:53:20.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:20 smithi190 ceph-mon[31421]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:53:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:20 smithi079 ceph-mon[27946]: pgmap v287: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1002 B/s wr, 6 op/s 2024-04-22T06:53:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:20 smithi079 ceph-mon[27946]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 -- 172.21.15.79:0/204773183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa00410c540 msgr2=0x7fa00410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 --2- 172.21.15.79:0/204773183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa00410c540 0x7fa00410e9b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009ca0 tx=0x7f9ffc02c180 comp rx=0 tx=0).stop 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 -- 172.21.15.79:0/204773183 shutdown_connections 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 --2- 172.21.15.79:0/204773183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa00410c540 0x7fa00410e9b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 --2- 172.21.15.79:0/204773183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa004103b20 0x7fa00410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 --2- 172.21.15.79:0/204773183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa004103200 0x7fa0041035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 -- 172.21.15.79:0/204773183 >> 172.21.15.79:0/204773183 conn(0x7fa004077fd0 msgr2=0x7fa0040783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 -- 172.21.15.79:0/204773183 shutdown_connections 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 -- 172.21.15.79:0/204773183 wait complete. 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 Processor -- start 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.353+0000 7fa00ad50640 1 -- start start 2024-04-22T06:53:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa00ad50640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa004103200 0x7fa0041986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa00ad50640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 0x7fa004198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa00ad50640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa00410c540 0x7fa00419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa00ad50640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041121a0 con 0x7fa004103b20 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa00ad50640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa004112020 con 0x7fa00410c540 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa00ad50640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa004112320 con 0x7fa004103200 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 0x7fa004198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 0x7fa004198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55862/0 (socket says 172.21.15.79:55862) 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 -- 172.21.15.79:0/1009831300 learned_addr learned my addr 172.21.15.79:0/1009831300 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa008aec640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa004103200 0x7fa0041986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa00410c540 0x7fa00419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 -- 172.21.15.79:0/1009831300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa004103200 msgr2=0x7fa0041986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa004103200 0x7fa0041986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 -- 172.21.15.79:0/1009831300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa00410c540 msgr2=0x7fa00419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa00410c540 0x7fa00419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 -- 172.21.15.79:0/1009831300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00419d690 con 0x7fa004103b20 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa00410c540 0x7fa00419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa003fff640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 0x7fa004198be0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800d770 tx=0x7f9ff800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.354+0000 7fa001ffb640 1 -- 172.21.15.79:0/1009831300 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8017070 con 0x7fa004103b20 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.355+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00419d980 con 0x7fa004103b20 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.355+0000 7fa001ffb640 1 -- 172.21.15.79:0/1009831300 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff80054d0 con 0x7fa004103b20 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.355+0000 7fa001ffb640 1 -- 172.21.15.79:0/1009831300 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8004c30 con 0x7fa004103b20 2024-04-22T06:53:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.355+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0041a5210 con 0x7fa004103b20 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.357+0000 7fa001ffb640 1 -- 172.21.15.79:0/1009831300 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ff8004e10 con 0x7fa004103b20 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.357+0000 7fa00da1e740 1 Processor -- start 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.357+0000 7fa001ffb640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9fec0a3dc0 0x7f9fec0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.357+0000 7fa00da1e740 1 -- start start 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc9637c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9fec0a3dc0 0x7f9fec0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963884d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa00da1e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55cc962e18d0 con 0x55cc9637cb60 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa00da1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cc9627ee80 con 0x55cc963860e0 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa00da1e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc962dd890 con 0x55cc9637d8d0 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa001ffb640 1 -- 172.21.15.79:0/1009831300 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f9ff80d0660 con 0x7fa004103b20 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55874/0 (socket says 172.21.15.79:55874) 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 -- 172.21.15.79:0/1326149987 learned_addr learned my addr 172.21.15.79:0/1326149987 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa008aec640 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc9637c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 -- 172.21.15.79:0/1326149987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 msgr2=0x55cc9637c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc9637c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 -- 172.21.15.79:0/1326149987 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 msgr2=0x55cc963884d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963884d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 -- 172.21.15.79:0/1326149987 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc962a0320 con 0x55cc9637cb60 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9fec0a3dc0 0x7f9fec0a6280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc005e30 tx=0x7f9ffc009a20 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7fa003fff640 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637cfe0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8098660 tx=0x7f9ff809c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ed82dcf89fa165 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.358+0000 7f9feaffd640 1 -- 172.21.15.79:0/1326149987 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8020040 con 0x55cc9637cb60 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7f9feaffd640 1 -- 172.21.15.79:0/1326149987 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff80d0070 con 0x55cc9637cb60 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7f9feaffd640 1 -- 172.21.15.79:0/1326149987 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80a0120 con 0x55cc9637cb60 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 -- 172.21.15.79:0/1326149987 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 msgr2=0x55cc9637cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637cfe0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8098660 tx=0x7f9ff809c9c0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 -- 172.21.15.79:0/1326149987 shutdown_connections 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637cfe0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1326149987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc9637c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 -- 172.21.15.79:0/1326149987 >> 172.21.15.79:0/1326149987 conn(0x55cc96343c90 msgr2=0x55cc96377850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 -- 172.21.15.79:0/1326149987 shutdown_connections 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.359+0000 7fa00da1e740 1 -- 172.21.15.79:0/1326149987 wait complete. 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 -- start start 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 Processor -- start 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96385a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96382e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 0x55cc963833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55cc96388e10 con 0x55cc963860e0 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55cc96269eb0 con 0x55cc9637cb60 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc96388f90 con 0x55cc9637d8d0 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa00da1e740 10 client.0 _ll_register_callbacks cb 0x55cc96296b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa0092ed640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 0x55cc963833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa0092ed640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 0x55cc963833b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55890/0 (socket says 172.21.15.79:55890) 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa0092ed640 1 -- 172.21.15.79:0/1968892193 learned_addr learned my addr 172.21.15.79:0/1968892193 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa003fff640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96382e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.360+0000 7fa008aec640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96385a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa0092ed640 1 -- 172.21.15.79:0/1968892193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 msgr2=0x55cc96382e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96382e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa0092ed640 1 -- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 msgr2=0x55cc96385a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96385a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa0092ed640 1 -- 172.21.15.79:0/1968892193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc95ff4210 con 0x55cc963860e0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa0092ed640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 0x55cc963833b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc042040 tx=0x7f9ffc02c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad821ecbbd620a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7f9fe3fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7f9fe3fff640 1 -- 172.21.15.79:0/1968892193 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc04f020 con 0x55cc963860e0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7f9fe3fff640 1 -- 172.21.15.79:0/1968892193 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f9ffc042910 con 0x55cc963860e0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ff80d0070 con 0x55cc963860e0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7f9fe3fff640 1 -- 172.21.15.79:0/1968892193 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f9ffc05a050 con 0x55cc963860e0 2024-04-22T06:53:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa00da1e740 10 client.17283 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55cc960c0620 con 0x55cc963860e0 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7f9fe3fff640 1 -- 172.21.15.79:0/1968892193 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff80d0070 con 0x55cc963860e0 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.361+0000 7f9fe3fff640 1 -- 172.21.15.79:0/1968892193 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f9ffc03f070 con 0x55cc963860e0 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa00da1e740 10 client.17283 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa00da1e740 10 client.17283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa00da1e740 20 client.17283 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc96396100 0x55cc963984f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa00da1e740 4 client.17283 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc962f5880 con 0x55cc96396100 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa008aec640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc96396100 0x55cc963984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7f9fe3fff640 10 client.17283 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.362+0000 7fa008aec640 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc96396100 0x55cc963984f0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d414ebf60480ea server_cookie=f2115bd05b040635 in_seq=0 out_seq=0 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7f9fe3fff640 1 -- 172.21.15.79:0/1968892193 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc962f5880 con 0x55cc96396100 2024-04-22T06:53:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7f9fe3fff640 10 client.17283 handle_command_reply: tid=0 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 client.17283 shutdown 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 msgr2=0x55cc963833b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 0x55cc963833b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc042040 tx=0x7f9ffc02c180 comp rx=0 tx=0).stop 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 shutdown_connections 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc96396100 0x55cc963984f0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc963860e0 0x55cc963833b0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96382e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1968892193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96385a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7f9fe3fff640 0 client.17283 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.363+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 >> 172.21.15.79:0/1968892193 conn(0x55cc96343c90 msgr2=0x55cc963795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.364+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 shutdown_connections 2024-04-22T06:53:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.364+0000 7fa00da1e740 1 -- 172.21.15.79:0/1968892193 wait complete. 2024-04-22T06:53:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.364+0000 7fa00da1e740 20 client.17283 trim_cache size 0 max 16384 2024-04-22T06:53:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 Processor -- start 2024-04-22T06:53:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 -- start start 2024-04-22T06:53:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96391d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 0x55cc9637c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc963860e0 0x55cc9639de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041b0cc0 con 0x55cc9637d8d0 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041ae590 con 0x55cc9637cb60 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa00da1e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041ac400 con 0x55cc963860e0 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa003fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 0x55cc9637c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa003fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 0x55cc9637c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55900/0 (socket says 172.21.15.79:55900) 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.378+0000 7fa003fff640 1 -- 172.21.15.79:0/1223664734 learned_addr learned my addr 172.21.15.79:0/1223664734 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa008aec640 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96391d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa003fff640 1 -- 172.21.15.79:0/1223664734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc963860e0 msgr2=0x55cc9639de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa003fff640 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc963860e0 0x55cc9639de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa003fff640 1 -- 172.21.15.79:0/1223664734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 msgr2=0x55cc96391d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa003fff640 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96391d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa003fff640 1 -- 172.21.15.79:0/1223664734 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc96398b70 con 0x55cc9637d8d0 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa008aec640 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96391d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa003fff640 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 0x55cc9637c080 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8009610 tx=0x7f9ff8005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2721ea882ad20ec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7f9fea7fc640 1 -- 172.21.15.79:0/1223664734 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8020040 con 0x55cc9637d8d0 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7f9fea7fc640 1 -- 172.21.15.79:0/1223664734 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff80d0040 con 0x55cc9637d8d0 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7f9fea7fc640 1 -- 172.21.15.79:0/1223664734 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80a0120 con 0x55cc9637d8d0 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa00da1e740 1 -- 172.21.15.79:0/1223664734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 msgr2=0x55cc9637c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.379+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 0x55cc9637c080 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8009610 tx=0x7f9ff8005730 comp rx=0 tx=0).stop 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- 172.21.15.79:0/1223664734 shutdown_connections 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc963860e0 0x55cc9639de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637d8d0 0x55cc9637c080 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 --2- 172.21.15.79:0/1223664734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9637cb60 0x55cc96391d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- 172.21.15.79:0/1223664734 >> 172.21.15.79:0/1223664734 conn(0x55cc96343c90 msgr2=0x55cc9637e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- 172.21.15.79:0/1223664734 shutdown_connections 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- 172.21.15.79:0/1223664734 wait complete. 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- start start 2024-04-22T06:53:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 Processor -- start 2024-04-22T06:53:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96380290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041aca50 con 0x55cc9637cb60 2024-04-22T06:53:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041adec0 con 0x55cc963860e0 2024-04-22T06:53:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041ad120 con 0x55cc9637d8d0 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.380+0000 7fa00da1e740 10 client.0 _ll_register_callbacks cb 0x7f9ffc037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55916/0 (socket says 172.21.15.79:55916) 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 -- 172.21.15.79:0/30698537 learned_addr learned my addr 172.21.15.79:0/30698537 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa003fff640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96380290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa0092ed640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 -- 172.21.15.79:0/30698537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 msgr2=0x55cc96380290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96380290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 -- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 msgr2=0x55cc963807d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963807d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 -- 172.21.15.79:0/30698537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc9637be30 con 0x55cc9637cb60 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7fa008aec640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637fd50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4010660 tx=0x7f9ff4010b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91664be0e06872ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.381+0000 7f9fe3fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7f9fe3fff640 1 -- 172.21.15.79:0/30698537 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4010e60 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7f9fe3fff640 1 -- 172.21.15.79:0/30698537 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f9ff400fb80 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ff80d0040 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7f9fe3fff640 1 -- 172.21.15.79:0/30698537 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f9ff4030050 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 10 client.17289 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55cc963fc990 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7f9fe3fff640 1 -- 172.21.15.79:0/30698537 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff80d0040 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7f9fe3fff640 1 -- 172.21.15.79:0/30698537 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f9ff40284b0 con 0x55cc9637cb60 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 10 client.17289 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 10 client.17289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 20 client.17289 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc963fd0b0 0x55cc963ff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 4 client.17289 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc9637c850 con 0x55cc963fd0b0 2024-04-22T06:53:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.384+0000 7fa003fff640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc963fd0b0 0x55cc963ff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.385+0000 7fa003fff640 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc963fd0b0 0x55cc963ff4d0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2035438a912059e server_cookie=774d00f763c2b0a9 in_seq=0 out_seq=0 2024-04-22T06:53:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.385+0000 7f9fe3fff640 10 client.17289 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7f9fe3fff640 1 -- 172.21.15.79:0/30698537 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc9637c850 con 0x55cc963fd0b0 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7f9fe3fff640 10 client.17289 handle_command_reply: tid=0 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 client.17289 shutdown 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 msgr2=0x55cc9637fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637fd50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4010660 tx=0x7f9ff4010b20 comp rx=0 tx=0).stop 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 shutdown_connections 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55cc963fd0b0 0x55cc963ff4d0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc963860e0 0x55cc963807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7f9fe3fff640 0 client.17289 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9637d8d0 0x55cc96380290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 --2- 172.21.15.79:0/30698537 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55cc9637cb60 0x55cc9637fd50 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 >> 172.21.15.79:0/30698537 conn(0x55cc96343c90 msgr2=0x55cc96381680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 shutdown_connections 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 1 -- 172.21.15.79:0/30698537 wait complete. 2024-04-22T06:53:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.392+0000 7fa00da1e740 20 client.17289 trim_cache size 0 max 16384 2024-04-22T06:53:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:53:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:53:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:53:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9fec0a3dc0 msgr2=0x7f9fec0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9fec0a3dc0 0x7f9fec0a6280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc005e30 tx=0x7f9ffc009a20 comp rx=0 tx=0).stop 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 msgr2=0x7fa004198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 0x7fa004198be0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800d770 tx=0x7f9ff800dc30 comp rx=0 tx=0).stop 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 shutdown_connections 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f9fec0a3dc0 0x7f9fec0a6280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa00410c540 0x7fa00419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa004103b20 0x7fa004198be0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 --2- 172.21.15.79:0/1009831300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa004103200 0x7fa0041986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 >> 172.21.15.79:0/1009831300 conn(0x7fa004077fd0 msgr2=0x7fa0040fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 shutdown_connections 2024-04-22T06:53:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:21.393+0000 7fa00ad50640 1 -- 172.21.15.79:0/1009831300 wait complete. 2024-04-22T06:53:22.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:22 smithi193 ceph-mon[31815]: pgmap v289: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:22.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:22 smithi190 ceph-mon[31421]: pgmap v289: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:22.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:22 smithi079 ceph-mon[27946]: pgmap v289: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:24.078 INFO:tasks.quiescer.fs.[cephfs]:Total duration (36.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T06:53:24.078 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.05 2024-04-22T06:53:24.079 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 -- 172.21.15.79:0/4076097236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9024107260 msgr2=0x7f902410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 --2- 172.21.15.79:0/4076097236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9024107260 0x7f902410ee30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f901c009ca0 tx=0x7f901c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 -- 172.21.15.79:0/4076097236 shutdown_connections 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 --2- 172.21.15.79:0/4076097236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f902406b200 0x7f9024106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 --2- 172.21.15.79:0/4076097236 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9024107260 0x7f902410ee30 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 --2- 172.21.15.79:0/4076097236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f902406a850 0x7f902406ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 -- 172.21.15.79:0/4076097236 >> 172.21.15.79:0/4076097236 conn(0x7f9024077c20 msgr2=0x7f9024078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 -- 172.21.15.79:0/4076097236 shutdown_connections 2024-04-22T06:53:24.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 -- 172.21.15.79:0/4076097236 wait complete. 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.192+0000 7f902973d640 1 Processor -- start 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 -- start start 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 0x7f9024198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9024107260 0x7f9024198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f902419cfc0 0x7f902419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f9024112220 con 0x7f902406a850 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f90241120a0 con 0x7f9024107260 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f902973d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90241123a0 con 0x7f902419cfc0 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 0x7f9024198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 0x7f9024198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51988/0 (socket says 172.21.15.79:51988) 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 -- 172.21.15.79:0/2756602571 learned_addr learned my addr 172.21.15.79:0/2756602571 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f90227fc640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9024107260 0x7f9024198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f90237fe640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f902419cfc0 0x7f902419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 -- 172.21.15.79:0/2756602571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f902419cfc0 msgr2=0x7f902419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f902419cfc0 0x7f902419d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 -- 172.21.15.79:0/2756602571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9024107260 msgr2=0x7f9024198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9024107260 0x7f9024198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 -- 172.21.15.79:0/2756602571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902419dab0 con 0x7f902406a850 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f90227fc640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9024107260 0x7f9024198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f9022ffd640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 0x7f9024198700 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f901400b4a0 tx=0x7f901400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.193+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014014070 con 0x7f902406a850 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.194+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9014010ac0 con 0x7f902406a850 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.194+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902419dda0 con 0x7f902406a850 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.194+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014017c30 con 0x7f902406a850 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.194+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90241a5540 con 0x7f902406a850 2024-04-22T06:53:24.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.194+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9024103660 con 0x7f902406a850 2024-04-22T06:53:24.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.196+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9014017070 con 0x7f902406a850 2024-04-22T06:53:24.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.196+0000 7f900bfff640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f900c0a3d20 0x7f900c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.196 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.197+0000 7f90227fc640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f900c0a3d20 0x7f900c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.197+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f90140d0340 con 0x7f902406a850 2024-04-22T06:53:24.197 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.197+0000 7f90227fc640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f900c0a3d20 0x7f900c0a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9024100d20 tx=0x7f901800a5c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.198 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.200+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f901409c6f0 con 0x7f902406a850 2024-04-22T06:53:24.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.415+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f902410a4f0 con 0x7f902406a850 2024-04-22T06:53:24.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.416+0000 7f900bfff640 1 -- 172.21.15.79:0/2756602571 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+11882 (secure 0 0 0) 0x7f90140bc020 con 0x7f902406a850 2024-04-22T06:53:24.415 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:53:24.415 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:52:52.023532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25937,"mds_2":25927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25927":{"gid":25927,"name":"i","rank":2,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.193:6839/3285898346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3285898346},{"type":"v1","addr":"172.21.15.193:6839","nonce":3285898346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25937":{"gid":25937,"name":"h","rank":1,"incarnation":66,"state":"up:active","state_seq":26,"addr":"172.21.15.190:6837/1832563615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1832563615},{"type":"v1","addr":"172.21.15.190:6837","nonce":1832563615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25951":{"gid":25951,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/30505216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":30505216},{"type":"v1","addr":"172.21.15.193:6837","nonce":30505216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26000":{"gid":26000,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1769517674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1769517674},{"type":"v1","addr":"172.21.15.190:6835","nonce":1769517674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26003":{"gid":26003,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/2674386425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2674386425},{"type":"v1","addr":"172.21.15.193:6835","nonce":2674386425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25927,24467,25937]},"id":1}]} 2024-04-22T06:53:24.415 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 71 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f900c0a3d20 msgr2=0x7f900c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f900c0a3d20 0x7f900c0a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9024100d20 tx=0x7f901800a5c0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 msgr2=0x7f9024198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 0x7f9024198700 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f901400b4a0 tx=0x7f901400b960 comp rx=0 tx=0).stop 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 shutdown_connections 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f900c0a3d20 0x7f900c0a61e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9024107260 0x7f9024198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f902419cfc0 0x7f902419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 --2- 172.21.15.79:0/2756602571 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f902406a850 0x7f9024198700 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 >> 172.21.15.79:0/2756602571 conn(0x7f9024077c20 msgr2=0x7f90240ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 shutdown_connections 2024-04-22T06:53:24.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.418+0000 7f902973d640 1 -- 172.21.15.79:0/2756602571 wait complete. 2024-04-22T06:53:24.428 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 78.05 --await' 2024-04-22T06:53:24.428 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 78.05 --await 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 -- 172.21.15.79:0/818964424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff34101110 msgr2=0x7eff341014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/818964424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff34101110 0x7eff341014f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7eff30009900 tx=0x7eff3002bc30 comp rx=0 tx=0).stop 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 -- 172.21.15.79:0/818964424 shutdown_connections 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/818964424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34106190 0x7eff3410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/818964424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34101ac0 0x7eff34105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/818964424 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff34101110 0x7eff341014f0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 -- 172.21.15.79:0/818964424 >> 172.21.15.79:0/818964424 conn(0x7eff34077fb0 msgr2=0x7eff340783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 -- 172.21.15.79:0/818964424 shutdown_connections 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.537+0000 7eff3bdb2640 1 -- 172.21.15.79:0/818964424 wait complete. 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 Processor -- start 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 -- start start 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34101110 0x7eff341976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34106190 0x7eff34197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 0x7eff3419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff34111150 con 0x7eff3419bf90 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff34110fd0 con 0x7eff34101110 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3bdb2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff341112d0 con 0x7eff34106190 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 0x7eff3419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 0x7eff3419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52006/0 (socket says 172.21.15.79:52006) 2024-04-22T06:53:24.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 -- 172.21.15.79:0/324551911 learned_addr learned my addr 172.21.15.79:0/324551911 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff39b4e640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34101110 0x7eff341976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3934d640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34106190 0x7eff34197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 -- 172.21.15.79:0/324551911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34106190 msgr2=0x7eff34197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34106190 0x7eff34197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 -- 172.21.15.79:0/324551911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34101110 msgr2=0x7eff341976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34101110 0x7eff341976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 -- 172.21.15.79:0/324551911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3419ca80 con 0x7eff3419bf90 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff39b4e640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34101110 0x7eff341976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3934d640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34106190 0x7eff34197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.538+0000 7eff3a34f640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 0x7eff3419c440 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00cab0 tx=0x7eff2c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.539+0000 7eff2affd640 1 -- 172.21.15.79:0/324551911 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00adc0 con 0x7eff3419bf90 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.539+0000 7eff2affd640 1 -- 172.21.15.79:0/324551911 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff2c00f900 con 0x7eff3419bf90 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.539+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3419cd70 con 0x7eff3419bf90 2024-04-22T06:53:24.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.539+0000 7eff2affd640 1 -- 172.21.15.79:0/324551911 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0189a0 con 0x7eff3419bf90 2024-04-22T06:53:24.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.539+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff341a4510 con 0x7eff3419bf90 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff2affd640 1 -- 172.21.15.79:0/324551911 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7eff2c00f450 con 0x7eff3419bf90 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 Processor -- start 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 -- start start 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208687a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 0x55a820873b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8208740d0 0x55a8208764c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a8207c90b0 con 0x55a820868d70 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a82077a680 con 0x55a8208740d0 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a820786c60 con 0x55a8208683c0 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff39b4e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 0x55a820873b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff39b4e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 0x55a820873b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52020/0 (socket says 172.21.15.79:52020) 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff39b4e640 1 -- 172.21.15.79:0/3725823857 learned_addr learned my addr 172.21.15.79:0/3725823857 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3934d640 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8208740d0 0x55a8208764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff2affd640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff200a3d20 0x7eff200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3934d640 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208687a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.542+0000 7eff3a34f640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff200a3d20 0x7eff200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff39b4e640 1 -- 172.21.15.79:0/3725823857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 msgr2=0x55a8208687a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff39b4e640 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208687a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff39b4e640 1 -- 172.21.15.79:0/3725823857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8208740d0 msgr2=0x55a8208764c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff39b4e640 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8208740d0 0x55a8208764c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff39b4e640 1 -- 172.21.15.79:0/3725823857 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8205abc90 con 0x55a820868d70 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff3934d640 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8208740d0 0x55a8208764c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff2affd640 1 -- 172.21.15.79:0/324551911 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7eff2c0d0fa0 con 0x7eff3419bf90 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff39b4e640 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 0x55a820873b90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7eff30000c00 tx=0x7eff30004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60baa74979cb1b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff1ffff640 1 -- 172.21.15.79:0/3725823857 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30044020 con 0x55a820868d70 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff3a34f640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff200a3d20 0x7eff200a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7eff2c098fa0 tx=0x7eff2c06e580 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff1ffff640 1 -- 172.21.15.79:0/3725823857 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff3002cc90 con 0x55a820868d70 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.543+0000 7eff1ffff640 1 -- 172.21.15.79:0/3725823857 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30004920 con 0x55a820868d70 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3725823857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 msgr2=0x55a820873b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 0x55a820873b90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7eff30000c00 tx=0x7eff30004040 comp rx=0 tx=0).stop 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3725823857 shutdown_connections 2024-04-22T06:53:24.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a8208740d0 0x55a8208764c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a820868d70 0x55a820873b90 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3725823857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208687a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3725823857 >> 172.21.15.79:0/3725823857 conn(0x55a8208630f0 msgr2=0x55a8208634d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3725823857 shutdown_connections 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3725823857 wait complete. 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 -- start start 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.544+0000 7eff3e9a4740 1 Processor -- start 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a820871600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a820871b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a8207cc400 con 0x55a8208740d0 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a820788840 con 0x55a820868d70 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8207d1720 con 0x55a8208683c0 2024-04-22T06:53:24.550 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 10 client.0 _ll_register_callbacks cb 0x55a8207d9ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a820871b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a820871b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52032/0 (socket says 172.21.15.79:52032) 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 -- 172.21.15.79:0/1465931575 learned_addr learned my addr 172.21.15.79:0/1465931575 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff39b4e640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a820871600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 -- 172.21.15.79:0/1465931575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 msgr2=0x55a8208710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 -- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 msgr2=0x55a820871600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a820871600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 -- 172.21.15.79:0/1465931575 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a82060c420 con 0x55a8208740d0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3934d640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a820871b40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7eff24016dc0 tx=0x7eff240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95b464de95f9b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff1d7fa640 1 -- 172.21.15.79:0/1465931575 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2400f040 con 0x55a8208740d0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff3002cc90 con 0x55a8208740d0 2024-04-22T06:53:24.551 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff1d7fa640 1 -- 172.21.15.79:0/1465931575 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7eff24019400 con 0x55a8208740d0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff1d7fa640 1 -- 172.21.15.79:0/1465931575 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7eff24027050 con 0x55a8208740d0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 10 client.17307 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.545+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a82049f580 con 0x55a8208740d0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff1d7fa640 1 -- 172.21.15.79:0/1465931575 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff3002cc90 con 0x55a8208740d0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff1d7fa640 1 -- 172.21.15.79:0/1465931575 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7eff240183f0 con 0x55a8208740d0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff3e9a4740 10 client.17307 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff3e9a4740 10 client.17307 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff3e9a4740 20 client.17307 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a820883d80 0x55a8208861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff3e9a4740 4 client.17307 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8207dab10 con 0x55a820883d80 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff39b4e640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a820883d80 0x55a8208861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff1d7fa640 10 client.17307 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.546+0000 7eff39b4e640 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a820883d80 0x55a8208861b0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d8d4a3397d0d178 server_cookie=29c8bdd64d2c1004 in_seq=0 out_seq=0 2024-04-22T06:53:24.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff1d7fa640 1 -- 172.21.15.79:0/1465931575 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8207dab10 con 0x55a820883d80 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff1d7fa640 10 client.17307 handle_command_reply: tid=0 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 client.17307 shutdown 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 msgr2=0x55a820871b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a820871b40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7eff24016dc0 tx=0x7eff240046c0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 shutdown_connections 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a820883d80 0x55a8208861b0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a820871600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a820871b40 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff1d7fa640 0 client.17307 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/1465931575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 >> 172.21.15.79:0/1465931575 conn(0x55a8208630f0 msgr2=0x55a820864e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 shutdown_connections 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.547+0000 7eff3e9a4740 1 -- 172.21.15.79:0/1465931575 wait complete. 2024-04-22T06:53:24.553 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.548+0000 7eff3e9a4740 20 client.17307 trim_cache size 0 max 16384 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.564+0000 7eff3e9a4740 1 Processor -- start 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.564+0000 7eff3e9a4740 1 -- start start 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208737f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82087fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82087c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff341ac440 con 0x55a8208740d0 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff341aa2b0 con 0x55a820868d70 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff341abd70 con 0x55a8208683c0 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82087c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82087c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52042/0 (socket says 172.21.15.79:52042) 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 -- 172.21.15.79:0/3515222314 learned_addr learned my addr 172.21.15.79:0/3515222314 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 -- 172.21.15.79:0/3515222314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 msgr2=0x55a8208737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 -- 172.21.15.79:0/3515222314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 msgr2=0x55a82087fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82087fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 -- 172.21.15.79:0/3515222314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8208764c0 con 0x55a8208740d0 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff3934d640 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82087c380 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7eff24000c00 tx=0x7eff24016eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3a5ea485515093 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff1f7fe640 1 -- 172.21.15.79:0/3515222314 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2402f070 con 0x55a8208740d0 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff1f7fe640 1 -- 172.21.15.79:0/3515222314 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff2401e070 con 0x55a8208740d0 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.565+0000 7eff1f7fe640 1 -- 172.21.15.79:0/3515222314 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24011e10 con 0x55a8208740d0 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3515222314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 msgr2=0x55a82087c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82087c380 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7eff24000c00 tx=0x7eff24016eb0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3515222314 shutdown_connections 2024-04-22T06:53:24.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82087fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82087c380 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/3515222314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a8208737f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3515222314 >> 172.21.15.79:0/3515222314 conn(0x55a8208630f0 msgr2=0x55a820871910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3515222314 shutdown_connections 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 -- 172.21.15.79:0/3515222314 wait complete. 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 -- start start 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.566+0000 7eff3e9a4740 1 Processor -- start 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a82086d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82086da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82086e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7eff341aafd0 con 0x55a8208740d0 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff341aa900 con 0x55a820868d70 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff341ab6a0 con 0x55a8208683c0 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 10 client.0 _ll_register_callbacks cb 0x7eff2400ca10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:24.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82086e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82086e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52054/0 (socket says 172.21.15.79:52054) 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 -- 172.21.15.79:0/144858748 learned_addr learned my addr 172.21.15.79:0/144858748 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a82086d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff39b4e640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82086da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 -- 172.21.15.79:0/144858748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 msgr2=0x55a82086d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a82086d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 -- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 msgr2=0x55a82086da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82086da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 -- 172.21.15.79:0/144858748 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a82086e8e0 con 0x55a8208740d0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3934d640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82086e0e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7eff24004700 tx=0x7eff24008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b72f5d0432acff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff1d7fa640 1 -- 172.21.15.79:0/144858748 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2403b060 con 0x55a8208740d0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff1d7fa640 1 -- 172.21.15.79:0/144858748 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7eff24018a30 con 0x55a8208740d0 2024-04-22T06:53:24.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff2401e070 con 0x55a8208740d0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff1d7fa640 1 -- 172.21.15.79:0/144858748 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7eff2402f040 con 0x55a8208740d0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 10 client.17313 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.567+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55a8208ecbc0 con 0x55a8208740d0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff1d7fa640 1 -- 172.21.15.79:0/144858748 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff2401e070 con 0x55a8208740d0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff1d7fa640 1 -- 172.21.15.79:0/144858748 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7eff24023b00 con 0x55a8208740d0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff3e9a4740 10 client.17313 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff3e9a4740 10 client.17313 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff3e9a4740 20 client.17313 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a8208ed240 0x55a8208ef660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff3e9a4740 4 client.17313 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.05, "await": true}] 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.05, "await": true}) -- 0x55a8207d4980 con 0x55a8208ed240 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff39b4e640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a8208ed240 0x55a8208ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff39b4e640 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a8208ed240 0x55a8208ef660 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7504cdafc8a64527 server_cookie=ef7c82d1f3986 in_seq=0 out_seq=0 2024-04-22T06:53:24.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.568+0000 7eff1d7fa640 10 client.17313 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:24.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.580+0000 7eff1d7fa640 1 -- 172.21.15.79:0/144858748 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55a8207d4980 con 0x55a8208ed240 2024-04-22T06:53:24.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.580+0000 7eff1d7fa640 10 client.17313 handle_command_reply: tid=0 2024-04-22T06:53:24.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.580+0000 7eff3e9a4740 1 client.17313 shutdown 2024-04-22T06:53:24.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.580+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 msgr2=0x55a82086e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.580+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82086e0e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7eff24004700 tx=0x7eff24008040 comp rx=0 tx=0).stop 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 shutdown_connections 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a8208ed240 0x55a8208ef660 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a820868d70 0x55a82086da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a8208740d0 0x55a82086e0e0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff1d7fa640 0 client.17313 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 --2- 172.21.15.79:0/144858748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8208683c0 0x55a82086d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 >> 172.21.15.79:0/144858748 conn(0x55a8208630f0 msgr2=0x55a820861940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 shutdown_connections 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 1 -- 172.21.15.79:0/144858748 wait complete. 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3e9a4740 20 client.17313 trim_cache size 0 max 16384 2024-04-22T06:53:24.580 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 71, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 28, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "71d1df0": { 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 28, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 78.0, 2024-04-22T06:53:24.581 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:24.582 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:53:24.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff200a3d20 msgr2=0x7eff200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff200a3d20 0x7eff200a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7eff2c098fa0 tx=0x7eff2c06e580 comp rx=0 tx=0).stop 2024-04-22T06:53:24.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 msgr2=0x7eff3419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 0x7eff3419c440 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00cab0 tx=0x7eff2c00cf70 comp rx=0 tx=0).stop 2024-04-22T06:53:24.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 shutdown_connections 2024-04-22T06:53:24.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7eff3419bf90 0x7eff3419c440 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff34106190 0x7eff34197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7eff200a3d20 0x7eff200a61e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 --2- 172.21.15.79:0/324551911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff34101110 0x7eff341976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 >> 172.21.15.79:0/324551911 conn(0x7eff34077fb0 msgr2=0x7eff340ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 shutdown_connections 2024-04-22T06:53:24.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:24.581+0000 7eff3bdb2640 1 -- 172.21.15.79:0/324551911 wait complete. 2024-04-22T06:53:24.590 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 71d1df0, quiesce took 0.5 seconds. Will release after: 18.05 2024-04-22T06:53:24.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:24 smithi193 ceph-mon[31815]: pgmap v290: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:24.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:24 smithi190 ceph-mon[31421]: pgmap v290: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:24 smithi079 ceph-mon[27946]: pgmap v290: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:25.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:25 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2756602571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:25.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:25 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2756602571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:25.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:25 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2756602571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:26.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:26 smithi193 ceph-mon[31815]: pgmap v291: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:26.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:26 smithi190 ceph-mon[31421]: pgmap v291: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:26.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:26 smithi079 ceph-mon[27946]: pgmap v291: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:28.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:28 smithi193 ceph-mon[31815]: pgmap v292: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:28 smithi190 ceph-mon[31421]: pgmap v292: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:28.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:28 smithi079 ceph-mon[27946]: pgmap v292: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:30.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:30 smithi193 ceph-mon[31815]: pgmap v293: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:30.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:30 smithi190 ceph-mon[31421]: pgmap v293: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:30.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:30 smithi079 ceph-mon[27946]: pgmap v293: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:32 smithi193 ceph-mon[31815]: pgmap v294: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:53:32.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:32 smithi190 ceph-mon[31421]: pgmap v294: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:53:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:32 smithi079 ceph-mon[27946]: pgmap v294: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:53:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 -- 172.21.15.79:0/3905618125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 msgr2=0x7f37f0110d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 --2- 172.21.15.79:0/3905618125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f0110d20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009ca0 tx=0x7f37e802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 -- 172.21.15.79:0/3905618125 shutdown_connections 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 --2- 172.21.15.79:0/3905618125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f0110d20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 --2- 172.21.15.79:0/3905618125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f01016a0 0x7f37f010e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 --2- 172.21.15.79:0/3905618125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f0100d80 0x7f37f0101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 -- 172.21.15.79:0/3905618125 >> 172.21.15.79:0/3905618125 conn(0x7f37f0077fd0 msgr2=0x7f37f00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 -- 172.21.15.79:0/3905618125 shutdown_connections 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 -- 172.21.15.79:0/3905618125 wait complete. 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.489+0000 7f37f6def640 1 Processor -- start 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 -- start start 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f0100d80 0x7f37f01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f01016a0 0x7f37f01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0114410 con 0x7f37f010e930 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0114290 con 0x7f37f01016a0 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f6def640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0114590 con 0x7f37f0100d80 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59312/0 (socket says 172.21.15.79:59312) 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 -- 172.21.15.79:0/3627025302 learned_addr learned my addr 172.21.15.79:0/3627025302 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37effff640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f01016a0 0x7f37f01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 -- 172.21.15.79:0/3627025302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f0100d80 msgr2=0x7f37f01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f0100d80 0x7f37f01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f0100d80 0x7f37f01a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 -- 172.21.15.79:0/3627025302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f01016a0 msgr2=0x7f37f01a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f01016a0 0x7f37f01a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 -- 172.21.15.79:0/3627025302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f019b030 con 0x7f37f010e930 2024-04-22T06:53:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37effff640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f01016a0 0x7f37f01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37f538c640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f019a770 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f37e8033000 tx=0x7f37e80346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37edffb640 1 -- 172.21.15.79:0/3627025302 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8041020 con 0x7f37f010e930 2024-04-22T06:53:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.490+0000 7f37edffb640 1 -- 172.21.15.79:0/3627025302 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e8032970 con 0x7f37f010e930 2024-04-22T06:53:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.491+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f019b2c0 con 0x7f37f010e930 2024-04-22T06:53:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.491+0000 7f37edffb640 1 -- 172.21.15.79:0/3627025302 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8039040 con 0x7f37f010e930 2024-04-22T06:53:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.491+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f01a7610 con 0x7f37f010e930 2024-04-22T06:53:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.493+0000 7f37edffb640 1 -- 172.21.15.79:0/3627025302 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f37e8044020 con 0x7f37f010e930 2024-04-22T06:53:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.493+0000 7f37f99dc740 1 Processor -- start 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.493+0000 7f37f99dc740 1 -- start start 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d14620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 0x55e5d7d14fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d204d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f99dc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e5d7c798d0 con 0x55e5d7d14b60 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f99dc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e5d7c16e80 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f99dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5d7c75890 con 0x55e5d7d158d0 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37edffb640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37e00a3dc0 0x7f37e00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 0x55e5d7d14fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 0x55e5d7d14fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59328/0 (socket says 172.21.15.79:59328) 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 -- 172.21.15.79:0/3907404034 learned_addr learned my addr 172.21.15.79:0/3907404034 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37effff640 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d14620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f538c640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37e00a3dc0 0x7f37e00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37effff640 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d204d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 -- 172.21.15.79:0/3907404034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 msgr2=0x55e5d7d14620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d14620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 -- 172.21.15.79:0/3907404034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d1e0e0 msgr2=0x55e5d7d204d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d204d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 -- 172.21.15.79:0/3907404034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d7c38320 con 0x55e5d7d14b60 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37edffb640 1 -- 172.21.15.79:0/3627025302 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f37e80f2950 con 0x7f37f010e930 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37effff640 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d204d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 0x55e5d7d14fe0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f37e400a920 tx=0x7f37e400ade0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d997756b6a61a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.494+0000 7f37f538c640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37e00a3dc0 0x7f37e00a6280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f37e80efe60 tx=0x7f37e808cfb0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37daffd640 1 -- 172.21.15.79:0/3907404034 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4015070 con 0x55e5d7d14b60 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37daffd640 1 -- 172.21.15.79:0/3907404034 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e400cc00 con 0x55e5d7d14b60 2024-04-22T06:53:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37daffd640 1 -- 172.21.15.79:0/3907404034 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40185c0 con 0x55e5d7d14b60 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 -- 172.21.15.79:0/3907404034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 msgr2=0x55e5d7d14fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 0x55e5d7d14fe0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f37e400a920 tx=0x7f37e400ade0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 -- 172.21.15.79:0/3907404034 shutdown_connections 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d204d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d14b60 0x55e5d7d14fe0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3907404034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d14620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 -- 172.21.15.79:0/3907404034 >> 172.21.15.79:0/3907404034 conn(0x55e5d7cdbc90 msgr2=0x55e5d7d0f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 -- 172.21.15.79:0/3907404034 shutdown_connections 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.495+0000 7f37f99dc740 1 -- 172.21.15.79:0/3907404034 wait complete. 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 -- start start 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 Processor -- start 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d14b60 0x55e5d7d1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e5d7d20e10 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e5d7c01eb0 con 0x55e5d7d158d0 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5d7d20f90 con 0x55e5d7d14b60 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f99dc740 10 client.0 _ll_register_callbacks cb 0x55e5d7c2eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37effff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37effff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59330/0 (socket says 172.21.15.79:59330) 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37effff640 1 -- 172.21.15.79:0/3699390057 learned_addr learned my addr 172.21.15.79:0/3699390057 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.496+0000 7f37effff640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d14b60 0x55e5d7d1da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37effff640 1 -- 172.21.15.79:0/3699390057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d14b60 msgr2=0x55e5d7d1da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37effff640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d14b60 0x55e5d7d1da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37effff640 1 -- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d158d0 msgr2=0x55e5d7d1ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37effff640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37effff640 1 -- 172.21.15.79:0/3699390057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d798c210 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37effff640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1b340 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00ce90 tx=0x7f37dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab82e5460f2bd55a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37cffff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37cffff640 1 -- 172.21.15.79:0/3699390057 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc004030 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37cffff640 1 -- 172.21.15.79:0/3699390057 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f37dc0041d0 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e400cc00 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37cffff640 1 -- 172.21.15.79:0/3699390057 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f37dc026050 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 10 client.17325 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55e5d7a58620 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37cffff640 1 -- 172.21.15.79:0/3699390057 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e400cc00 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37cffff640 1 -- 172.21.15.79:0/3699390057 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f37dc011440 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 10 client.17325 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 10 client.17325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 20 client.17325 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.497+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d2e060 0x55e5d7d30450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.498+0000 7f37f99dc740 4 client.17325 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.498+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5d7c8d880 con 0x55e5d7d2e060 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.498+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d2e060 0x55e5d7d30450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.498+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d2e060 0x55e5d7d30450 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2031ebde5a216c7e server_cookie=ab898e3ff9909ba5 in_seq=0 out_seq=0 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.498+0000 7f37cffff640 10 client.17325 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37cffff640 1 -- 172.21.15.79:0/3699390057 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5d7c8d880 con 0x55e5d7d2e060 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37cffff640 10 client.17325 handle_command_reply: tid=0 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 client.17325 shutdown 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 msgr2=0x55e5d7d1b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1b340 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00ce90 tx=0x7f37dc0046c0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 shutdown_connections 2024-04-22T06:53:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d2e060 0x55e5d7d30450 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1b340 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37cffff640 0 client.17325 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 --2- 172.21.15.79:0/3699390057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d14b60 0x55e5d7d1da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 >> 172.21.15.79:0/3699390057 conn(0x55e5d7cdbc90 msgr2=0x55e5d7d115a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 shutdown_connections 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 1 -- 172.21.15.79:0/3699390057 wait complete. 2024-04-22T06:53:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.499+0000 7f37f99dc740 20 client.17325 trim_cache size 0 max 16384 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 Processor -- start 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 -- start start 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 0x55e5d7d29d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01b2fc0 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01b0890 con 0x55e5d7d14b60 2024-04-22T06:53:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37f99dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01ae700 con 0x55e5d7d158d0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37effff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37effff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59336/0 (socket says 172.21.15.79:59336) 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.514+0000 7f37effff640 1 -- 172.21.15.79:0/1407353595 learned_addr learned my addr 172.21.15.79:0/1407353595 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37effff640 1 -- 172.21.15.79:0/1407353595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 msgr2=0x55e5d7d1d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37effff640 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37effff640 1 -- 172.21.15.79:0/1407353595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 msgr2=0x55e5d7d29d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37effff640 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 0x55e5d7d29d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37effff640 1 -- 172.21.15.79:0/1407353595 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d7d307c0 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37effff640 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1da50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00c760 tx=0x7f37dc00ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cdf73f55105264 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37da7fc640 1 -- 172.21.15.79:0/1407353595 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc02e070 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37da7fc640 1 -- 172.21.15.79:0/1407353595 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37dc00c870 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.515+0000 7f37da7fc640 1 -- 172.21.15.79:0/1407353595 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc02a8e0 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- 172.21.15.79:0/1407353595 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 msgr2=0x55e5d7d1da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1da50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00c760 tx=0x7f37dc00ff80 comp rx=0 tx=0).stop 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- 172.21.15.79:0/1407353595 shutdown_connections 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d1da50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d158d0 0x55e5d7d1d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- 172.21.15.79:0/1407353595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 0x55e5d7d29d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- 172.21.15.79:0/1407353595 >> 172.21.15.79:0/1407353595 conn(0x55e5d7cdbc90 msgr2=0x55e5d7d0f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- 172.21.15.79:0/1407353595 shutdown_connections 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- 172.21.15.79:0/1407353595 wait complete. 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- start start 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 Processor -- start 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 0x55e5d7d26520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 0x55e5d7d17730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d17c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01aed50 con 0x55e5d7d158d0 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01b01c0 con 0x55e5d7d14b60 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37f01af420 con 0x55e5d7d1e0e0 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f99dc740 10 client.0 _ll_register_callbacks cb 0x7f37dc00c790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f4b8b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 0x55e5d7d17730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f4b8b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 0x55e5d7d17730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59350/0 (socket says 172.21.15.79:59350) 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.516+0000 7f37f4b8b640 1 -- 172.21.15.79:0/554552169 learned_addr learned my addr 172.21.15.79:0/554552169 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f4b8b640 1 -- 172.21.15.79:0/554552169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d1e0e0 msgr2=0x55e5d7d17c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d17c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f4b8b640 1 -- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 msgr2=0x55e5d7d26520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 0x55e5d7d26520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f4b8b640 1 -- 172.21.15.79:0/554552169 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d7d183b0 con 0x55e5d7d158d0 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f4b8b640 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 0x55e5d7d17730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f37e400b010 tx=0x7f37e400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3298923232ea8fa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:34.521 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37cffff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37cffff640 1 -- 172.21.15.79:0/554552169 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4015040 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37cffff640 1 -- 172.21.15.79:0/554552169 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f37e40113f0 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc00c870 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37cffff640 1 -- 172.21.15.79:0/554552169 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f37e4034050 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f99dc740 10 client.17331 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x55e5d7d94540 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37cffff640 1 -- 172.21.15.79:0/554552169 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37dc00c870 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37cffff640 1 -- 172.21.15.79:0/554552169 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f37e402f030 con 0x55e5d7d158d0 2024-04-22T06:53:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.517+0000 7f37f99dc740 10 client.17331 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37f99dc740 10 client.17331 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37f99dc740 20 client.17331 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37f99dc740 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d94a40 0x55e5d7d96e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37f99dc740 4 client.17331 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5d7c193d0 con 0x55e5d7d94a40 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37effff640 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d94a40 0x55e5d7d96e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37cffff640 10 client.17331 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:34.524 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.518+0000 7f37effff640 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d94a40 0x55e5d7d96e60 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e11a643892ed9ade server_cookie=c768989eaa589659 in_seq=0 out_seq=0 2024-04-22T06:53:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.522+0000 7f37cffff640 1 -- 172.21.15.79:0/554552169 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e5d7c193d0 con 0x55e5d7d94a40 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37cffff640 10 client.17331 handle_command_reply: tid=0 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 client.17331 shutdown 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 msgr2=0x55e5d7d17730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 0x55e5d7d17730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f37e400b010 tx=0x7f37e400a430 comp rx=0 tx=0).stop 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 shutdown_connections 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5d7d94a40 0x55e5d7d96e60 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5d7d1e0e0 0x55e5d7d17c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5d7d158d0 0x55e5d7d17730 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37cffff640 0 client.17331 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 --2- 172.21.15.79:0/554552169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5d7d14b60 0x55e5d7d26520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 >> 172.21.15.79:0/554552169 conn(0x55e5d7cdbc90 msgr2=0x55e5d7d19560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 shutdown_connections 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 1 -- 172.21.15.79:0/554552169 wait complete. 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f99dc740 20 client.17331 trim_cache size 0 max 16384 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:53:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:53:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:53:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37e00a3dc0 msgr2=0x7f37e00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f6def640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37e00a3dc0 0x7f37e00a6280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f37e80efe60 tx=0x7f37e808cfb0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 msgr2=0x7f37f019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.523+0000 7f37f6def640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f019a770 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f37e8033000 tx=0x7f37e80346e0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 shutdown_connections 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37e00a3dc0 0x7f37e00a6280 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37f01016a0 0x7f37f01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37f010e930 0x7f37f019a770 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 --2- 172.21.15.79:0/3627025302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f0100d80 0x7f37f01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 >> 172.21.15.79:0/3627025302 conn(0x7f37f0077fd0 msgr2=0x7f37f010fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 shutdown_connections 2024-04-22T06:53:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:34.524+0000 7f37f6def640 1 -- 172.21.15.79:0/3627025302 wait complete. 2024-04-22T06:53:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:34 smithi193 ceph-mon[31815]: pgmap v295: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:53:34.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:34 smithi190 ceph-mon[31421]: pgmap v295: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:53:34.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:34 smithi079 ceph-mon[27946]: pgmap v295: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:53:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:36 smithi193 ceph-mon[31815]: pgmap v296: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:53:36.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:36 smithi190 ceph-mon[31421]: pgmap v296: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:53:36.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:36 smithi079 ceph-mon[27946]: pgmap v296: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:53:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:38 smithi193 ceph-mon[31815]: pgmap v297: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:53:38.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:38 smithi190 ceph-mon[31421]: pgmap v297: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:53:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:38 smithi079 ceph-mon[27946]: pgmap v297: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:53:40.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:40 smithi193 ceph-mon[31815]: pgmap v298: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:40.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:40 smithi190 ceph-mon[31421]: pgmap v298: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:40 smithi079 ceph-mon[27946]: pgmap v298: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:42.641 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:53:42.667 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:42 smithi079 ceph-mon[27946]: pgmap v299: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:42.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:42 smithi193 ceph-mon[31815]: pgmap v299: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:42.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:42 smithi190 ceph-mon[31421]: pgmap v299: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:42.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 -- 172.21.15.79:0/1469393825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c101b40 msgr2=0x7f2b8c10dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:42.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 --2- 172.21.15.79:0/1469393825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c101b40 0x7f2b8c10dcb0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009ca0 tx=0x7f2b8402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 -- 172.21.15.79:0/1469393825 shutdown_connections 2024-04-22T06:53:42.754 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 --2- 172.21.15.79:0/1469393825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 --2- 172.21.15.79:0/1469393825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c101b40 0x7f2b8c10dcb0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 --2- 172.21.15.79:0/1469393825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c105ea0 0x7f2b8c106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 -- 172.21.15.79:0/1469393825 >> 172.21.15.79:0/1469393825 conn(0x7f2b8c077e30 msgr2=0x7f2b8c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 -- 172.21.15.79:0/1469393825 shutdown_connections 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.753+0000 7f2b9406d640 1 -- 172.21.15.79:0/1469393825 wait complete. 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 Processor -- start 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 -- start start 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c10e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b8c101b40 0x7f2b8c10ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c105ea0 0x7f2b8c10f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c112260 con 0x7f2b8c1011a0 2024-04-22T06:53:42.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c1120e0 con 0x7f2b8c101b40 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9406d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c1123e0 con 0x7f2b8c105ea0 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c10e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c10e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32790/0 (socket says 172.21.15.79:32790) 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 -- 172.21.15.79:0/79210667 learned_addr learned my addr 172.21.15.79:0/79210667 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b9260a640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c105ea0 0x7f2b8c10f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91608640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b8c101b40 0x7f2b8c10ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 -- 172.21.15.79:0/79210667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c105ea0 msgr2=0x7f2b8c10f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c105ea0 0x7f2b8c10f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 -- 172.21.15.79:0/79210667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b8c101b40 msgr2=0x7f2b8c10ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b8c101b40 0x7f2b8c10ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.754+0000 7f2b91e09640 1 -- 172.21.15.79:0/79210667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c1a6ba0 con 0x7f2b8c1011a0 2024-04-22T06:53:42.756 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.755+0000 7f2b91e09640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c10e740 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c890 tx=0x7f2b8000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:42.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.755+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80004220 con 0x7f2b8c1011a0 2024-04-22T06:53:42.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.755+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b800043c0 con 0x7f2b8c1011a0 2024-04-22T06:53:42.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.755+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c1a6e90 con 0x7f2b8c1011a0 2024-04-22T06:53:42.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.755+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80004a00 con 0x7f2b8c1011a0 2024-04-22T06:53:42.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.755+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c1a7350 con 0x7f2b8c1011a0 2024-04-22T06:53:42.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.756+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8c069230 con 0x7f2b8c1011a0 2024-04-22T06:53:42.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.757+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2b800079b0 con 0x7f2b8c1011a0 2024-04-22T06:53:42.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.758+0000 7f2b7effd640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b780a3d50 0x7f2b780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:42.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.758+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f2b800d0900 con 0x7f2b8c1011a0 2024-04-22T06:53:42.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.758+0000 7f2b91608640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b780a3d50 0x7f2b780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:42.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.758+0000 7f2b91608640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b780a3d50 0x7f2b780a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c110180 tx=0x7f2b88008040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.762+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b80014030 con 0x7f2b8c1011a0 2024-04-22T06:53:42.975 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.977+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b8c108de0 con 0x7f2b8c1011a0 2024-04-22T06:53:42.976 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.978+0000 7f2b7effd640 1 -- 172.21.15.79:0/79210667 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+11882 (secure 0 0 0) 0x7f2b8009cba0 con 0x7f2b8c1011a0 2024-04-22T06:53:42.976 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:53:42.976 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:52:52.023532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25937,"mds_2":25927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25927":{"gid":25927,"name":"i","rank":2,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.193:6839/3285898346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3285898346},{"type":"v1","addr":"172.21.15.193:6839","nonce":3285898346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25937":{"gid":25937,"name":"h","rank":1,"incarnation":66,"state":"up:active","state_seq":26,"addr":"172.21.15.190:6837/1832563615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1832563615},{"type":"v1","addr":"172.21.15.190:6837","nonce":1832563615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25951":{"gid":25951,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/30505216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":30505216},{"type":"v1","addr":"172.21.15.193:6837","nonce":30505216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26000":{"gid":26000,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1769517674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1769517674},{"type":"v1","addr":"172.21.15.190:6835","nonce":1769517674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26003":{"gid":26003,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/2674386425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2674386425},{"type":"v1","addr":"172.21.15.193:6835","nonce":2674386425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25927,24467,25937]},"id":1}]} 2024-04-22T06:53:42.977 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 71 2024-04-22T06:53:42.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b780a3d50 msgr2=0x7f2b780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:42.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b780a3d50 0x7f2b780a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c110180 tx=0x7f2b88008040 comp rx=0 tx=0).stop 2024-04-22T06:53:42.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 msgr2=0x7f2b8c10e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c10e740 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000c890 tx=0x7f2b8000cd50 comp rx=0 tx=0).stop 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 shutdown_connections 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f2b780a3d50 0x7f2b780a6210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b8c101b40 0x7f2b8c10ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c105ea0 0x7f2b8c10f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 --2- 172.21.15.79:0/79210667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b8c1011a0 0x7f2b8c10e740 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 >> 172.21.15.79:0/79210667 conn(0x7f2b8c077e30 msgr2=0x7f2b8c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 shutdown_connections 2024-04-22T06:53:42.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:42.980+0000 7f2b9406d640 1 -- 172.21.15.79:0/79210667 wait complete. 2024-04-22T06:53:42.989 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id 71d1df0 --release --await' 2024-04-22T06:53:42.989 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id 71d1df0 --release --await 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 -- 172.21.15.79:0/2158321494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f81031d0 msgr2=0x7f82f81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 --2- 172.21.15.79:0/2158321494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f81031d0 0x7f82f81035b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009ca0 tx=0x7f82f402c230 comp rx=0 tx=0).stop 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 -- 172.21.15.79:0/2158321494 shutdown_connections 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 --2- 172.21.15.79:0/2158321494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82f8103af0 0x7f82f810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 --2- 172.21.15.79:0/2158321494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f810c510 0x7f82f810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 --2- 172.21.15.79:0/2158321494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f81031d0 0x7f82f81035b0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.135+0000 7f82ffad2640 1 -- 172.21.15.79:0/2158321494 >> 172.21.15.79:0/2158321494 conn(0x7f82f8077f50 msgr2=0x7f82f8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 -- 172.21.15.79:0/2158321494 shutdown_connections 2024-04-22T06:53:43.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 -- 172.21.15.79:0/2158321494 wait complete. 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 Processor -- start 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 -- start start 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f81031d0 0x7f82f8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82f8103af0 0x7f82f8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 0x7f82f819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81121a0 con 0x7f82f810c510 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f82f8112020 con 0x7f82f8103af0 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82ffad2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f82f8112320 con 0x7f82f81031d0 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82fe06f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 0x7f82f819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82fe06f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 0x7f82f819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32808/0 (socket says 172.21.15.79:32808) 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82fe06f640 1 -- 172.21.15.79:0/1208339507 learned_addr learned my addr 172.21.15.79:0/1208339507 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.136+0000 7f82fd86e640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f81031d0 0x7f82f8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fe06f640 1 -- 172.21.15.79:0/1208339507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f81031d0 msgr2=0x7f82f8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fe06f640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f81031d0 0x7f82f8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fe06f640 1 -- 172.21.15.79:0/1208339507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82f8103af0 msgr2=0x7f82f8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fe06f640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82f8103af0 0x7f82f8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fe06f640 1 -- 172.21.15.79:0/1208339507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f819d730 con 0x7f82f810c510 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fd86e640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f81031d0 0x7f82f8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82fe06f640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 0x7f82f819d000 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f82f000e970 tx=0x7f82f000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82eeffd640 1 -- 172.21.15.79:0/1208339507 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f000ccc0 con 0x7f82f810c510 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82eeffd640 1 -- 172.21.15.79:0/1208339507 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f000ce60 con 0x7f82f810c510 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f819da20 con 0x7f82f810c510 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82eeffd640 1 -- 172.21.15.79:0/1208339507 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0010430 con 0x7f82f810c510 2024-04-22T06:53:43.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.137+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f81a52b0 con 0x7f82f810c510 2024-04-22T06:53:43.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.139+0000 7f82eeffd640 1 -- 172.21.15.79:0/1208339507 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f82f0004f90 con 0x7f82f810c510 2024-04-22T06:53:43.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 Processor -- start 2024-04-22T06:53:43.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82eeffd640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f82e40a3d20 0x7f82e40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 -- start start 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580310 0x564db25806f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 0x564db25892d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fd06d640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f82e40a3d20 0x7f82e40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db258bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82eeffd640 1 -- 172.21.15.79:0/1208339507 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f82f00d1090 con 0x7f82f810c510 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564db24e5500 con 0x564db2580c30 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564db248b6a0 con 0x564db2589810 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f830279e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564db24e14c0 con 0x564db2580310 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fd06d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 0x564db25892d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fd06d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 0x564db25892d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32816/0 (socket says 172.21.15.79:32816) 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fd06d640 1 -- 172.21.15.79:0/777867508 learned_addr learned my addr 172.21.15.79:0/777867508 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fe06f640 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db258bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fd86e640 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580310 0x564db25806f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.140+0000 7f82fd06d640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f82e40a3d20 0x7f82e40a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f82e800bf60 tx=0x7f82e800bf90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd06d640 1 -- 172.21.15.79:0/777867508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580310 msgr2=0x564db25806f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd06d640 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580310 0x564db25806f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd06d640 1 -- 172.21.15.79:0/777867508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 msgr2=0x564db258bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd06d640 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db258bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd06d640 1 -- 172.21.15.79:0/777867508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db24aa5e0 con 0x564db2580c30 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd86e640 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580310 0x564db25806f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:53:43.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82fd06d640 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 0x564db25892d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f82e8011510 tx=0x7f82e801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4837cc83edc9fad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82e3fff640 1 -- 172.21.15.79:0/777867508 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8014230 con 0x564db2580c30 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82e3fff640 1 -- 172.21.15.79:0/777867508 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82e80035f0 con 0x564db2580c30 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.141+0000 7f82e3fff640 1 -- 172.21.15.79:0/777867508 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e801e670 con 0x564db2580c30 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 -- 172.21.15.79:0/777867508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 msgr2=0x564db25892d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 0x564db25892d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f82e8011510 tx=0x7f82e801c000 comp rx=0 tx=0).stop 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 -- 172.21.15.79:0/777867508 shutdown_connections 2024-04-22T06:53:43.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db258bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580c30 0x564db25892d0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 --2- 172.21.15.79:0/777867508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580310 0x564db25806f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 -- 172.21.15.79:0/777867508 >> 172.21.15.79:0/777867508 conn(0x564db257aa40 msgr2=0x564db257ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 -- 172.21.15.79:0/777867508 shutdown_connections 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 -- 172.21.15.79:0/777867508 wait complete. 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 -- start start 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.142+0000 7f830279e740 1 Processor -- start 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 0x564db25866d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.149 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db2586c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db2587150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564db258c5a0 con 0x564db2580310 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564db249ea60 con 0x564db2589810 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564db258c720 con 0x564db2580c30 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 10 client.0 _ll_register_callbacks cb 0x564db21f7110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 0x564db25866d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 0x564db25866d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32818/0 (socket says 172.21.15.79:32818) 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 -- 172.21.15.79:0/3164866176 learned_addr learned my addr 172.21.15.79:0/3164866176 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd06d640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db2586c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fe06f640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db2587150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 -- 172.21.15.79:0/3164866176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 msgr2=0x564db2586c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db2586c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 -- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 msgr2=0x564db2587150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db2587150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 -- 172.21.15.79:0/3164866176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db21f7920 con 0x564db2580310 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd06d640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db2586c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82fd86e640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 0x564db25866d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f82f402c770 tx=0x7f82f4003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa3300006be3ee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82e17fa640 1 -- 172.21.15.79:0/3164866176 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4043070 con 0x564db2580310 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82e17fa640 1 -- 172.21.15.79:0/3164866176 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f82f4004240 con 0x564db2580310 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e80035f0 con 0x564db2580310 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f82e17fa640 1 -- 172.21.15.79:0/3164866176 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f82f404f050 con 0x564db2580310 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 10 client.17349 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:43.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.143+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x564db22c3d30 con 0x564db2580310 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f82e17fa640 1 -- 172.21.15.79:0/3164866176 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82e80035f0 con 0x564db2580310 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f82e17fa640 1 -- 172.21.15.79:0/3164866176 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f82f4032c90 con 0x564db2580310 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f830279e740 10 client.17349 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f830279e740 10 client.17349 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f830279e740 20 client.17349 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f830279e740 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2599960 0x564db259be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f830279e740 4 client.17349 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564db24b8a10 con 0x564db2599960 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.144+0000 7f82fd06d640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2599960 0x564db259be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f82fd06d640 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2599960 0x564db259be00 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f11e045a0ef07c2 server_cookie=51c453e199adfe27 in_seq=0 out_seq=0 2024-04-22T06:53:43.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f82e17fa640 10 client.17349 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f82e17fa640 1 -- 172.21.15.79:0/3164866176 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564db24b8a10 con 0x564db2599960 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f82e17fa640 10 client.17349 handle_command_reply: tid=0 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 client.17349 shutdown 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 msgr2=0x564db25866d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 0x564db25866d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f82f402c770 tx=0x7f82f4003eb0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 shutdown_connections 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2599960 0x564db259be00 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2589810 0x564db2587150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f82e17fa640 0 client.17349 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db2586c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 --2- 172.21.15.79:0/3164866176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2580310 0x564db25866d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.145+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 >> 172.21.15.79:0/3164866176 conn(0x564db257aa40 msgr2=0x564db257c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.146+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 shutdown_connections 2024-04-22T06:53:43.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.146+0000 7f830279e740 1 -- 172.21.15.79:0/3164866176 wait complete. 2024-04-22T06:53:43.154 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.146+0000 7f830279e740 20 client.17349 trim_cache size 0 max 16384 2024-04-22T06:53:43.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.162+0000 7f830279e740 1 Processor -- start 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.162+0000 7f830279e740 1 -- start start 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f830279e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db2595460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f830279e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db25798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f830279e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db25a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f830279e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81b0e60 con 0x564db2589810 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f830279e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81ae730 con 0x564db2580310 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f830279e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81ac5a0 con 0x564db2580c30 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db25a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db25a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32820/0 (socket says 172.21.15.79:32820) 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 -- 172.21.15.79:0/2367896051 learned_addr learned my addr 172.21.15.79:0/2367896051 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.166 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fd06d640 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db25798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fd86e640 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db2595460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 -- 172.21.15.79:0/2367896051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 msgr2=0x564db25798e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db25798e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 -- 172.21.15.79:0/2367896051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 msgr2=0x564db2595460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db2595460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 -- 172.21.15.79:0/2367896051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db2580a60 con 0x564db2589810 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fd86e640 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db2595460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82fe06f640 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db25a16d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ce440 tx=0x7f82f009c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df8396a2f2485de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82e37fe640 1 -- 172.21.15.79:0/2367896051 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00b6070 con 0x564db2589810 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82e37fe640 1 -- 172.21.15.79:0/2367896051 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f009d330 con 0x564db2589810 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.163+0000 7f82e37fe640 1 -- 172.21.15.79:0/2367896051 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0087040 con 0x564db2589810 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 -- 172.21.15.79:0/2367896051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 msgr2=0x564db25a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db25a16d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ce440 tx=0x7f82f009c6a0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.167 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 -- 172.21.15.79:0/2367896051 shutdown_connections 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db25a16d0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db25798e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 --2- 172.21.15.79:0/2367896051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db2595460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 -- 172.21.15.79:0/2367896051 >> 172.21.15.79:0/2367896051 conn(0x564db257aa40 msgr2=0x564db257c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 -- 172.21.15.79:0/2367896051 shutdown_connections 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 -- 172.21.15.79:0/2367896051 wait complete. 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 -- start start 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 Processor -- start 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db258e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.164+0000 7f830279e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db258e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.168 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db258ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81acbf0 con 0x564db2589810 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81ae060 con 0x564db2580310 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f82f81ad2c0 con 0x564db2580c30 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 10 client.0 _ll_register_callbacks cb 0x7f82f0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db258ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db258ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32834/0 (socket says 172.21.15.79:32834) 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 -- 172.21.15.79:0/1084308485 learned_addr learned my addr 172.21.15.79:0/1084308485 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fd06d640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db258e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 -- 172.21.15.79:0/1084308485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 msgr2=0x564db258e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db258e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fd86e640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db258e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 -- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 msgr2=0x564db258e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db258e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 -- 172.21.15.79:0/1084308485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db25fff50 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fd06d640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db258e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82fe06f640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db258ed60 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f82f000e940 tx=0x7f82f0004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ae2ed0bc4fe949 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82e17fa640 1 -- 172.21.15.79:0/1084308485 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0087040 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82e17fa640 1 -- 172.21.15.79:0/1084308485 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f82f00992a0 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f009d330 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82e17fa640 1 -- 172.21.15.79:0/1084308485 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f82f00b6040 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 10 client.17355 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x564db2601140 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.165+0000 7f82e17fa640 1 -- 172.21.15.79:0/1084308485 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82f009d330 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f82e17fa640 1 -- 172.21.15.79:0/1084308485 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f82f00a5410 con 0x564db2589810 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f830279e740 10 client.17355 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f830279e740 10 client.17355 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:53:43.170 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f830279e740 20 client.17355 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:43.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f830279e740 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2601700 0x564db2603af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f830279e740 4 client.17355 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "71d1df0", "release": true, "await": true}] 2024-04-22T06:53:43.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "71d1df0", "release": true, "await": true}) -- 0x564db24f2f20 con 0x564db2601700 2024-04-22T06:53:43.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f82fd86e640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2601700 0x564db2603af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f82fd86e640 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2601700 0x564db2603af0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49f0e3b1a451484 server_cookie=7d3109d94838ee08 in_seq=0 out_seq=0 2024-04-22T06:53:43.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.166+0000 7f82e17fa640 10 client.17355 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f82e17fa640 1 -- 172.21.15.79:0/1084308485 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x564db24f2f20 con 0x564db2601700 2024-04-22T06:53:43.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f82e17fa640 10 client.17355 handle_command_reply: tid=0 2024-04-22T06:53:43.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 client.17355 shutdown 2024-04-22T06:53:43.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 msgr2=0x564db258ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db258ed60 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f82f000e940 tx=0x7f82f0004780 comp rx=0 tx=0).stop 2024-04-22T06:53:43.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 shutdown_connections 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564db2601700 0x564db2603af0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564db2589810 0x564db258ed60 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f82e17fa640 0 client.17355 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564db2580c30 0x564db258e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 --2- 172.21.15.79:0/1084308485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564db2580310 0x564db258e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 >> 172.21.15.79:0/1084308485 conn(0x564db257aa40 msgr2=0x564db257bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 shutdown_connections 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 1 -- 172.21.15.79:0/1084308485 wait complete. 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.169+0000 7f830279e740 20 client.17355 trim_cache size 0 max 16384 2024-04-22T06:53:43.173 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 71, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 30, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "71d1df0": { 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 30, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 78.0, 2024-04-22T06:53:43.174 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:43.175 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:53:43.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f82e40a3d20 msgr2=0x7f82e40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f82e40a3d20 0x7f82e40a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f82e800bf60 tx=0x7f82e800bf90 comp rx=0 tx=0).stop 2024-04-22T06:53:43.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 msgr2=0x7f82f819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 0x7f82f819d000 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f82f000e970 tx=0x7f82f000ed70 comp rx=0 tx=0).stop 2024-04-22T06:53:43.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 shutdown_connections 2024-04-22T06:53:43.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f82e40a3d20 0x7f82e40a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82f8103af0 0x7f82f8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82f810c510 0x7f82f819d000 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 --2- 172.21.15.79:0/1208339507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f81031d0 0x7f82f8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 >> 172.21.15.79:0/1208339507 conn(0x7f82f8077f50 msgr2=0x7f82f80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 shutdown_connections 2024-04-22T06:53:43.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.170+0000 7f82ffad2640 1 -- 172.21.15.79:0/1208339507 wait complete. 2024-04-22T06:53:43.179 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 71d1df0, seconds elapsed: 19.1 2024-04-22T06:53:43.179 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.05 2024-04-22T06:53:43.179 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:53:43.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 -- 172.21.15.79:0/2529815730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 msgr2=0x7f811010ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 --2- 172.21.15.79:0/2529815730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f811010ee10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8108009ca0 tx=0x7f810802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 -- 172.21.15.79:0/2529815730 shutdown_connections 2024-04-22T06:53:43.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 --2- 172.21.15.79:0/2529815730 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f811010ee10 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 --2- 172.21.15.79:0/2529815730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8110102b20 0x7f8110106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.280 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 --2- 172.21.15.79:0/2529815730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8110102170 0x7f8110102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 -- 172.21.15.79:0/2529815730 >> 172.21.15.79:0/2529815730 conn(0x7f81100793a0 msgr2=0x7f81100797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 -- 172.21.15.79:0/2529815730 shutdown_connections 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.279+0000 7f811609a640 1 -- 172.21.15.79:0/2529815730 wait complete. 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 Processor -- start 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 -- start start 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8110102170 0x7f81101986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f8110198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811019cf80 0x7f811019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f81101121a0 con 0x7f8110107240 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8110112020 con 0x7f811019cf80 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f811609a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8110112320 con 0x7f8110102170 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f810effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f8110198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f810effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f8110198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32850/0 (socket says 172.21.15.79:32850) 2024-04-22T06:53:43.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f810effd640 1 -- 172.21.15.79:0/3119424939 learned_addr learned my addr 172.21.15.79:0/3119424939 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.280+0000 7f810f7fe640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8110102170 0x7f81101986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810ffff640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811019cf80 0x7f811019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810effd640 1 -- 172.21.15.79:0/3119424939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8110102170 msgr2=0x7f81101986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810effd640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8110102170 0x7f81101986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810effd640 1 -- 172.21.15.79:0/3119424939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811019cf80 msgr2=0x7f811019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810effd640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811019cf80 0x7f811019d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810effd640 1 -- 172.21.15.79:0/3119424939 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811019da70 con 0x7f8110107240 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810effd640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f8110198c00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f810400e9f0 tx=0x7f810400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810400ccc0 con 0x7f8110107240 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81040054d0 con 0x7f8110107240 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104010430 con 0x7f8110107240 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811019dd60 con 0x7f8110107240 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.281+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81101a5550 con 0x7f8110107240 2024-04-22T06:53:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.282+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8110069230 con 0x7f8110107240 2024-04-22T06:53:43.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.283+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f81040046c0 con 0x7f8110107240 2024-04-22T06:53:43.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.284+0000 7f810cff9640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80f80a3da0 0x7f80f80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.283 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.284+0000 7f810f7fe640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80f80a3da0 0x7f80f80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.284+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f81040d11d0 con 0x7f8110107240 2024-04-22T06:53:43.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.284+0000 7f810f7fe640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80f80a3da0 0x7f80f80a6260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8100009b60 tx=0x7f8100005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.287+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8104053ff0 con 0x7f8110107240 2024-04-22T06:53:43.425 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f790d2f8700 time 2024-04-22T06:53:43.168265+0000 2024-04-22T06:53:43.425 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.425 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f79209333dd] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: clone() 2024-04-22T06:53:43.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: *** Caught signal (Aborted) ** 2024-04-22T06:53:43.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: in thread 7f790d2f8700 thread_name:quiesce.agt 2024-04-22T06:53:43.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.168+0000 7f790d2f8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f790d2f8700 time 2024-04-22T06:53:43.168265+0000 2024-04-22T06:53:43.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f79209333dd] 2024-04-22T06:53:43.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: clone() 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.430 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f791f6bbd20] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: gsignal() 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: abort() 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7920933437] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.431 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 13: clone() 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.169+0000 7f790d2f8700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: in thread 7f790d2f8700 thread_name:quiesce.agt 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f791f6bbd20] 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: gsignal() 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: abort() 2024-04-22T06:53:43.432 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7920933437] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.433 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 13: clone() 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: -2> 2024-04-22T06:53:43.168+0000 7f790d2f8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f790d2f8700 time 2024-04-22T06:53:43.168265+0000 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f79209333dd] 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.434 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: clone() 2024-04-22T06:53:43.435 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 0> 2024-04-22T06:53:43.169+0000 7f790d2f8700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: in thread 7f790d2f8700 thread_name:quiesce.agt 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f791f6bbd20] 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: gsignal() 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: abort() 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7920933437] 2024-04-22T06:53:43.436 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 13: clone() 2024-04-22T06:53:43.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: -2> 2024-04-22T06:53:43.168+0000 7f790d2f8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f790d2f8700 time 2024-04-22T06:53:43.168265+0000 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f79209333dd] 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.438 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: clone() 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 0> 2024-04-22T06:53:43.169+0000 7f790d2f8700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: in thread 7f790d2f8700 thread_name:quiesce.agt 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f791f6bbd20] 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2: gsignal() 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 3: abort() 2024-04-22T06:53:43.439 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7920933437] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f79209335a3] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563571776e9c] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563571776f9f] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5635716a50d3] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5635716a503c] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563571924d3f] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56357198d560] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f791f6b11ca] 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 13: clone() 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[67921]: 2024-04-22T06:53:43.440 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 podman[68474]: 2024-04-22 06:53:43.227820463 +0000 UTC m=+0.021565388 container died 93d7c27cf50a56f622538e7240da7cb55ccf431f5d8d62a537073a02e541ac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f842aae8700 time 2024-04-22T06:53:43.168108+0000 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f843e1233dd] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.441 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: clone() 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.169+0000 7f842aae8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f842aae8700 time 2024-04-22T06:53:43.168108+0000 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f843e1233dd] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.442 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: clone() 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: *** Caught signal (Aborted) ** 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: in thread 7f842aae8700 thread_name:quiesce.agt 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f843ceabd20] 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: gsignal() 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: abort() 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f843e123437] 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.443 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 13: clone() 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.170+0000 7f842aae8700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: in thread 7f842aae8700 thread_name:quiesce.agt 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f843ceabd20] 2024-04-22T06:53:43.444 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: gsignal() 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: abort() 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f843e123437] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 13: clone() 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:53:43.445 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: -2> 2024-04-22T06:53:43.169+0000 7f842aae8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f842aae8700 time 2024-04-22T06:53:43.168108+0000 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f843e1233dd] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.446 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: clone() 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 0> 2024-04-22T06:53:43.170+0000 7f842aae8700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: in thread 7f842aae8700 thread_name:quiesce.agt 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f843ceabd20] 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: gsignal() 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: abort() 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f843e123437] 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.447 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 13: clone() 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: -2> 2024-04-22T06:53:43.169+0000 7f842aae8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f842aae8700 time 2024-04-22T06:53:43.168108+0000 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.448 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f843e1233dd] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: clone() 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 0> 2024-04-22T06:53:43.170+0000 7f842aae8700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:53:43.449 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: in thread 7f842aae8700 thread_name:quiesce.agt 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f843ceabd20] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2: gsignal() 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 3: abort() 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f843e123437] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f843e1235a3] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e578243e9c] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e578243f9f] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e5781720d3] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e57817203c] 2024-04-22T06:53:43.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e5783f1d3f] 2024-04-22T06:53:43.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e57845a560] 2024-04-22T06:53:43.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f843cea11ca] 2024-04-22T06:53:43.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 13: clone() 2024-04-22T06:53:43.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:53:43.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[66770]: 2024-04-22T06:53:43.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:43 smithi190 podman[67242]: 2024-04-22 06:53:43.229910421 +0000 UTC m=+0.021539428 container died 74377163fcad3dc5106d81795dd503e663692d2f838215a96ecd317381a1ad8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:53:43.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.505+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8110199a10 con 0x7f8110107240 2024-04-22T06:53:43.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.506+0000 7f810cff9640 1 -- 172.21.15.79:0/3119424939 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+11882 (secure 0 0 0) 0x7f8110199a10 con 0x7f8110107240 2024-04-22T06:53:43.504 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:53:43.505 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:52:52.023532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":25937,"mds_2":25927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25927":{"gid":25927,"name":"i","rank":2,"incarnation":64,"state":"up:active","state_seq":26,"addr":"172.21.15.193:6839/3285898346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3285898346},{"type":"v1","addr":"172.21.15.193:6839","nonce":3285898346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25937":{"gid":25937,"name":"h","rank":1,"incarnation":66,"state":"up:active","state_seq":26,"addr":"172.21.15.190:6837/1832563615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1832563615},{"type":"v1","addr":"172.21.15.190:6837","nonce":1832563615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25951":{"gid":25951,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/30505216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":30505216},{"type":"v1","addr":"172.21.15.193:6837","nonce":30505216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26000":{"gid":26000,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1769517674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1769517674},{"type":"v1","addr":"172.21.15.190:6835","nonce":1769517674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26003":{"gid":26003,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/2674386425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2674386425},{"type":"v1","addr":"172.21.15.193:6835","nonce":2674386425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[25927,24467,25937]},"id":1}]} 2024-04-22T06:53:43.505 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 71 2024-04-22T06:53:43.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.507+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80f80a3da0 msgr2=0x7f80f80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.507+0000 7f811609a640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80f80a3da0 0x7f80f80a6260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8100009b60 tx=0x7f8100005f90 comp rx=0 tx=0).stop 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.507+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 msgr2=0x7f8110198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.507+0000 7f811609a640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f8110198c00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f810400e9f0 tx=0x7f810400eeb0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 shutdown_connections 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f80f80a3da0 0x7f80f80a6260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f811019cf80 0x7f811019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8110107240 0x7f8110198c00 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 --2- 172.21.15.79:0/3119424939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8110102170 0x7f81101986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 >> 172.21.15.79:0/3119424939 conn(0x7f81100793a0 msgr2=0x7f81100fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 shutdown_connections 2024-04-22T06:53:43.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.508+0000 7f811609a640 1 -- 172.21.15.79:0/3119424939 wait complete. 2024-04-22T06:53:43.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:43 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/79210667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:43.517 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 78.05 --await' 2024-04-22T06:53:43.517 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 78.05 --await 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 -- 172.21.15.79:0/1624200016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4101a50 msgr2=0x7f04b4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 --2- 172.21.15.79:0/1624200016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4101a50 0x7f04b4105a90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f049c009900 tx=0x7f049c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 -- 172.21.15.79:0/1624200016 shutdown_connections 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 --2- 172.21.15.79:0/1624200016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4106170 0x7f04b410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 --2- 172.21.15.79:0/1624200016 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4101a50 0x7f04b4105a90 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 --2- 172.21.15.79:0/1624200016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b41010a0 0x7f04b4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 -- 172.21.15.79:0/1624200016 >> 172.21.15.79:0/1624200016 conn(0x7f04b4077f20 msgr2=0x7f04b4078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 -- 172.21.15.79:0/1624200016 shutdown_connections 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.626+0000 7f04b99d5640 1 -- 172.21.15.79:0/1624200016 wait complete. 2024-04-22T06:53:43.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 Processor -- start 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 -- start start 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b41010a0 0x7f04b41973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4106170 0x7f04b41978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4197e20 0x7f04b41a6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4111110 con 0x7f04b4106170 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4110f90 con 0x7f04b4197e20 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b99d5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4111290 con 0x7f04b41010a0 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4106170 0x7f04b41978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4106170 0x7f04b41978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32868/0 (socket says 172.21.15.79:32868) 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 -- 172.21.15.79:0/1809909744 learned_addr learned my addr 172.21.15.79:0/1809909744 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b41010a0 0x7f04b41973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4197e20 0x7f04b41a6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 -- 172.21.15.79:0/1809909744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b41010a0 msgr2=0x7f04b41973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b41010a0 0x7f04b41973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 -- 172.21.15.79:0/1809909744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4197e20 msgr2=0x7f04b41a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4197e20 0x7f04b41a6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 -- 172.21.15.79:0/1809909744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b41a6fe0 con 0x7f04b4106170 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b41010a0 0x7f04b41973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.627+0000 7f04abfff640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4106170 0x7f04b41978e0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f049c000c00 tx=0x7f049c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.628+0000 7f04b89d3640 1 -- 172.21.15.79:0/1809909744 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c043020 con 0x7f04b4106170 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.628+0000 7f04b99d5640 1 -- 172.21.15.79:0/1809909744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b41a7270 con 0x7f04b4106170 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.628+0000 7f04b89d3640 1 -- 172.21.15.79:0/1809909744 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049c0052b0 con 0x7f04b4106170 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.628+0000 7f04b89d3640 1 -- 172.21.15.79:0/1809909744 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c003d40 con 0x7f04b4106170 2024-04-22T06:53:43.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.628+0000 7f04b99d5640 1 -- 172.21.15.79:0/1809909744 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b41a77b0 con 0x7f04b4106170 2024-04-22T06:53:43.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b89d3640 1 -- 172.21.15.79:0/1809909744 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f049c046020 con 0x7f04b4106170 2024-04-22T06:53:43.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 Processor -- start 2024-04-22T06:53:43.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 -- start start 2024-04-22T06:53:43.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 0x562b11f14da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562b11e750b0 con 0x562b11f149c0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562b11e26680 con 0x562b11f1dec0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b11e32c60 con 0x562b11f152e0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b89d3640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a00a3d20 0x7f04a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 0x562b11f14da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 0x562b11f14da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32884/0 (socket says 172.21.15.79:32884) 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b37fe640 1 -- 172.21.15.79:0/1347118289 learned_addr learned my addr 172.21.15.79:0/1347118289 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a00a3d20 0x7f04a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b89d3640 1 -- 172.21.15.79:0/1809909744 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f049c0f29d0 con 0x7f04b4106170 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04abfff640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.630+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 -- 172.21.15.79:0/1347118289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 msgr2=0x562b11f1d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 -- 172.21.15.79:0/1347118289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 msgr2=0x562b11f202b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f202b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 -- 172.21.15.79:0/1347118289 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b11c57c90 con 0x562b11f149c0 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04abfff640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f202b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 0x562b11f14da0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0149a0 tx=0x7f04ac014e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311d3f0229b2f241 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04a9ffb640 1 -- 172.21.15.79:0/1347118289 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac021070 con 0x562b11f149c0 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04a9ffb640 1 -- 172.21.15.79:0/1347118289 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ac01cc70 con 0x562b11f149c0 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04a9ffb640 1 -- 172.21.15.79:0/1347118289 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac013960 con 0x562b11f149c0 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1809909744 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a00a3d20 0x7f04a00a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f04ac01e000 tx=0x7f04ac01a000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1347118289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 msgr2=0x562b11f14da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.631+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 0x562b11f14da0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0149a0 tx=0x7f04ac014e60 comp rx=0 tx=0).stop 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1347118289 shutdown_connections 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f202b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1347118289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f149c0 0x562b11f14da0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1347118289 >> 172.21.15.79:0/1347118289 conn(0x562b11f0f0f0 msgr2=0x562b11f0f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1347118289 shutdown_connections 2024-04-22T06:53:43.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1347118289 wait complete. 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- start start 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 Processor -- start 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f149c0 0x562b11f1aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562b11e78400 con 0x562b11f1dec0 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562b11e34840 con 0x562b11f149c0 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b11e7d720 con 0x562b11f152e0 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04bc5c7740 10 client.0 _ll_register_callbacks cb 0x562b11e85ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:43.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.632+0000 7f04b37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32890/0 (socket says 172.21.15.79:32890) 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 -- 172.21.15.79:0/1802161554 learned_addr learned my addr 172.21.15.79:0/1802161554 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f149c0 0x562b11f1aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 -- 172.21.15.79:0/1802161554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 msgr2=0x562b11f1b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 -- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f149c0 msgr2=0x562b11f1aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f149c0 0x562b11f1aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 -- 172.21.15.79:0/1802161554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b11cb8420 con 0x562b11f1dec0 2024-04-22T06:53:43.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04b37fe640 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b930 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0135e0 tx=0x7f04ac013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9c1f93ff4da098 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f0492ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f0492ffd640 1 -- 172.21.15.79:0/1802161554 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac021070 con 0x562b11f1dec0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f0492ffd640 1 -- 172.21.15.79:0/1802161554 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f04ac013950 con 0x562b11f1dec0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04ac01cc70 con 0x562b11f1dec0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f0492ffd640 1 -- 172.21.15.79:0/1802161554 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f04ac0049c0 con 0x562b11f1dec0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04bc5c7740 10 client.17373 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x562b11b4b580 con 0x562b11f1dec0 2024-04-22T06:53:43.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.633+0000 7f0492ffd640 1 -- 172.21.15.79:0/1802161554 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ac01cc70 con 0x562b11f1dec0 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f0492ffd640 1 -- 172.21.15.79:0/1802161554 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f04ac02c070 con 0x562b11f1dec0 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04bc5c7740 10 client.17373 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04bc5c7740 10 client.17373 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04bc5c7740 20 client.17373 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f2d960 0x562b11f2fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04bc5c7740 4 client.17373 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b11e86b10 con 0x562b11f2d960 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f2d960 0x562b11f2fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.645 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f2d960 0x562b11f2fd70 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=824a073e8b1d64c1 server_cookie=3c0d980a50bc136f in_seq=0 out_seq=0 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.634+0000 7f0492ffd640 10 client.17373 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f0492ffd640 1 -- 172.21.15.79:0/1802161554 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b11e86b10 con 0x562b11f2d960 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f0492ffd640 10 client.17373 handle_command_reply: tid=0 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 client.17373 shutdown 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 msgr2=0x562b11f1b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b930 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0135e0 tx=0x7f04ac013f90 comp rx=0 tx=0).stop 2024-04-22T06:53:43.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 shutdown_connections 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f2d960 0x562b11f2fd70 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b930 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f0492ffd640 0 client.17373 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f152e0 0x562b11f1b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/1802161554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f149c0 0x562b11f1aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 >> 172.21.15.79:0/1802161554 conn(0x562b11f0f0f0 msgr2=0x562b11f10e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 shutdown_connections 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 1 -- 172.21.15.79:0/1802161554 wait complete. 2024-04-22T06:53:43.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.635+0000 7f04bc5c7740 20 client.17373 trim_cache size 0 max 16384 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.652+0000 7f04bc5c7740 1 Processor -- start 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.652+0000 7f04bc5c7740 1 -- start start 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.652+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f29870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.652+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f152e0 0x562b11f1a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.652+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41b0810 con 0x562b11f1dec0 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41ae680 con 0x562b11f152e0 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41b0140 con 0x562b11f149c0 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32896/0 (socket says 172.21.15.79:32896) 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 -- 172.21.15.79:0/2890541410 learned_addr learned my addr 172.21.15.79:0/2890541410 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f29870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 -- 172.21.15.79:0/2890541410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 msgr2=0x562b11f29870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f29870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 -- 172.21.15.79:0/2890541410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f152e0 msgr2=0x562b11f1a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f152e0 0x562b11f1a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 -- 172.21.15.79:0/2890541410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b11f19400 con 0x562b11f1dec0 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f29870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04b37fe640 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b580 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00aab0 tx=0x7f04ac013690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40112bdb4e69069e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04a97fa640 1 -- 172.21.15.79:0/2890541410 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac018020 con 0x562b11f1dec0 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04a97fa640 1 -- 172.21.15.79:0/2890541410 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ac0117b0 con 0x562b11f1dec0 2024-04-22T06:53:43.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.653+0000 7f04a97fa640 1 -- 172.21.15.79:0/2890541410 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac021070 con 0x562b11f1dec0 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2890541410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 msgr2=0x562b11f1b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b580 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00aab0 tx=0x7f04ac013690 comp rx=0 tx=0).stop 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2890541410 shutdown_connections 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f152e0 0x562b11f1a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f1dec0 0x562b11f1b580 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2890541410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f29870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2890541410 >> 172.21.15.79:0/2890541410 conn(0x562b11f0f0f0 msgr2=0x562b11f1ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2890541410 shutdown_connections 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2890541410 wait complete. 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- start start 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 Processor -- start 2024-04-22T06:53:43.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f21710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 0x562b11f21c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f22190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41af3a0 con 0x562b11f152e0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41aecd0 con 0x562b11f1dec0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41afa70 con 0x562b11f149c0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.654+0000 7f04bc5c7740 10 client.0 _ll_register_callbacks cb 0x7f04ac01f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 0x562b11f21c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 0x562b11f21c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32900/0 (socket says 172.21.15.79:32900) 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 -- 172.21.15.79:0/2334434582 learned_addr learned my addr 172.21.15.79:0/2334434582 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04b37fe640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f22190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f21710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 -- 172.21.15.79:0/2334434582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 msgr2=0x562b11f21710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f21710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 -- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 msgr2=0x562b11f22190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f22190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 -- 172.21.15.79:0/2334434582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b11f92eb0 con 0x562b11f152e0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04abfff640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 0x562b11f21c50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f049c005ec0 tx=0x7f049c0ba9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7509b4c9d0405d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f0492ffd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f0492ffd640 1 -- 172.21.15.79:0/2334434582 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0bdfe0 con 0x562b11f152e0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f0492ffd640 1 -- 172.21.15.79:0/2334434582 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f049c0bc980 con 0x562b11f152e0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04ac0117b0 con 0x562b11f152e0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f0492ffd640 1 -- 172.21.15.79:0/2334434582 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f049c0bed80 con 0x562b11f152e0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04bc5c7740 10 client.17379 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x562b11f940f0 con 0x562b11f152e0 2024-04-22T06:53:43.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.655+0000 7f0492ffd640 1 -- 172.21.15.79:0/2334434582 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ac0117b0 con 0x562b11f152e0 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f0492ffd640 1 -- 172.21.15.79:0/2334434582 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f049c0bacf0 con 0x562b11f152e0 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04bc5c7740 10 client.17379 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04bc5c7740 10 client.17379 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04bc5c7740 20 client.17379 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f94930 0x562b11f96d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04bc5c7740 4 client.17379 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.05, "await": true}] 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.05, "await": true}) -- 0x562b11e80980 con 0x562b11f94930 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f94930 0x562b11f96d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f0492ffd640 10 client.17379 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:43.661 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:43.656+0000 7f04b2ffd640 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f94930 0x562b11f96d50 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c040fa24d2cc51be server_cookie=b5db695b58590eae in_seq=0 out_seq=0 2024-04-22T06:53:43.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:43 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/79210667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:43.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:43 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/79210667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:44.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 podman[68474]: 2024-04-22 06:53:43.957858533 +0000 UTC m=+0.751603468 container remove 93d7c27cf50a56f622538e7240da7cb55ccf431f5d8d62a537073a02e541ac3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:53:44.426 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:43 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:53:44.427 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:44 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-22T06:53:44.427 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:44 smithi190 podman[67242]: 2024-04-22 06:53:44.073121907 +0000 UTC m=+0.864750941 container remove 74377163fcad3dc5106d81795dd503e663692d2f838215a96ecd317381a1ad8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T06:53:44.427 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:44 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:53:44.427 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:44 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:53:44.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:44 smithi193 ceph-mon[31815]: pgmap v300: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:44.691 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:44 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3119424939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:44 smithi190 ceph-mon[31421]: pgmap v300: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:44.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:44 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3119424939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:44.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:44 smithi079 ceph-mon[27946]: pgmap v300: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:44.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:44 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3119424939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:46.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:46 smithi193 ceph-mon[31815]: pgmap v301: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:46 smithi190 ceph-mon[31421]: pgmap v301: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:46.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:46 smithi079 ceph-mon[27946]: pgmap v301: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:47.611 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.610+0000 7f88b2175640 1 -- 172.21.15.79:0/1288168922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 msgr2=0x7f88ac101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.610+0000 7f88b2175640 1 --2- 172.21.15.79:0/1288168922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac101620 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8894009900 tx=0x7f889402bc80 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- 172.21.15.79:0/1288168922 shutdown_connections 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 --2- 172.21.15.79:0/1288168922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac101b60 0x7f88ac10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 --2- 172.21.15.79:0/1288168922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac101620 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 --2- 172.21.15.79:0/1288168922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ac106fb0 0x7f88ac107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- 172.21.15.79:0/1288168922 >> 172.21.15.79:0/1288168922 conn(0x7f88ac076c50 msgr2=0x7f88ac077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- 172.21.15.79:0/1288168922 shutdown_connections 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- 172.21.15.79:0/1288168922 wait complete. 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 Processor -- start 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- start start 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ac101b60 0x7f88ac1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac106fb0 0x7f88ac19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac111060 con 0x7f88ac1011a0 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac110ee0 con 0x7f88ac106fb0 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88b2175640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1111e0 con 0x7f88ac101b60 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac1974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32906/0 (socket says 172.21.15.79:32906) 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.611+0000 7f88ab7fe640 1 -- 172.21.15.79:0/2276765120 learned_addr learned my addr 172.21.15.79:0/2276765120 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88a3fff640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ac101b60 0x7f88ac1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88ab7fe640 1 -- 172.21.15.79:0/2276765120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ac101b60 msgr2=0x7f88ac1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ac101b60 0x7f88ac1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88ab7fe640 1 -- 172.21.15.79:0/2276765120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac106fb0 msgr2=0x7f88ac19bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88abfff640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac106fb0 0x7f88ac19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac106fb0 0x7f88ac19bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88ab7fe640 1 -- 172.21.15.79:0/2276765120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac19c490 con 0x7f88ac1011a0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88abfff640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac106fb0 0x7f88ac19bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac1974a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f889c00d950 tx=0x7f889c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88a97fa640 1 -- 172.21.15.79:0/2276765120 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c017030 con 0x7f88ac1011a0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88a97fa640 1 -- 172.21.15.79:0/2276765120 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f889c004020 con 0x7f88ac1011a0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ac19c780 con 0x7f88ac1011a0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88a97fa640 1 -- 172.21.15.79:0/2276765120 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c0047a0 con 0x7f88ac1011a0 2024-04-22T06:53:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.612+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ac1a4010 con 0x7f88ac1011a0 2024-04-22T06:53:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.614+0000 7f88a97fa640 1 -- 172.21.15.79:0/2276765120 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f889c009a30 con 0x7f88ac1011a0 2024-04-22T06:53:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 Processor -- start 2024-04-22T06:53:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 -- start start 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23d5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23d61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23e16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5648c23388d0 con 0x5648c23d6ae0 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5648c22d5e80 con 0x5648c23df2f0 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88b4d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648c2334890 con 0x5648c23d5d70 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88a97fa640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88980a3dc0 0x7f88980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23d5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23d5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32910/0 (socket says 172.21.15.79:32910) 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 -- 172.21.15.79:0/3126623475 learned_addr learned my addr 172.21.15.79:0/3126623475 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88980a3dc0 0x7f88980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88a3fff640 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23e16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88a97fa640 1 -- 172.21.15.79:0/2276765120 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f889c0d0680 con 0x7f88ac1011a0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23d61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 -- 172.21.15.79:0/3126623475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 msgr2=0x5648c23d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23d61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 -- 172.21.15.79:0/3126623475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 msgr2=0x5648c23e16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23e16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 -- 172.21.15.79:0/3126623475 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648c22f7320 con 0x5648c23d6ae0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88a3fff640 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23e16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.615+0000 7f88abfff640 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23d5830 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f88a40149d0 tx=0x7f88a4014e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e17754733f007 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88a1ffb640 1 -- 172.21.15.79:0/3126623475 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4021070 con 0x5648c23d6ae0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88a1ffb640 1 -- 172.21.15.79:0/3126623475 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a401cbb0 con 0x5648c23d6ae0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88abfff640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88980a3dc0 0x7f88980a6280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f88a401e000 tx=0x7f88a401b000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88a1ffb640 1 -- 172.21.15.79:0/3126623475 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4005680 con 0x5648c23d6ae0 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 -- 172.21.15.79:0/3126623475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 msgr2=0x5648c23d5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23d5830 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f88a40149d0 tx=0x7f88a4014e90 comp rx=0 tx=0).stop 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 -- 172.21.15.79:0/3126623475 shutdown_connections 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23e16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23d61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3126623475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23d5830 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 -- 172.21.15.79:0/3126623475 >> 172.21.15.79:0/3126623475 conn(0x5648c239ac90 msgr2=0x5648c23d0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 -- 172.21.15.79:0/3126623475 shutdown_connections 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.616+0000 7f88b4d64740 1 -- 172.21.15.79:0/3126623475 wait complete. 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 -- start start 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 Processor -- start 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23dc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5648c23e2020 con 0x5648c23d6ae0 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5648c22c0eb0 con 0x5648c23df2f0 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648c23e21a0 con 0x5648c23d5d70 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88b4d64740 10 client.0 _ll_register_callbacks cb 0x5648c22edb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23dc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32920/0 (socket says 172.21.15.79:32920) 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 -- 172.21.15.79:0/1500487151 learned_addr learned my addr 172.21.15.79:0/1500487151 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88abfff640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23dc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 -- 172.21.15.79:0/1500487151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 msgr2=0x5648c23decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23decc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 -- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 msgr2=0x5648c23dc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23dc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.617+0000 7f88a3fff640 1 -- 172.21.15.79:0/1500487151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648c204b210 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88a3fff640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23dc050 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f8894009900 tx=0x7f889402ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114db847c5c8490 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f888affd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f888affd640 1 -- 172.21.15.79:0/1500487151 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894005030 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f888affd640 1 -- 172.21.15.79:0/1500487151 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f88940051d0 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88a401cbb0 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f888affd640 1 -- 172.21.15.79:0/1500487151 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f88940053d0 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 10 client.17391 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x5648c2117620 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f888affd640 1 -- 172.21.15.79:0/1500487151 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a401cbb0 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f888affd640 1 -- 172.21.15.79:0/1500487151 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f8894003860 con 0x5648c23d6ae0 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 10 client.17391 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 10 client.17391 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 20 client.17391 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c23ef330 0x5648c23f1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 4 client.17391 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.618+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648c234c880 con 0x5648c23ef330 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.619+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c23ef330 0x5648c23f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.619+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c23ef330 0x5648c23f1720 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3053d2b15d0c70ae server_cookie=a46f030ed6aa038c in_seq=0 out_seq=0 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.619+0000 7f888affd640 10 client.17391 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.619+0000 7f888affd640 1 -- 172.21.15.79:0/1500487151 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648c234c880 con 0x5648c23ef330 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.619+0000 7f888affd640 10 client.17391 handle_command_reply: tid=0 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 client.17391 shutdown 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 msgr2=0x5648c23dc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23dc050 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f8894009900 tx=0x7f889402ca00 comp rx=0 tx=0).stop 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 shutdown_connections 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c23ef330 0x5648c23f1720 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23df2f0 0x5648c23dc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f888affd640 0 client.17391 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d6ae0 0x5648c23dc050 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1500487151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23d5d70 0x5648c23decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 >> 172.21.15.79:0/1500487151 conn(0x5648c239ac90 msgr2=0x5648c23d27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 shutdown_connections 2024-04-22T06:53:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 1 -- 172.21.15.79:0/1500487151 wait complete. 2024-04-22T06:53:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.620+0000 7f88b4d64740 20 client.17391 trim_cache size 0 max 16384 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.634+0000 7f88b4d64740 1 Processor -- start 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.634+0000 7f88b4d64740 1 -- start start 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23de780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88b4d64740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1aea80 con 0x5648c23d5d70 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88b4d64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1ac350 con 0x5648c23d6ae0 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88b4d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1aa1c0 con 0x5648c23df2f0 2024-04-22T06:53:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23eaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32922/0 (socket says 172.21.15.79:32922) 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 -- 172.21.15.79:0/1614875678 learned_addr learned my addr 172.21.15.79:0/1614875678 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88a3fff640 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88abfff640 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23de780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 -- 172.21.15.79:0/1614875678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 msgr2=0x5648c23de780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23de780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 -- 172.21.15.79:0/1614875678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 msgr2=0x5648c23d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 -- 172.21.15.79:0/1614875678 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648c23ec140 con 0x5648c23d5d70 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23eaf20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f889c0cdfd0 tx=0x7f889c098500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ac89101a047d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88a17fa640 1 -- 172.21.15.79:0/1614875678 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c0b5070 con 0x5648c23d5d70 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88a17fa640 1 -- 172.21.15.79:0/1614875678 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f889c0a05d0 con 0x5648c23d5d70 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.635+0000 7f88a17fa640 1 -- 172.21.15.79:0/1614875678 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c098890 con 0x5648c23d5d70 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 -- 172.21.15.79:0/1614875678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 msgr2=0x5648c23eaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23eaf20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f889c0cdfd0 tx=0x7f889c098500 comp rx=0 tx=0).stop 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 -- 172.21.15.79:0/1614875678 shutdown_connections 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23de780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 --2- 172.21.15.79:0/1614875678 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23eaf20 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 -- 172.21.15.79:0/1614875678 >> 172.21.15.79:0/1614875678 conn(0x5648c239ac90 msgr2=0x5648c23d70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 -- 172.21.15.79:0/1614875678 shutdown_connections 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 -- 172.21.15.79:0/1614875678 wait complete. 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 -- start start 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.636+0000 7f88b4d64740 1 Processor -- start 2024-04-22T06:53:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23d8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23d9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1aa810 con 0x5648c23d5d70 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1abc80 con 0x5648c23d6ae0 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1aaee0 con 0x5648c23df2f0 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 10 client.0 _ll_register_callbacks cb 0x7f8894005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23d8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23d8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:32934/0 (socket says 172.21.15.79:32934) 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 -- 172.21.15.79:0/3058144514 learned_addr learned my addr 172.21.15.79:0/3058144514 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:53:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88abfff640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23d9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88a3fff640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 -- 172.21.15.79:0/3058144514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 msgr2=0x5648c23d9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23d9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 -- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 msgr2=0x5648c23d9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 -- 172.21.15.79:0/3058144514 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648c24548a0 con 0x5648c23d5d70 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88ab7fe640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23d8c00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f889c09c1b0 tx=0x7f889c00b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38da463e2137646 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f888affd640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f888affd640 1 -- 172.21.15.79:0/3058144514 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c0b9020 con 0x5648c23d5d70 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f888affd640 1 -- 172.21.15.79:0/3058144514 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9545+0+0 (secure 0 0 0) 0x7f889c0cdc50 con 0x5648c23d5d70 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.637+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f889c0a05d0 con 0x5648c23d5d70 2024-04-22T06:53:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f888affd640 1 -- 172.21.15.79:0/3058144514 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f889c0b5040 con 0x5648c23d5d70 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 10 client.17397 fetch_fsmap learned FSMap version 71 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=4+}) -- 0x5648c2455a90 con 0x5648c23d5d70 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f888affd640 1 -- 172.21.15.79:0/3058144514 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f889c0a05d0 con 0x5648c23d5d70 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f888affd640 1 -- 172.21.15.79:0/3058144514 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 71) ==== 8338+0+0 (secure 0 0 0) 0x7f889c09b700 con 0x5648c23d5d70 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 10 client.17397 fetch_fsmap finished waiting for FSMap version 71 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 10 client.17397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 20 client.17397 populate_metadata read hostname 'smithi079' 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c2455f90 0x5648c24583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 4 client.17397 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648c22d83d0 con 0x5648c2455f90 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88a3fff640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c2455f90 0x5648c24583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f888affd640 10 client.17397 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.638+0000 7f88a3fff640 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c2455f90 0x5648c24583b0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aabdf0460cde0290 server_cookie=16a2bfbed32a8315 in_seq=0 out_seq=0 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f888affd640 1 -- 172.21.15.79:0/3058144514 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648c22d83d0 con 0x5648c2455f90 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f888affd640 10 client.17397 handle_command_reply: tid=0 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 client.17397 shutdown 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 msgr2=0x5648c23d8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23d8c00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f889c09c1b0 tx=0x7f889c00b510 comp rx=0 tx=0).stop 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 shutdown_connections 2024-04-22T06:53:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5648c2455f90 0x5648c24583b0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5648c23d6ae0 0x5648c23d9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c23df2f0 0x5648c23d9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f888affd640 0 client.17397 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 --2- 172.21.15.79:0/3058144514 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5648c23d5d70 0x5648c23d8c00 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 >> 172.21.15.79:0/3058144514 conn(0x5648c239ac90 msgr2=0x5648c23cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 shutdown_connections 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 1 -- 172.21.15.79:0/3058144514 wait complete. 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.642+0000 7f88b4d64740 20 client.17397 trim_cache size 0 max 16384 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:53:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:53:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:53:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88980a3dc0 msgr2=0x7f88980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88980a3dc0 0x7f88980a6280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f88a401e000 tx=0x7f88a401b000 comp rx=0 tx=0).stop 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 msgr2=0x7f88ac1974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac1974a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f889c00d950 tx=0x7f889c00de10 comp rx=0 tx=0).stop 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 shutdown_connections 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88980a3dc0 0x7f88980a6280 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88ac106fb0 0x7f88ac19bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88ac101b60 0x7f88ac1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 --2- 172.21.15.79:0/2276765120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88ac1011a0 0x7f88ac1974a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 >> 172.21.15.79:0/2276765120 conn(0x7f88ac076c50 msgr2=0x7f88ac0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 shutdown_connections 2024-04-22T06:53:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:53:47.643+0000 7f88b2175640 1 -- 172.21.15.79:0/2276765120 wait complete. 2024-04-22T06:53:47.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:53:47.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:53:47.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:53:47.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:53:47.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:53:47.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:53:47.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:53:47.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:53:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:53:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:53:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:53:47.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:53:48.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:48 smithi190 ceph-mon[31421]: pgmap v302: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:48.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:48 smithi079 ceph-mon[27946]: pgmap v302: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:48.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:48 smithi193 ceph-mon[31815]: pgmap v302: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:50.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:50 smithi190 ceph-mon[31421]: pgmap v303: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:53:50.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:50 smithi079 ceph-mon[27946]: pgmap v303: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:53:50.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:50 smithi193 ceph-mon[31815]: pgmap v303: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:53:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:52 smithi190 ceph-mon[31421]: pgmap v304: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:52.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:52 smithi079 ceph-mon[27946]: pgmap v304: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:52 smithi193 ceph-mon[31815]: pgmap v304: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:54.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:54 smithi193 ceph-mon[31815]: pgmap v305: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:54.699 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-04-22T06:53:54.699 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 systemd[1]: Stopped Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:54.699 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 systemd[1]: Starting Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:53:54.699 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 podman[68696]: 2024-04-22 06:53:54.582850747 +0000 UTC m=+0.018894436 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:53:54.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:54 smithi190 ceph-mon[31421]: pgmap v305: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:54.702 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:54 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-04-22T06:53:54.703 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:54 smithi190 systemd[1]: Stopped Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:54.703 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:54 smithi190 systemd[1]: Starting Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:53:54.703 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:54 smithi190 podman[67470]: 2024-04-22 06:53:54.597472889 +0000 UTC m=+0.017777327 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:53:54.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:54 smithi079 ceph-mon[27946]: pgmap v305: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:54.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f0492ffd640 1 -- 172.21.15.79:0/2334434582 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: -125 ) ==== 8+0+591 (crc 0 0 0) 0x562b11e80980 con 0x562b11f94930 2024-04-22T06:53:54.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f0492ffd640 10 client.17379 handle_command_reply: tid=0 2024-04-22T06:53:54.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 client.17379 shutdown 2024-04-22T06:53:54.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 msgr2=0x562b11f21c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:54.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 0x562b11f21c50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f049c005ec0 tx=0x7f049c0ba9e0 comp rx=0 tx=0).stop 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 shutdown_connections 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562b11f94930 0x562b11f96d50 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562b11f1dec0 0x562b11f22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562b11f152e0 0x562b11f21c50 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.953+0000 7f04bc5c7740 1 --2- 172.21.15.79:0/2334434582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b11f149c0 0x562b11f21710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.954+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 >> 172.21.15.79:0/2334434582 conn(0x562b11f0f0f0 msgr2=0x562b11f1b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.954+0000 7f0492ffd640 0 client.17379 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.954+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 shutdown_connections 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.954+0000 7f04bc5c7740 1 -- 172.21.15.79:0/2334434582 wait complete. 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:53:54.954+0000 7f04bc5c7740 20 client.17379 trim_cache size 0 max 16384 2024-04-22T06:53:54.953 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 72, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 32, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "570319f3": { 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 32, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "CANCELED", 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:53:54.954 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 78.0, 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": true, 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCING", 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 11.3 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:53:54.955 INFO:teuthology.orchestra.run.smithi079.stderr:Error ECANCELED: 2024-04-22T06:53:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-22T06:53:54.957 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 9).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-04-22T06:53:55.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 podman[68696]: 2024-04-22 06:53:54.701406669 +0000 UTC m=+0.137450376 container create 3323c3a73c73d8bdbe5cf94a182950b337b599e3055c480da0e5ffc313536bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-22T06:53:55.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 podman[68696]: 2024-04-22 06:53:54.921716644 +0000 UTC m=+0.357760315 container init 3323c3a73c73d8bdbe5cf94a182950b337b599e3055c480da0e5ffc313536bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:53:55.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 podman[68696]: 2024-04-22 06:53:54.924348449 +0000 UTC m=+0.360392117 container start 3323c3a73c73d8bdbe5cf94a182950b337b599e3055c480da0e5ffc313536bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:53:55.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 ceph-mds[68714]: -- 172.21.15.193:0/4060654342 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6801e5a40 con 0x55e6801b8c00 2024-04-22T06:53:55.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[68710]: starting mds.i at 2024-04-22T06:53:55.003 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:55 smithi193 bash[68696]: 3323c3a73c73d8bdbe5cf94a182950b337b599e3055c480da0e5ffc313536bd3 2024-04-22T06:53:55.116 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:54 smithi190 podman[67470]: 2024-04-22 06:53:54.848837291 +0000 UTC m=+0.269141719 container create 282b8251bd3244d42e2f064d4a3de60d5607e44ecf56735abca31eb4610bf4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T06:53:55.117 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:55 smithi190 podman[67470]: 2024-04-22 06:53:55.093301752 +0000 UTC m=+0.513606177 container init 282b8251bd3244d42e2f064d4a3de60d5607e44ecf56735abca31eb4610bf4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T06:53:55.117 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:55 smithi190 podman[67470]: 2024-04-22 06:53:55.096097662 +0000 UTC m=+0.516402099 container start 282b8251bd3244d42e2f064d4a3de60d5607e44ecf56735abca31eb4610bf4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:53:55.273 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:53:55 smithi193 systemd[1]: Started Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:55.450 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mds[67489]: -- 172.21.15.190:0/399445271 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55638a5eba40 con 0x55638a5bec00 2024-04-22T06:53:55.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[67485]: starting mds.h at 2024-04-22T06:53:55.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:55 smithi190 bash[67470]: 282b8251bd3244d42e2f064d4a3de60d5607e44ecf56735abca31eb4610bf4ab 2024-04-22T06:53:55.451 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:53:55 smithi190 systemd[1]: Started Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:55.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: daemon mds.i restarted 2024-04-22T06:53:55.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:53:55.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:53:55.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:boot 2024-04-22T06:53:55.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: daemon mds.h restarted 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:boot 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,2=e=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:53:55.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:55 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=e=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:55.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: daemon mds.i restarted 2024-04-22T06:53:55.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:boot 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: daemon mds.h restarted 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:boot 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,2=e=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:53:55.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:55 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=e=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: daemon mds.i restarted 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:boot 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:53:55.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: daemon mds.h restarted 2024-04-22T06:53:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:53:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:boot 2024-04-22T06:53:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-04-22T06:53:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,2=e=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:53:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:53:55.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:55 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:replay,2=e=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:57 smithi193 ceph-mon[31815]: pgmap v308: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:53:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:57 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:resolve 2024-04-22T06:53:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:57 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:resolve 2024-04-22T06:53:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:57 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=e=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:57 smithi190 ceph-mon[31421]: pgmap v308: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:53:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:57 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:resolve 2024-04-22T06:53:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:57 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:resolve 2024-04-22T06:53:57.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:57 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=e=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:57 smithi079 ceph-mon[27946]: pgmap v308: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:53:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:57 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:resolve 2024-04-22T06:53:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:57 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:resolve 2024-04-22T06:53:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:57 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:resolve,2=e=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:53:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0 2024-04-22T06:53:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:53:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:reconnect 2024-04-22T06:53:58.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:reconnect 2024-04-22T06:53:58.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=e=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:58.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:58 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0039999 2024-04-22T06:53:58.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:53:58.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0 2024-04-22T06:53:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:53:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:reconnect 2024-04-22T06:53:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:reconnect 2024-04-22T06:53:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=e=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:58.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:58 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0039999 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:reconnect 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:reconnect 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:reconnect,2=e=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:58.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:58 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0.0039999 2024-04-22T06:53:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: pgmap v309: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-22T06:53:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:rejoin 2024-04-22T06:53:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:rejoin 2024-04-22T06:53:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=e=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:active 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:active 2024-04-22T06:53:59.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:53:59 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:59.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: pgmap v309: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:rejoin 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:rejoin 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=e=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:active 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:active 2024-04-22T06:53:59.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:53:59 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: pgmap v309: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:rejoin 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:rejoin 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=f=up:rejoin,2=e=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:53:59.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:53:59.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1769517674,v1:172.21.15.190:6835/1769517674] up:active 2024-04-22T06:53:59.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/2674386425,v1:172.21.15.193:6835/2674386425] up:active 2024-04-22T06:53:59.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:53:59 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:00.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:00 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:00.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:00 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:00.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:00 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- 172.21.15.79:0/697919348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6354101aa0 msgr2=0x7f6354105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- 172.21.15.79:0/697919348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6354101aa0 0x7f6354105a90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f6340009900 tx=0x7f634002bc30 comp rx=0 tx=0).stop 2024-04-22T06:54:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- 172.21.15.79:0/697919348 shutdown_connections 2024-04-22T06:54:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- 172.21.15.79:0/697919348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6354106170 0x7f635410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- 172.21.15.79:0/697919348 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6354101aa0 0x7f6354105a90 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- 172.21.15.79:0/697919348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63541010f0 0x7f63541014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- 172.21.15.79:0/697919348 >> 172.21.15.79:0/697919348 conn(0x7f6354077fd0 msgr2=0x7f63540783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- 172.21.15.79:0/697919348 shutdown_connections 2024-04-22T06:54:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- 172.21.15.79:0/697919348 wait complete. 2024-04-22T06:54:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 Processor -- start 2024-04-22T06:54:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- start start 2024-04-22T06:54:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63541010f0 0x7f635419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6354106170 0x7f635419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 0x7f6354198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6354111110 con 0x7f63541985b0 2024-04-22T06:54:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6354110f90 con 0x7f63541010f0 2024-04-22T06:54:00.768 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.732+0000 7f635ac26640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6354111290 con 0x7f6354106170 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 0x7f6354198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 0x7f6354198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53668/0 (socket says 172.21.15.79:53668) 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 -- 172.21.15.79:0/2524489905 learned_addr learned my addr 172.21.15.79:0/2524489905 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f6353fff640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6354106170 0x7f635419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63589c2640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63541010f0 0x7f635419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 -- 172.21.15.79:0/2524489905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6354106170 msgr2=0x7f635419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:00.769 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6354106170 0x7f635419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 -- 172.21.15.79:0/2524489905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63541010f0 msgr2=0x7f635419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63541010f0 0x7f635419e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 -- 172.21.15.79:0/2524489905 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63541990d0 con 0x7f63541985b0 2024-04-22T06:54:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f6353fff640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6354106170 0x7f635419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f63591c3640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 0x7f6354198a60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f634c00cad0 tx=0x7f634c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:00.770 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f6351ffb640 1 -- 172.21.15.79:0/2524489905 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c00ae20 con 0x7f63541985b0 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63541993c0 con 0x7f63541985b0 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f6351ffb640 1 -- 172.21.15.79:0/2524489905 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634c005020 con 0x7f63541985b0 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f6351ffb640 1 -- 172.21.15.79:0/2524489905 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c00f6b0 con 0x7f63541985b0 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.733+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63541a5610 con 0x7f63541985b0 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.735+0000 7f6351ffb640 1 -- 172.21.15.79:0/2524489905 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f634c0051d0 con 0x7f63541985b0 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 Processor -- start 2024-04-22T06:54:00.771 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f6351ffb640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63440a3e40 0x7f63440a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 -- start start 2024-04-22T06:54:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f6353fff640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63440a3e40 0x7f63440a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084d18d0 0x55d3084d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f6351ffb640 1 -- 172.21.15.79:0/2524489905 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9673+0+0 (secure 0 0 0) 0x7f634c0d0e60 con 0x7f63541985b0 2024-04-22T06:54:00.772 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084da0e0 0x55d3084dc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d3084358d0 con 0x55d3084d0b60 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d3083d2e80 con 0x55d3084da0e0 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f635d80e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d308431890 con 0x55d3084d18d0 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f6353fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f6353fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53682/0 (socket says 172.21.15.79:53682) 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.736+0000 7f6353fff640 1 -- 172.21.15.79:0/3404587000 learned_addr learned my addr 172.21.15.79:0/3404587000 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:00.773 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f63591c3640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084da0e0 0x55d3084dc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f63589c2640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084d18d0 0x55d3084d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63440a3e40 0x7f63440a6300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6340005ab0 tx=0x7f6340004550 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 -- 172.21.15.79:0/3404587000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084d18d0 msgr2=0x55d3084d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084d18d0 0x55d3084d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 -- 172.21.15.79:0/3404587000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084da0e0 msgr2=0x55d3084dc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.774 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084da0e0 0x55d3084dc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 -- 172.21.15.79:0/3404587000 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3083f4320 con 0x55d3084d0b60 2024-04-22T06:54:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f63591c3640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084da0e0 0x55d3084dc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f63589c2640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084d18d0 0x55d3084d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f6353fff640 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084d0fe0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6340009900 tx=0x7f6340049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd8b567057b98d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f633effd640 1 -- 172.21.15.79:0/3404587000 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340005300 con 0x55d3084d0b60 2024-04-22T06:54:00.775 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f633effd640 1 -- 172.21.15.79:0/3404587000 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6340051070 con 0x55d3084d0b60 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.737+0000 7f633effd640 1 -- 172.21.15.79:0/3404587000 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340003950 con 0x55d3084d0b60 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- 172.21.15.79:0/3404587000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 msgr2=0x55d3084d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084d0fe0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6340009900 tx=0x7f6340049000 comp rx=0 tx=0).stop 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- 172.21.15.79:0/3404587000 shutdown_connections 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084da0e0 0x55d3084dc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084d0fe0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- 172.21.15.79:0/3404587000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084d18d0 0x55d3084d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.776 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- 172.21.15.79:0/3404587000 >> 172.21.15.79:0/3404587000 conn(0x55d308497c90 msgr2=0x55d3084cb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- 172.21.15.79:0/3404587000 shutdown_connections 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- 172.21.15.79:0/3404587000 wait complete. 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- start start 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 Processor -- start 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d0b60 0x55d3084d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 0x55d3084d6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d3084dcdb0 con 0x55d3084d18d0 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d3083bdeb0 con 0x55d3084d0b60 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.738+0000 7f635d80e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d3084dcf30 con 0x55d3084da0e0 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f635d80e740 10 client.0 _ll_register_callbacks cb 0x55d3083eab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 0x55d3084d6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 0x55d3084d6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53696/0 (socket says 172.21.15.79:53696) 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 -- 172.21.15.79:0/1641457148 learned_addr learned my addr 172.21.15.79:0/1641457148 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f63591c3640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f63589c2640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d0b60 0x55d3084d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 -- 172.21.15.79:0/1641457148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 msgr2=0x55d3084d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 -- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d0b60 msgr2=0x55d3084d9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d0b60 0x55d3084d9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 -- 172.21.15.79:0/1641457148 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d308148210 con 0x55d3084d18d0 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6353fff640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 0x55d3084d6e50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6340009850 tx=0x7f63400053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239b5e12cc0d50ab server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6333fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6333fff640 1 -- 172.21.15.79:0/1641457148 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340059030 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6333fff640 1 -- 172.21.15.79:0/1641457148 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9673+0+0 (secure 0 0 0) 0x7f63400048d0 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6340051070 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f6333fff640 1 -- 172.21.15.79:0/1641457148 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f6340066050 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f635d80e740 10 client.17412 fetch_fsmap learned FSMap version 80 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.739+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x55d308214620 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f6333fff640 1 -- 172.21.15.79:0/1641457148 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6340051070 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f6333fff640 1 -- 172.21.15.79:0/1641457148 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 80) ==== 8338+0+0 (secure 0 0 0) 0x7f6340043450 con 0x55d3084d18d0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f635d80e740 10 client.17412 fetch_fsmap finished waiting for FSMap version 80 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f635d80e740 10 client.17412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f635d80e740 20 client.17412 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f635d80e740 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d3084e9ed0 0x55d3084ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f635d80e740 4 client.17412 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d308449880 con 0x55d3084e9ed0 2024-04-22T06:54:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.740+0000 7f63589c2640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d3084e9ed0 0x55d3084ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f63589c2640 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d3084e9ed0 0x55d3084ec2c0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2e1eeadf2751f0a server_cookie=bd3461c27f2d98fd in_seq=0 out_seq=0 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f6333fff640 10 client.17412 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f6333fff640 1 -- 172.21.15.79:0/1641457148 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d308449880 con 0x55d3084e9ed0 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f6333fff640 10 client.17412 handle_command_reply: tid=0 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 client.17412 shutdown 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 msgr2=0x55d3084d6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 0x55d3084d6e50 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6340009850 tx=0x7f63400053b0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 shutdown_connections 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d3084e9ed0 0x55d3084ec2c0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f6333fff640 0 client.17412 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d18d0 0x55d3084d6e50 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 --2- 172.21.15.79:0/1641457148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d0b60 0x55d3084d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.741+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 >> 172.21.15.79:0/1641457148 conn(0x55d308497c90 msgr2=0x55d3084cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.742+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 shutdown_connections 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.742+0000 7f635d80e740 1 -- 172.21.15.79:0/1641457148 wait complete. 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.742+0000 7f635d80e740 20 client.17412 trim_cache size 0 max 16384 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 Processor -- start 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 -- start start 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084d5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f63541b0de0 con 0x55d3084d0b60 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63541ae6b0 con 0x55d3084d18d0 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f635d80e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63541ac500 con 0x55d3084da0e0 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f63589c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f63589c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53702/0 (socket says 172.21.15.79:53702) 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f63589c2640 1 -- 172.21.15.79:0/2643278879 learned_addr learned my addr 172.21.15.79:0/2643278879 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.756+0000 7f6353fff640 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084d5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f63589c2640 1 -- 172.21.15.79:0/2643278879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 msgr2=0x55d3084d9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f63589c2640 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f63589c2640 1 -- 172.21.15.79:0/2643278879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 msgr2=0x55d3084d5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f63589c2640 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084d5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f63589c2640 1 -- 172.21.15.79:0/2643278879 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3084cc9c0 con 0x55d3084d0b60 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f63589c2640 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e5cb0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6348009870 tx=0x7f6348009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e881f62db6b195d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:00.783 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f633e7fc640 1 -- 172.21.15.79:0/2643278879 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634800f040 con 0x55d3084d0b60 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f633e7fc640 1 -- 172.21.15.79:0/2643278879 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634801f030 con 0x55d3084d0b60 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f633e7fc640 1 -- 172.21.15.79:0/2643278879 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63480143b0 con 0x55d3084d0b60 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 -- 172.21.15.79:0/2643278879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 msgr2=0x55d3084e5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e5cb0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6348009870 tx=0x7f6348009dd0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 -- 172.21.15.79:0/2643278879 shutdown_connections 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084d5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084d9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 --2- 172.21.15.79:0/2643278879 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e5cb0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.757+0000 7f635d80e740 1 -- 172.21.15.79:0/2643278879 >> 172.21.15.79:0/2643278879 conn(0x55d308497c90 msgr2=0x55d3084d55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 -- 172.21.15.79:0/2643278879 shutdown_connections 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 -- 172.21.15.79:0/2643278879 wait complete. 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 -- start start 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 Processor -- start 2024-04-22T06:54:00.784 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084dd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084ddda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f63541acb70 con 0x55d3084d0b60 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63541adfe0 con 0x55d3084d18d0 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63541ad240 con 0x55d3084da0e0 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f635d80e740 10 client.0 _ll_register_callbacks cb 0x7f6340009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f63589c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f63589c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53716/0 (socket says 172.21.15.79:53716) 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.758+0000 7f63589c2640 1 -- 172.21.15.79:0/2326276607 learned_addr learned my addr 172.21.15.79:0/2326276607 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6353fff640 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084dd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f63589c2640 1 -- 172.21.15.79:0/2326276607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 msgr2=0x55d3084ddda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f63589c2640 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084ddda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f63589c2640 1 -- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 msgr2=0x55d3084dd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f63589c2640 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084dd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f63589c2640 1 -- 172.21.15.79:0/2326276607 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3084de5a0 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f63589c2640 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e24f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f6348023a90 tx=0x7f6348023d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9cc1f268908b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6333fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6333fff640 1 -- 172.21.15.79:0/2326276607 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634802a070 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6333fff640 1 -- 172.21.15.79:0/2326276607 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9673+0+0 (secure 0 0 0) 0x7f63480233d0 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f634801f030 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6333fff640 1 -- 172.21.15.79:0/2326276607 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f6348036050 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f635d80e740 10 client.17418 fetch_fsmap learned FSMap version 80 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=4+}) -- 0x55d3085500e0 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6333fff640 1 -- 172.21.15.79:0/2326276607 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634801f030 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f6333fff640 1 -- 172.21.15.79:0/2326276607 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 80) ==== 8338+0+0 (secure 0 0 0) 0x7f6348024680 con 0x55d3084d0b60 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f635d80e740 10 client.17418 fetch_fsmap finished waiting for FSMap version 80 2024-04-22T06:54:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f635d80e740 10 client.17418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.759+0000 7f635d80e740 20 client.17418 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.760+0000 7f635d80e740 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d308550770 0x55d308552b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.760+0000 7f635d80e740 4 client.17418 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.760+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d3083d53d0 con 0x55d308550770 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.760+0000 7f6353fff640 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d308550770 0x55d308552b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.760+0000 7f6333fff640 10 client.17418 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:00.760+0000 7f6353fff640 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d308550770 0x55d308552b90 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cbff7484b095caa server_cookie=dc4a5d86bb5e907a in_seq=0 out_seq=0 2024-04-22T06:54:01.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:01 smithi193 ceph-mon[31815]: pgmap v310: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:54:01.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:01 smithi190 ceph-mon[31421]: pgmap v310: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:54:01.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:01 smithi079 ceph-mon[27946]: pgmap v310: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:54:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f6333fff640 1 -- 172.21.15.79:0/2326276607 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d3083d53d0 con 0x55d308550770 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f6333fff640 10 client.17418 handle_command_reply: tid=0 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 client.17418 shutdown 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 msgr2=0x55d3084e24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e24f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f6348023a90 tx=0x7f6348023d70 comp rx=0 tx=0).stop 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 shutdown_connections 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d308550770 0x55d308552b90 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3084d18d0 0x55d3084dd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3084da0e0 0x55d3084ddda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f6333fff640 0 client.17418 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 --2- 172.21.15.79:0/2326276607 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d3084d0b60 0x55d3084e24f0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 >> 172.21.15.79:0/2326276607 conn(0x55d308497c90 msgr2=0x55d3084d5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 shutdown_connections 2024-04-22T06:54:03.423 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 1 -- 172.21.15.79:0/2326276607 wait complete. 2024-04-22T06:54:03.424 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.424+0000 7f635d80e740 20 client.17418 trim_cache size 0 max 16384 2024-04-22T06:54:03.424 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:54:03.424 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:54:03.424 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:54:03.424 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63440a3e40 msgr2=0x7f63440a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63440a3e40 0x7f63440a6300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6340005ab0 tx=0x7f6340004550 comp rx=0 tx=0).stop 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 msgr2=0x7f6354198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 0x7f6354198a60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f634c00cad0 tx=0x7f634c00cf90 comp rx=0 tx=0).stop 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 shutdown_connections 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63541985b0 0x7f6354198a60 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6354106170 0x7f635419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63440a3e40 0x7f63440a6300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 --2- 172.21.15.79:0/2524489905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63541010f0 0x7f635419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 >> 172.21.15.79:0/2524489905 conn(0x7f6354077fd0 msgr2=0x7f63540699d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:03.425 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 shutdown_connections 2024-04-22T06:54:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:03.425+0000 7f635ac26640 1 -- 172.21.15.79:0/2524489905 wait complete. 2024-04-22T06:54:03.434 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-mon[31815]: pgmap v311: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 36 KiB/s wr, 23 op/s 2024-04-22T06:54:03.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:03 smithi190 ceph-mon[31421]: pgmap v311: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 36 KiB/s wr, 23 op/s 2024-04-22T06:54:03.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:03 smithi079 ceph-mon[27946]: pgmap v311: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 36 KiB/s wr, 23 op/s 2024-04-22T06:54:03.690 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: in thread 7f51869d5700 thread_name:md_log_replay 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5197978d20] 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55d59ac6f34f] 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55d59ac72ae3] 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 4: (MDLog::_replay_thread()+0x617) [0x55d59accd607] 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55d59acd6cf1] 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f519796e1ca] 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 7: clone() 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.437+0000 7f51869d5700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:03.691 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: in thread 7f51869d5700 thread_name:md_log_replay 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5197978d20] 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55d59ac6f34f] 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55d59ac72ae3] 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 4: (MDLog::_replay_thread()+0x617) [0x55d59accd607] 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55d59acd6cf1] 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f519796e1ca] 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 7: clone() 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 0> 2024-04-22T06:54:03.437+0000 7f51869d5700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: in thread 7f51869d5700 thread_name:md_log_replay 2024-04-22T06:54:03.692 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5197978d20] 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55d59ac6f34f] 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55d59ac72ae3] 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 4: (MDLog::_replay_thread()+0x617) [0x55d59accd607] 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55d59acd6cf1] 2024-04-22T06:54:03.693 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f519796e1ca] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 7: clone() 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 0> 2024-04-22T06:54:03.437+0000 7f51869d5700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: in thread 7f51869d5700 thread_name:md_log_replay 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5197978d20] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55d59ac6f34f] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55d59ac72ae3] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 4: (MDLog::_replay_thread()+0x617) [0x55d59accd607] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55d59acd6cf1] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f519796e1ca] 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 7: clone() 2024-04-22T06:54:03.694 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:03.695 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[62951]: 2024-04-22T06:54:03.695 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:03 smithi193 podman[68763]: 2024-04-22 06:54:03.498330802 +0000 UTC m=+0.021224304 container died 98f45dbf9d45b84c73f7b520f13368b1ef1def3fc964449bbdd2bdc4f321682d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:54:04.439 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:04 smithi193 podman[68763]: 2024-04-22 06:54:04.094218337 +0000 UTC m=+0.617111829 container remove 98f45dbf9d45b84c73f7b520f13368b1ef1def3fc964449bbdd2bdc4f321682d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T06:54:04.439 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:04 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-22T06:54:04.939 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:04 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:54:04.939 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:04 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Consumed 34.617s CPU time. 2024-04-22T06:54:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:05 smithi193 ceph-mon[31815]: pgmap v312: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 65 KiB/s wr, 48 op/s 2024-04-22T06:54:05.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:05 smithi193 ceph-mon[31815]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:54:05.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:05 smithi190 ceph-mon[31421]: pgmap v312: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 65 KiB/s wr, 48 op/s 2024-04-22T06:54:05.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:05 smithi190 ceph-mon[31421]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:54:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:05 smithi079 ceph-mon[27946]: pgmap v312: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 65 KiB/s wr, 48 op/s 2024-04-22T06:54:05.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:05 smithi079 ceph-mon[27946]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:54:07.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:07 smithi193 ceph-mon[31815]: pgmap v314: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 54 KiB/s wr, 41 op/s 2024-04-22T06:54:07.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:07 smithi190 ceph-mon[31421]: pgmap v314: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 54 KiB/s wr, 41 op/s 2024-04-22T06:54:07.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:07 smithi079 ceph-mon[27946]: pgmap v314: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 54 KiB/s wr, 41 op/s 2024-04-22T06:54:09.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:09 smithi193 ceph-mon[31815]: pgmap v315: 97 pgs: 97 active+clean; 43 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 KiB/s wr, 40 op/s 2024-04-22T06:54:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:09 smithi190 ceph-mon[31421]: pgmap v315: 97 pgs: 97 active+clean; 43 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 KiB/s wr, 40 op/s 2024-04-22T06:54:09.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:09 smithi079 ceph-mon[27946]: pgmap v315: 97 pgs: 97 active+clean; 43 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 KiB/s wr, 40 op/s 2024-04-22T06:54:11.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:11 smithi193 ceph-mon[31815]: pgmap v316: 97 pgs: 97 active+clean; 40 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 KiB/s wr, 40 op/s 2024-04-22T06:54:11.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:11 smithi190 ceph-mon[31421]: pgmap v316: 97 pgs: 97 active+clean; 40 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 KiB/s wr, 40 op/s 2024-04-22T06:54:11.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:11 smithi079 ceph-mon[27946]: pgmap v316: 97 pgs: 97 active+clean; 40 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 KiB/s wr, 40 op/s 2024-04-22T06:54:13.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:13 smithi193 ceph-mon[31815]: pgmap v317: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-04-22T06:54:13.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:13 smithi190 ceph-mon[31421]: pgmap v317: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-04-22T06:54:13.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:13 smithi079 ceph-mon[27946]: pgmap v317: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 26 op/s 2024-04-22T06:54:14.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:14 smithi079 ceph-mon[27946]: pgmap v318: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:14.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:14 smithi193 ceph-mon[31815]: pgmap v318: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:14.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:14 smithi190 ceph-mon[31421]: pgmap v318: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:15.034 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:14 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:54:15.034 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:14 smithi193 systemd[1]: Stopped Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:15.034 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:14 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Consumed 34.617s CPU time. 2024-04-22T06:54:15.034 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:14 smithi193 systemd[1]: Starting Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:54:15.034 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:14 smithi193 podman[68859]: 2024-04-22 06:54:14.834809743 +0000 UTC m=+0.018420591 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:54:15.034 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 podman[68859]: 2024-04-22 06:54:15.036658335 +0000 UTC m=+0.220269187 container create 55e5a664d73d046f72f38b56fc671407a78d2d40e660c2b395a6075c47ab9e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:54:15.303 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 podman[68859]: 2024-04-22 06:54:15.281293593 +0000 UTC m=+0.464904442 container init 55e5a664d73d046f72f38b56fc671407a78d2d40e660c2b395a6075c47ab9e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T06:54:15.303 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 podman[68859]: 2024-04-22 06:54:15.283784886 +0000 UTC m=+0.467395744 container start 55e5a664d73d046f72f38b56fc671407a78d2d40e660c2b395a6075c47ab9e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-22T06:54:15.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:15 smithi193 ceph-mon[31815]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:54:15.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:15 smithi193 ceph-mon[31815]: daemon mds.c restarted 2024-04-22T06:54:15.689 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 ceph-mds[68878]: -- 172.21.15.193:0/2707978061 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557086747a40 con 0x55708671ac00 2024-04-22T06:54:15.690 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: starting mds.c at 2024-04-22T06:54:15.690 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 bash[68859]: 55e5a664d73d046f72f38b56fc671407a78d2d40e660c2b395a6075c47ab9e16 2024-04-22T06:54:15.690 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:54:15 smithi193 systemd[1]: Started Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:15.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:15 smithi190 ceph-mon[31421]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:54:15.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:15 smithi190 ceph-mon[31421]: daemon mds.c restarted 2024-04-22T06:54:15.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:15 smithi079 ceph-mon[27946]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:54:15.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:15 smithi079 ceph-mon[27946]: daemon mds.c restarted 2024-04-22T06:54:16.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 -- 172.21.15.79:0/1455573370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc1031e0 msgr2=0x7f05dc1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 --2- 172.21.15.79:0/1455573370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc1031e0 0x7f05dc1035c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009900 tx=0x7f05cc02bd60 comp rx=0 tx=0).stop 2024-04-22T06:54:16.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 -- 172.21.15.79:0/1455573370 shutdown_connections 2024-04-22T06:54:16.522 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 --2- 172.21.15.79:0/1455573370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05dc103b00 0x7f05dc10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 --2- 172.21.15.79:0/1455573370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 0x7f05dc10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 --2- 172.21.15.79:0/1455573370 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc1031e0 0x7f05dc1035c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.523 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 -- 172.21.15.79:0/1455573370 >> 172.21.15.79:0/1455573370 conn(0x7f05dc077fd0 msgr2=0x7f05dc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 -- 172.21.15.79:0/1455573370 shutdown_connections 2024-04-22T06:54:16.524 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 -- 172.21.15.79:0/1455573370 wait complete. 2024-04-22T06:54:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.523+0000 7f05e2893640 1 Processor -- start 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 -- start start 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05dc1031e0 0x7f05dc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 0x7f05dc198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 0x7f05dc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc112190 con 0x7f05dc103b00 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc112010 con 0x7f05dc1031e0 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e2893640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc112310 con 0x7f05dc10c520 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 0x7f05dc198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 0x7f05dc198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55952/0 (socket says 172.21.15.79:55952) 2024-04-22T06:54:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 -- 172.21.15.79:0/3519480469 learned_addr learned my addr 172.21.15.79:0/3519480469 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05dbfff640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05dc1031e0 0x7f05dc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e0e30640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 0x7f05dc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 -- 172.21.15.79:0/3519480469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 msgr2=0x7f05dc19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 0x7f05dc19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 -- 172.21.15.79:0/3519480469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05dc1031e0 msgr2=0x7f05dc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05dc1031e0 0x7f05dc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05db7fe640 1 -- 172.21.15.79:0/3519480469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc19d6d0 con 0x7f05dc103b00 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.524+0000 7f05e0e30640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 0x7f05dc19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.525+0000 7f05db7fe640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 0x7f05dc198c70 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f05d000d950 tx=0x7f05d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.525+0000 7f05d97fa640 1 -- 172.21.15.79:0/3519480469 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0017070 con 0x7f05dc103b00 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.525+0000 7f05d97fa640 1 -- 172.21.15.79:0/3519480469 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05d0009c30 con 0x7f05dc103b00 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.525+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05dc19d9c0 con 0x7f05dc103b00 2024-04-22T06:54:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.525+0000 7f05d97fa640 1 -- 172.21.15.79:0/3519480469 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d000f620 con 0x7f05dc103b00 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.525+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05dc1a5250 con 0x7f05dc103b00 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.527+0000 7f05d97fa640 1 -- 172.21.15.79:0/3519480469 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f05d0010460 con 0x7f05dc103b00 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.527+0000 7f05e5565740 1 Processor -- start 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.527+0000 7f05e5565740 1 -- start start 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e5565740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e5565740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05d97fa640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f05c40a3dc0 0x7f05c40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e5565740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935c04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e5565740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e5935198d0 con 0x55e5935be0e0 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e5565740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e5934b6e80 con 0x55e5935b4b60 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e5565740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e593515890 con 0x55e5935b58d0 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935c04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05db7fe640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f05c40a3dc0 0x7f05c40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935c04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55954/0 (socket says 172.21.15.79:55954) 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 -- 172.21.15.79:0/3988775472 learned_addr learned my addr 172.21.15.79:0/3988775472 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05dbfff640 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05d97fa640 1 -- 172.21.15.79:0/3519480469 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7f05d00d08e0 con 0x7f05dc103b00 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 -- 172.21.15.79:0/3988775472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 msgr2=0x55e5935b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 -- 172.21.15.79:0/3988775472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 msgr2=0x55e5935b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 -- 172.21.15.79:0/3988775472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5934d8320 con 0x55e5935be0e0 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05db7fe640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f05c40a3dc0 0x7f05c40a6280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05d00ce1b0 tx=0x7f05d006adb0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05e0e30640 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935c04d0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f05d4013b30 tx=0x7f05d400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde6285c2b31d038 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05c27fc640 1 -- 172.21.15.79:0/3988775472 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4013310 con 0x55e5935be0e0 2024-04-22T06:54:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05c27fc640 1 -- 172.21.15.79:0/3988775472 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05d4003970 con 0x55e5935be0e0 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.528+0000 7f05c27fc640 1 -- 172.21.15.79:0/3988775472 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40042f0 con 0x55e5935be0e0 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 -- 172.21.15.79:0/3988775472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 msgr2=0x55e5935c04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935c04d0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f05d4013b30 tx=0x7f05d400a660 comp rx=0 tx=0).stop 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 -- 172.21.15.79:0/3988775472 shutdown_connections 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935c04d0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 --2- 172.21.15.79:0/3988775472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 -- 172.21.15.79:0/3988775472 >> 172.21.15.79:0/3988775472 conn(0x55e59357bc90 msgr2=0x55e5935af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 -- 172.21.15.79:0/3988775472 shutdown_connections 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 -- 172.21.15.79:0/3988775472 wait complete. 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.529+0000 7f05e5565740 1 -- start start 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 Processor -- start 2024-04-22T06:54:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935bdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935bae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935bb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55e5935c0e10 con 0x55e5935be0e0 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55e5934a1eb0 con 0x55e5935b4b60 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5935c0f90 con 0x55e5935b58d0 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e5565740 10 client.0 _ll_register_callbacks cb 0x55e5934ceb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935bb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935bb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55964/0 (socket says 172.21.15.79:55964) 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 -- 172.21.15.79:0/1315813573 learned_addr learned my addr 172.21.15.79:0/1315813573 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935bdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05dbfff640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935bae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 -- 172.21.15.79:0/1315813573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 msgr2=0x55e5935bae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935bae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 -- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 msgr2=0x55e5935bdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935bdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.530+0000 7f05e0e30640 1 -- 172.21.15.79:0/1315813573 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e59322c210 con 0x55e5935be0e0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e0e30640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935bb440 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f05d4002230 tx=0x7f05d4005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9790a37fd37c2a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05bb7fe640 1 -- 172.21.15.79:0/1315813573 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4004490 con 0x55e5935be0e0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05bb7fe640 1 -- 172.21.15.79:0/1315813573 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7f05d4004630 con 0x55e5935be0e0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05d4003970 con 0x55e5935be0e0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05bb7fe640 1 -- 172.21.15.79:0/1315813573 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f05d400eae0 con 0x55e5935be0e0 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 10 client.17433 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55e5932f8620 con 0x55e5935be0e0 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05bb7fe640 1 -- 172.21.15.79:0/1315813573 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05d4003970 con 0x55e5935be0e0 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05bb7fe640 1 -- 172.21.15.79:0/1315813573 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f05d4005210 con 0x55e5935be0e0 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 10 client.17433 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 10 client.17433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 20 client.17433 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5935ce050 0x55e5935d0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 4 client.17433 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.531+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e59352d880 con 0x55e5935ce050 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.532+0000 7f05dbfff640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5935ce050 0x55e5935d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.532+0000 7f05dbfff640 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5935ce050 0x55e5935d0440 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea8b991fbbb92453 server_cookie=376b3d9fe5896c65 in_seq=0 out_seq=0 2024-04-22T06:54:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.532+0000 7f05bb7fe640 10 client.17433 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05bb7fe640 1 -- 172.21.15.79:0/1315813573 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e59352d880 con 0x55e5935ce050 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05bb7fe640 10 client.17433 handle_command_reply: tid=0 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 client.17433 shutdown 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 msgr2=0x55e5935bb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935bb440 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f05d4002230 tx=0x7f05d4005bf0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 shutdown_connections 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e5935ce050 0x55e5935d0440 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935bb440 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05bb7fe640 0 client.17433 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935bae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 --2- 172.21.15.79:0/1315813573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935bdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 >> 172.21.15.79:0/1315813573 conn(0x55e59357bc90 msgr2=0x55e5935b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 shutdown_connections 2024-04-22T06:54:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 1 -- 172.21.15.79:0/1315813573 wait complete. 2024-04-22T06:54:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.533+0000 7f05e5565740 20 client.17433 trim_cache size 0 max 16384 2024-04-22T06:54:16.553 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 Processor -- start 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 -- start start 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 0x55e5935ba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b58d0 0x55e5935b1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935be0e0 0x55e5935bd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc1b0e70 con 0x55e5935b4b60 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc1ae740 con 0x55e5935b58d0 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e5565740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc1ac590 con 0x55e5935be0e0 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 0x55e5935ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 0x55e5935ba710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55972/0 (socket says 172.21.15.79:55972) 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 -- 172.21.15.79:0/2189697928 learned_addr learned my addr 172.21.15.79:0/2189697928 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 -- 172.21.15.79:0/2189697928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935be0e0 msgr2=0x55e5935bd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.554 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935be0e0 0x55e5935bd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 -- 172.21.15.79:0/2189697928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b58d0 msgr2=0x55e5935b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b58d0 0x55e5935b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 -- 172.21.15.79:0/2189697928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5935b3e30 con 0x55e5935b4b60 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.548+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 0x55e5935ba710 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f05d40134b0 tx=0x7f05d4013700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec827b50a2d59b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05c1ffb640 1 -- 172.21.15.79:0/2189697928 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4023070 con 0x55e5935b4b60 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05c1ffb640 1 -- 172.21.15.79:0/2189697928 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05d4003970 con 0x55e5935b4b60 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05c1ffb640 1 -- 172.21.15.79:0/2189697928 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d400ec80 con 0x55e5935b4b60 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 -- 172.21.15.79:0/2189697928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 msgr2=0x55e5935ba710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 0x55e5935ba710 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f05d40134b0 tx=0x7f05d4013700 comp rx=0 tx=0).stop 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 -- 172.21.15.79:0/2189697928 shutdown_connections 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b58d0 0x55e5935b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935be0e0 0x55e5935bd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 --2- 172.21.15.79:0/2189697928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935b4b60 0x55e5935ba710 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 -- 172.21.15.79:0/2189697928 >> 172.21.15.79:0/2189697928 conn(0x55e59357bc90 msgr2=0x55e5935b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:16.555 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 -- 172.21.15.79:0/2189697928 shutdown_connections 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 -- 172.21.15.79:0/2189697928 wait complete. 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.549+0000 7f05e5565740 1 -- start start 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 Processor -- start 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935c6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc1acc00 con 0x55e5935be0e0 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc1ae070 con 0x55e5935b4b60 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc1ad2d0 con 0x55e5935b58d0 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e5565740 10 client.0 _ll_register_callbacks cb 0x7f05cc005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935b80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55982/0 (socket says 172.21.15.79:55982) 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 -- 172.21.15.79:0/2329113794 learned_addr learned my addr 172.21.15.79:0/2329113794 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:16.556 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05dbfff640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 -- 172.21.15.79:0/2329113794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 msgr2=0x55e5935c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 -- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 msgr2=0x55e5935c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.550+0000 7f05e0e30640 1 -- 172.21.15.79:0/2329113794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5935b88b0 con 0x55e5935be0e0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e0e30640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935b80b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f05d40138a0 tx=0x7f05d4005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7216136746514fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05bb7fe640 1 -- 172.21.15.79:0/2329113794 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4023040 con 0x55e5935be0e0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05bb7fe640 1 -- 172.21.15.79:0/2329113794 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7f05d4002030 con 0x55e5935be0e0 2024-04-22T06:54:16.557 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05d4003970 con 0x55e5935be0e0 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05bb7fe640 1 -- 172.21.15.79:0/2329113794 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f05d400a880 con 0x55e5935be0e0 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 10 client.17439 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55e5935bef90 con 0x55e5935be0e0 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05bb7fe640 1 -- 172.21.15.79:0/2329113794 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05d4003970 con 0x55e5935be0e0 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05bb7fe640 1 -- 172.21.15.79:0/2329113794 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f05d4011760 con 0x55e5935be0e0 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 10 client.17439 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 10 client.17439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 20 client.17439 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e593634310 0x55e593636700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 4 client.17439 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.551+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5934b93d0 con 0x55e593634310 2024-04-22T06:54:16.558 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.552+0000 7f05dbfff640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e593634310 0x55e593636700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:16.559 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.552+0000 7f05dbfff640 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e593634310 0x55e593636700 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e5ef831e09fc3a server_cookie=5f465dd70c5dcf1f in_seq=0 out_seq=0 2024-04-22T06:54:16.559 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.552+0000 7f05bb7fe640 10 client.17439 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05bb7fe640 1 -- 172.21.15.79:0/2329113794 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e5934b93d0 con 0x55e593634310 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05bb7fe640 10 client.17439 handle_command_reply: tid=0 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 client.17439 shutdown 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 msgr2=0x55e5935b80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935b80b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f05d40138a0 tx=0x7f05d4005d90 comp rx=0 tx=0).stop 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 shutdown_connections 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55e593634310 0x55e593636700 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55e5935be0e0 0x55e5935b80b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.560 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5935b58d0 0x55e5935c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 --2- 172.21.15.79:0/2329113794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e5935b4b60 0x55e5935c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05bb7fe640 0 client.17439 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.559+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 >> 172.21.15.79:0/2329113794 conn(0x55e59357bc90 msgr2=0x55e5935ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 shutdown_connections 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e5565740 1 -- 172.21.15.79:0/2329113794 wait complete. 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e5565740 20 client.17439 trim_cache size 0 max 16384 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:54:16.561 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:54:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f05c40a3dc0 msgr2=0x7f05c40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f05c40a3dc0 0x7f05c40a6280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05d00ce1b0 tx=0x7f05d006adb0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.562 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 msgr2=0x7f05dc198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 0x7f05dc198c70 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f05d000d950 tx=0x7f05d000de10 comp rx=0 tx=0).stop 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 shutdown_connections 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05dc10c520 0x7f05dc19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05dc103b00 0x7f05dc198c70 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f05c40a3dc0 0x7f05c40a6280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 --2- 172.21.15.79:0/3519480469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f05dc1031e0 0x7f05dc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 >> 172.21.15.79:0/3519480469 conn(0x7f05dc077fd0 msgr2=0x7f05dc0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 shutdown_connections 2024-04-22T06:54:16.563 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:16.560+0000 7f05e2893640 1 -- 172.21.15.79:0/3519480469 wait complete. 2024-04-22T06:54:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:16 smithi193 ceph-mon[31815]: pgmap v320: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:16 smithi193 ceph-mon[31815]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:54:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:16 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6836/4272953642,v1:172.21.15.193:6837/4272953642] up:boot 2024-04-22T06:54:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:16 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:54:16.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:16 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:54:16.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:16 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:16.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:16 smithi190 ceph-mon[31421]: pgmap v320: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:16 smithi190 ceph-mon[31421]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:54:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:16 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6836/4272953642,v1:172.21.15.193:6837/4272953642] up:boot 2024-04-22T06:54:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:16 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:54:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:16 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:54:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:16 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:16 smithi079 ceph-mon[27946]: pgmap v320: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:16 smithi079 ceph-mon[27946]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:54:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:16 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6836/4272953642,v1:172.21.15.193:6837/4272953642] up:boot 2024-04-22T06:54:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:16 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:54:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:16 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:54:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:16 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=f=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:18.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:18 smithi193 ceph-mon[31815]: pgmap v322: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 127 B/s wr, 5 op/s 2024-04-22T06:54:18.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:18 smithi190 ceph-mon[31421]: pgmap v322: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 127 B/s wr, 5 op/s 2024-04-22T06:54:18.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:18 smithi079 ceph-mon[27946]: pgmap v322: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 127 B/s wr, 5 op/s 2024-04-22T06:54:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:20 smithi193 ceph-mon[31815]: pgmap v323: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T06:54:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:20 smithi190 ceph-mon[31421]: pgmap v323: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T06:54:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:20 smithi079 ceph-mon[27946]: pgmap v323: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-22T06:54:22.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:22 smithi193 ceph-mon[31815]: pgmap v324: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:54:22.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:22 smithi190 ceph-mon[31421]: pgmap v324: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:54:22.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:22 smithi079 ceph-mon[27946]: pgmap v324: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:54:24.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:24 smithi193 ceph-mon[31815]: pgmap v325: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 460 B/s wr, 7 op/s 2024-04-22T06:54:24.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:24 smithi190 ceph-mon[31421]: pgmap v325: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 460 B/s wr, 7 op/s 2024-04-22T06:54:24.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:24 smithi079 ceph-mon[27946]: pgmap v325: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 460 B/s wr, 7 op/s 2024-04-22T06:54:26.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:26 smithi193 ceph-mon[31815]: pgmap v326: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:54:26.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:26 smithi190 ceph-mon[31421]: pgmap v326: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:54:26.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:26 smithi079 ceph-mon[27946]: pgmap v326: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:54:28.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:28 smithi193 ceph-mon[31815]: pgmap v327: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:54:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:28 smithi190 ceph-mon[31421]: pgmap v327: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:54:28.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:28 smithi079 ceph-mon[27946]: pgmap v327: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 -- 172.21.15.79:0/1079379901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb781016a0 msgr2=0x7fcb7810e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 --2- 172.21.15.79:0/1079379901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb781016a0 0x7fcb7810e3f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009900 tx=0x7fcb6802bc30 comp rx=0 tx=0).stop 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 -- 172.21.15.79:0/1079379901 shutdown_connections 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 --2- 172.21.15.79:0/1079379901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb7810e930 0x7fcb78110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 --2- 172.21.15.79:0/1079379901 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb781016a0 0x7fcb7810e3f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 --2- 172.21.15.79:0/1079379901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb78100d80 0x7fcb78101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 -- 172.21.15.79:0/1079379901 >> 172.21.15.79:0/1079379901 conn(0x7fcb78077fd0 msgr2=0x7fcb780783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 -- 172.21.15.79:0/1079379901 shutdown_connections 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 -- 172.21.15.79:0/1079379901 wait complete. 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.656+0000 7fcb8054b640 1 Processor -- start 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 -- start start 2024-04-22T06:54:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 0x7fcb781a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb781016a0 0x7fcb781a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb7810e930 0x7fcb7819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781143c0 con 0x7fcb78100d80 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78114240 con 0x7fcb781016a0 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb8054b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78114540 con 0x7fcb7810e930 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 0x7fcb781a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 0x7fcb781a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42476/0 (socket says 172.21.15.79:42476) 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 -- 172.21.15.79:0/2909276465 learned_addr learned my addr 172.21.15.79:0/2909276465 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb7810e930 0x7fcb7819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb781016a0 0x7fcb781a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 -- 172.21.15.79:0/2909276465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb7810e930 msgr2=0x7fcb7819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb7810e930 0x7fcb7819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 -- 172.21.15.79:0/2909276465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb781016a0 msgr2=0x7fcb781a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb781016a0 0x7fcb781a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 -- 172.21.15.79:0/2909276465 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7819af70 con 0x7fcb78100d80 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb781016a0 0x7fcb781a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 0x7fcb781a0630 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400eab0 tx=0x7fcb7400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.657+0000 7fcb6f7fe640 1 -- 172.21.15.79:0/2909276465 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7400ce40 con 0x7fcb78100d80 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.658+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7819b260 con 0x7fcb78100d80 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.658+0000 7fcb6f7fe640 1 -- 172.21.15.79:0/2909276465 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb74011920 con 0x7fcb78100d80 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.658+0000 7fcb6f7fe640 1 -- 172.21.15.79:0/2909276465 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74010a30 con 0x7fcb78100d80 2024-04-22T06:54:29.659 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.658+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb781a7480 con 0x7fcb78100d80 2024-04-22T06:54:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.660+0000 7fcb6f7fe640 1 -- 172.21.15.79:0/2909276465 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcb74010bd0 con 0x7fcb78100d80 2024-04-22T06:54:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.660+0000 7fcb83214740 1 Processor -- start 2024-04-22T06:54:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 -- start start 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb6f7fe640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcb640a3dc0 0x7fcb640a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b38d0 0x561f320b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 0x561f320b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320bc0e0 0x561f320be4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x561f320178d0 con 0x561f320b2b60 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561f31fb4e80 con 0x561f320b38d0 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb83214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561f32013890 con 0x561f320bc0e0 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb7dae6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 0x561f320b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcb640a3dc0 0x7fcb640a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb6f7fe640 1 -- 172.21.15.79:0/2909276465 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fcb740d12b0 con 0x7fcb78100d80 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb7dae6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 0x561f320b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42478/0 (socket says 172.21.15.79:42478) 2024-04-22T06:54:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb7dae6640 1 -- 172.21.15.79:0/1071867811 learned_addr learned my addr 172.21.15.79:0/1071867811 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b38d0 0x561f320b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.661+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320bc0e0 0x561f320be4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7dae6640 1 -- 172.21.15.79:0/1071867811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320bc0e0 msgr2=0x561f320be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320bc0e0 0x561f320be4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7dae6640 1 -- 172.21.15.79:0/1071867811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b38d0 msgr2=0x561f320b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b38d0 0x561f320b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7dae6640 1 -- 172.21.15.79:0/1071867811 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f31fd6320 con 0x561f320b2b60 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcb640a3dc0 0x7fcb640a6280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcb78102f60 tx=0x7fcb7000bb00 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 0x561f320b2fe0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcb68006010 tx=0x7fcb6802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26b97634194c388 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb6cbfe640 1 -- 172.21.15.79:0/1071867811 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68043020 con 0x561f320b2b60 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb6cbfe640 1 -- 172.21.15.79:0/1071867811 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6802c7e0 con 0x561f320b2b60 2024-04-22T06:54:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.662+0000 7fcb6cbfe640 1 -- 172.21.15.79:0/1071867811 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb680054e0 con 0x561f320b2b60 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 -- 172.21.15.79:0/1071867811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 msgr2=0x561f320b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 0x561f320b2fe0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcb68006010 tx=0x7fcb6802c650 comp rx=0 tx=0).stop 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 -- 172.21.15.79:0/1071867811 shutdown_connections 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320bc0e0 0x561f320be4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320b2b60 0x561f320b2fe0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 --2- 172.21.15.79:0/1071867811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b38d0 0x561f320b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 -- 172.21.15.79:0/1071867811 >> 172.21.15.79:0/1071867811 conn(0x561f32079c90 msgr2=0x561f320ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 -- 172.21.15.79:0/1071867811 shutdown_connections 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 -- 172.21.15.79:0/1071867811 wait complete. 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 -- start start 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 Processor -- start 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.663+0000 7fcb83214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320bba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x561f320bedb0 con 0x561f320bc0e0 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x561f31f9feb0 con 0x561f320b2b60 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561f320bef30 con 0x561f320b38d0 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 10 client.0 _ll_register_callbacks cb 0x561f31fccb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320b93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42486/0 (socket says 172.21.15.79:42486) 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2016911096 learned_addr learned my addr 172.21.15.79:0/2016911096 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2016911096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 msgr2=0x561f320b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 msgr2=0x561f320bba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320bba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2016911096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f31d2a210 con 0x561f320bc0e0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320b93b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcb70003cc0 tx=0x7fcb70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f60adf4eb1e358 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2016911096 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70027400 con 0x561f320bc0e0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2016911096 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fcb700275a0 con 0x561f320bc0e0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb6802c7e0 con 0x561f320bc0e0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.664+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2016911096 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fcb70030050 con 0x561f320bc0e0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 10 client.17451 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x561f31df6620 con 0x561f320bc0e0 2024-04-22T06:54:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2016911096 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6802c7e0 con 0x561f320bc0e0 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2016911096 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fcb7000f450 con 0x561f320bc0e0 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 10 client.17451 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 10 client.17451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 20 client.17451 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f320cc090 0x561f320ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 4 client.17451 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f3202b880 con 0x561f320cc090 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.665+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f320cc090 0x561f320ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb61ffb640 10 client.17451 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f320cc090 0x561f320ce4c0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bc450ca3c17baa4 server_cookie=de28c3f59412fb7b in_seq=0 out_seq=0 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2016911096 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f3202b880 con 0x561f320cc090 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb61ffb640 10 client.17451 handle_command_reply: tid=0 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 client.17451 shutdown 2024-04-22T06:54:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 msgr2=0x561f320b93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320b93b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcb70003cc0 tx=0x7fcb70004730 comp rx=0 tx=0).stop 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 shutdown_connections 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f320cc090 0x561f320ce4c0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320b93b0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 --2- 172.21.15.79:0/2016911096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb61ffb640 0 client.17451 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.666+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 >> 172.21.15.79:0/2016911096 conn(0x561f32079c90 msgr2=0x561f320af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.667+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 shutdown_connections 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.667+0000 7fcb83214740 1 -- 172.21.15.79:0/2016911096 wait complete. 2024-04-22T06:54:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.667+0000 7fcb83214740 20 client.17451 trim_cache size 0 max 16384 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 Processor -- start 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 -- start start 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320bba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781b3060 con 0x561f320bc0e0 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781b0930 con 0x561f320b2b60 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.681+0000 7fcb83214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781ae780 con 0x561f320b38d0 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320bba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42488/0 (socket says 172.21.15.79:42488) 2024-04-22T06:54:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2143587382 learned_addr learned my addr 172.21.15.79:0/2143587382 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2143587382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 msgr2=0x561f320bb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320bb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2143587382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 msgr2=0x561f320c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2143587382 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f320b1e30 con 0x561f320bc0e0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320bba80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000f7f0 tx=0x7fcb70003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8d6500479246c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb63fff640 1 -- 172.21.15.79:0/2143587382 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70003a80 con 0x561f320bc0e0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb63fff640 1 -- 172.21.15.79:0/2143587382 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb7000f3c0 con 0x561f320bc0e0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.682+0000 7fcb63fff640 1 -- 172.21.15.79:0/2143587382 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70027b80 con 0x561f320bc0e0 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- 172.21.15.79:0/2143587382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 msgr2=0x561f320bba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320bba80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000f7f0 tx=0x7fcb70003f00 comp rx=0 tx=0).stop 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- 172.21.15.79:0/2143587382 shutdown_connections 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320bba80 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- 172.21.15.79:0/2143587382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- 172.21.15.79:0/2143587382 >> 172.21.15.79:0/2143587382 conn(0x561f32079c90 msgr2=0x561f320ad380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- 172.21.15.79:0/2143587382 shutdown_connections 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- 172.21.15.79:0/2143587382 wait complete. 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- start start 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 Processor -- start 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320c4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320c1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781aedf0 con 0x561f320bc0e0 2024-04-22T06:54:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781b0260 con 0x561f320b2b60 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781af4c0 con 0x561f320b38d0 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb83214740 10 client.0 _ll_register_callbacks cb 0x7fcb70005c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb7eae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320c1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb7eae8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320c1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42496/0 (socket says 172.21.15.79:42496) 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.683+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2496453613 learned_addr learned my addr 172.21.15.79:0/2496453613 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2496453613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 msgr2=0x561f320c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 msgr2=0x561f320c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7eae8640 1 -- 172.21.15.79:0/2496453613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f320c19c0 con 0x561f320bc0e0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7dae6640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb7eae8640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320c1190 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000f910 tx=0x7fcb700277c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237e61a9ee829f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2496453613 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70003a80 con 0x561f320bc0e0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2496453613 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fcb700278c0 con 0x561f320bc0e0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb7000f3c0 con 0x561f320bc0e0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2496453613 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fcb7004a050 con 0x561f320bc0e0 2024-04-22T06:54:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb83214740 10 client.17457 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x561f321330f0 con 0x561f320bc0e0 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2496453613 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb7000f3c0 con 0x561f320bc0e0 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.684+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2496453613 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fcb70033070 con 0x561f320bc0e0 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb83214740 10 client.17457 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb83214740 10 client.17457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb83214740 20 client.17457 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb83214740 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f32133740 0x561f32135b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb83214740 4 client.17457 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f31fb73d0 con 0x561f32133740 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f32133740 0x561f32135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb7e2e7640 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f32133740 0x561f32135b30 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e52b6787748b8ad server_cookie=148521078e62b95e in_seq=0 out_seq=0 2024-04-22T06:54:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.685+0000 7fcb61ffb640 10 client.17457 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb61ffb640 1 -- 172.21.15.79:0/2496453613 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f31fb73d0 con 0x561f32133740 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb61ffb640 10 client.17457 handle_command_reply: tid=0 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 client.17457 shutdown 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 msgr2=0x561f320c1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320c1190 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000f910 tx=0x7fcb700277c0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 shutdown_connections 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x561f32133740 0x561f32135b30 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x561f320bc0e0 0x561f320c1190 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f320b38d0 0x561f320c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 --2- 172.21.15.79:0/2496453613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561f320b2b60 0x561f320c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 >> 172.21.15.79:0/2496453613 conn(0x561f32079c90 msgr2=0x561f320ac040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb61ffb640 0 client.17457 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 shutdown_connections 2024-04-22T06:54:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 1 -- 172.21.15.79:0/2496453613 wait complete. 2024-04-22T06:54:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb83214740 20 client.17457 trim_cache size 0 max 16384 2024-04-22T06:54:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:54:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:54:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:54:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcb640a3dc0 msgr2=0x7fcb640a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb8054b640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcb640a3dc0 0x7fcb640a6280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcb78102f60 tx=0x7fcb7000bb00 comp rx=0 tx=0).stop 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 msgr2=0x7fcb781a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.690+0000 7fcb8054b640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 0x7fcb781a0630 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400eab0 tx=0x7fcb7400ef70 comp rx=0 tx=0).stop 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 shutdown_connections 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fcb640a3dc0 0x7fcb640a6280 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb781016a0 0x7fcb781a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb7810e930 0x7fcb7819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 --2- 172.21.15.79:0/2909276465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcb78100d80 0x7fcb781a0630 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 >> 172.21.15.79:0/2909276465 conn(0x7fcb78077fd0 msgr2=0x7fcb7810ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:29.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 shutdown_connections 2024-04-22T06:54:29.694 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:29.691+0000 7fcb8054b640 1 -- 172.21.15.79:0/2909276465 wait complete. 2024-04-22T06:54:30.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:30 smithi193 ceph-mon[31815]: pgmap v328: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:54:30.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:30 smithi190 ceph-mon[31421]: pgmap v328: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:54:30.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:30 smithi079 ceph-mon[27946]: pgmap v328: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:54:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:32 smithi193 ceph-mon[31815]: pgmap v329: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:54:32.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:32 smithi190 ceph-mon[31421]: pgmap v329: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:54:32.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:32 smithi079 ceph-mon[27946]: pgmap v329: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:54:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:34 smithi193 ceph-mon[31815]: pgmap v330: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:34.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:34 smithi190 ceph-mon[31421]: pgmap v330: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:34.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:34 smithi079 ceph-mon[27946]: pgmap v330: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:36 smithi193 ceph-mon[31815]: pgmap v331: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:36.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:36 smithi190 ceph-mon[31421]: pgmap v331: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:36.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:36 smithi079 ceph-mon[27946]: pgmap v331: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:38 smithi193 ceph-mon[31815]: pgmap v332: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:38.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:38 smithi190 ceph-mon[31421]: pgmap v332: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:38 smithi079 ceph-mon[27946]: pgmap v332: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:40.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:40 smithi193 ceph-mon[31815]: pgmap v333: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:40.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:40 smithi190 ceph-mon[31421]: pgmap v333: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:40 smithi079 ceph-mon[27946]: pgmap v333: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:42.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:42 smithi190 ceph-mon[31421]: pgmap v334: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:42 smithi079 ceph-mon[27946]: pgmap v334: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 -- 172.21.15.79:0/3479924525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c106170 msgr2=0x7fa37c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3479924525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c106170 0x7fa37c10dd40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fa374009ca0 tx=0x7fa37402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 -- 172.21.15.79:0/3479924525 shutdown_connections 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3479924525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c106170 0x7fa37c10dd40 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3479924525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c101aa0 0x7fa37c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3479924525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1010f0 0x7fa37c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.786+0000 7fa382cd2640 1 -- 172.21.15.79:0/3479924525 >> 172.21.15.79:0/3479924525 conn(0x7fa37c077fd0 msgr2=0x7fa37c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 -- 172.21.15.79:0/3479924525 shutdown_connections 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 -- 172.21.15.79:0/3479924525 wait complete. 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 Processor -- start 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 -- start start 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 0x7fa37c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c106170 0x7fa37c103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1035b0 0x7fa37c1a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1110e0 con 0x7fa37c1010f0 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c110f60 con 0x7fa37c1035b0 2024-04-22T06:54:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa382cd2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c111260 con 0x7fa37c106170 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 0x7fa37c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 0x7fa37c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57672/0 (socket says 172.21.15.79:57672) 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.787+0000 7fa380a6e640 1 -- 172.21.15.79:0/3126814050 learned_addr learned my addr 172.21.15.79:0/3126814050 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa37bfff640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c106170 0x7fa37c103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa380a6e640 1 -- 172.21.15.79:0/3126814050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c106170 msgr2=0x7fa37c103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa380a6e640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c106170 0x7fa37c103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa38126f640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1035b0 0x7fa37c1a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa380a6e640 1 -- 172.21.15.79:0/3126814050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1035b0 msgr2=0x7fa37c1a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa380a6e640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1035b0 0x7fa37c1a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa380a6e640 1 -- 172.21.15.79:0/3126814050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37c1a6f30 con 0x7fa37c1010f0 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa38126f640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1035b0 0x7fa37c1a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa37bfff640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c106170 0x7fa37c103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa380a6e640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 0x7fa37c102b30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa37000d950 tx=0x7fa37000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa379ffb640 1 -- 172.21.15.79:0/3126814050 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370017030 con 0x7fa37c1010f0 2024-04-22T06:54:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa379ffb640 1 -- 172.21.15.79:0/3126814050 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa370004020 con 0x7fa37c1010f0 2024-04-22T06:54:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa379ffb640 1 -- 172.21.15.79:0/3126814050 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3700047a0 con 0x7fa37c1010f0 2024-04-22T06:54:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa37c1a7220 con 0x7fa37c1010f0 2024-04-22T06:54:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.788+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa37c1a7770 con 0x7fa37c1010f0 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.790+0000 7fa379ffb640 1 -- 172.21.15.79:0/3126814050 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa370009a30 con 0x7fa37c1010f0 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 Processor -- start 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa379ffb640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa36c0a3dc0 0x7fa36c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 -- start start 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38126f640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa36c0a3dc0 0x7fa36c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa379ffb640 1 -- 172.21.15.79:0/3126814050 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fa3700d0770 con 0x7fa37c1010f0 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c689620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c689fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c6954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d60c5ee8d0 con 0x55d60c689b60 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d60c58be80 con 0x55d60c6930e0 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa38599e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d60c5ea890 con 0x55d60c68a8d0 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa37bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c689fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa37bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c689fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57680/0 (socket says 172.21.15.79:57680) 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.791+0000 7fa37bfff640 1 -- 172.21.15.79:0/1611164356 learned_addr learned my addr 172.21.15.79:0/1611164356 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa380a6e640 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c689620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa38126f640 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c6954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa37bfff640 1 -- 172.21.15.79:0/1611164356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 msgr2=0x55d60c689620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa37bfff640 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c689620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa37bfff640 1 -- 172.21.15.79:0/1611164356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 msgr2=0x55d60c6954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa37bfff640 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c6954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa37bfff640 1 -- 172.21.15.79:0/1611164356 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d60c5ad320 con 0x55d60c689b60 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa38126f640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa36c0a3dc0 0x7fa36c0a6280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa374006010 tx=0x7fa374005d80 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa37bfff640 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c689fe0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa36800bc80 tx=0x7fa36800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15653b34fcb7d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa366ffd640 1 -- 172.21.15.79:0/1611164356 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368015070 con 0x55d60c689b60 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa366ffd640 1 -- 172.21.15.79:0/1611164356 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3680054d0 con 0x55d60c689b60 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.792+0000 7fa366ffd640 1 -- 172.21.15.79:0/1611164356 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368011440 con 0x55d60c689b60 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 -- 172.21.15.79:0/1611164356 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 msgr2=0x55d60c689fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c689fe0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa36800bc80 tx=0x7fa36800f730 comp rx=0 tx=0).stop 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 -- 172.21.15.79:0/1611164356 shutdown_connections 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c6954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c689fe0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 --2- 172.21.15.79:0/1611164356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c689620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 -- 172.21.15.79:0/1611164356 >> 172.21.15.79:0/1611164356 conn(0x55d60c650c90 msgr2=0x55d60c684850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 -- 172.21.15.79:0/1611164356 shutdown_connections 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 -- 172.21.15.79:0/1611164356 wait complete. 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 -- start start 2024-04-22T06:54:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.793+0000 7fa38599e740 1 Processor -- start 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c692ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c68a8d0 0x55d60c68fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c6930e0 0x55d60c690460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d60c695e10 con 0x55d60c689b60 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d60c576eb0 con 0x55d60c68a8d0 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d60c695f90 con 0x55d60c6930e0 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 10 client.0 _ll_register_callbacks cb 0x55d60c5a3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c692ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c692ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57684/0 (socket says 172.21.15.79:57684) 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 -- 172.21.15.79:0/2367895185 learned_addr learned my addr 172.21.15.79:0/2367895185 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38126f640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c6930e0 0x55d60c690460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa37bfff640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c68a8d0 0x55d60c68fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 -- 172.21.15.79:0/2367895185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c6930e0 msgr2=0x55d60c690460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c6930e0 0x55d60c690460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 -- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c68a8d0 msgr2=0x55d60c68fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c68a8d0 0x55d60c68fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 -- 172.21.15.79:0/2367895185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d60c301210 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa380a6e640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c692ac0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa370098c60 tx=0x7fa37009cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d30c03d62895d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa35bfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa35bfff640 1 -- 172.21.15.79:0/2367895185 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3700a4f00 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa35bfff640 1 -- 172.21.15.79:0/2367895185 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fa370009d20 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3680054d0 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa35bfff640 1 -- 172.21.15.79:0/2367895185 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fa3700bb120 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 10 client.17469 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.794+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d60c3cd620 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa35bfff640 1 -- 172.21.15.79:0/2367895185 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3680054d0 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa35bfff640 1 -- 172.21.15.79:0/2367895185 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fa3700a0170 con 0x55d60c689b60 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa38599e740 10 client.17469 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa38599e740 10 client.17469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa38599e740 20 client.17469 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa38599e740 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c6a3190 0x55d60c6a5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa38599e740 4 client.17469 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d60c602880 con 0x55d60c6a3190 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa37bfff640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c6a3190 0x55d60c6a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa35bfff640 10 client.17469 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.795+0000 7fa37bfff640 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c6a3190 0x55d60c6a5580 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4252c2b185c91ff server_cookie=8343aa9491db9e9d in_seq=0 out_seq=0 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa35bfff640 1 -- 172.21.15.79:0/2367895185 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d60c602880 con 0x55d60c6a3190 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa35bfff640 10 client.17469 handle_command_reply: tid=0 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 client.17469 shutdown 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 msgr2=0x55d60c692ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c692ac0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa370098c60 tx=0x7fa37009cad0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 shutdown_connections 2024-04-22T06:54:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c6a3190 0x55d60c6a5580 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c68a8d0 0x55d60c68fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c6930e0 0x55d60c690460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 --2- 172.21.15.79:0/2367895185 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c692ac0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa35bfff640 0 client.17469 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.796+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 >> 172.21.15.79:0/2367895185 conn(0x55d60c650c90 msgr2=0x55d60c6865a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.797+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 shutdown_connections 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.797+0000 7fa38599e740 1 -- 172.21.15.79:0/2367895185 wait complete. 2024-04-22T06:54:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.797+0000 7fa38599e740 20 client.17469 trim_cache size 0 max 16384 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 Processor -- start 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 -- start start 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c69ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c692580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c692ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1b2fb0 con 0x55d60c689b60 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1b0880 con 0x55d60c6930e0 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.811+0000 7fa38599e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1ae6d0 con 0x55d60c68a8d0 2024-04-22T06:54:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c69ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c69ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57698/0 (socket says 172.21.15.79:57698) 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 -- 172.21.15.79:0/4073251546 learned_addr learned my addr 172.21.15.79:0/4073251546 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 -- 172.21.15.79:0/4073251546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 msgr2=0x55d60c692580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa37bfff640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c692580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa38126f640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c692ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c692580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 -- 172.21.15.79:0/4073251546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 msgr2=0x55d60c692ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c692ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 -- 172.21.15.79:0/4073251546 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d60c68e3a0 con 0x55d60c689b60 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa37bfff640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c692580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa38126f640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c692ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa380a6e640 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c69ed10 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa37009a540 tx=0x7fa37009a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b181b70eb06d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa3667fc640 1 -- 172.21.15.79:0/4073251546 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370014070 con 0x55d60c689b60 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa3667fc640 1 -- 172.21.15.79:0/4073251546 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3700a4f00 con 0x55d60c689b60 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.812+0000 7fa3667fc640 1 -- 172.21.15.79:0/4073251546 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3700cdcc0 con 0x55d60c689b60 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 -- 172.21.15.79:0/4073251546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 msgr2=0x55d60c69ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c69ed10 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa37009a540 tx=0x7fa37009a7a0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 -- 172.21.15.79:0/4073251546 shutdown_connections 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c692ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c692580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 --2- 172.21.15.79:0/4073251546 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c69ed10 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 -- 172.21.15.79:0/4073251546 >> 172.21.15.79:0/4073251546 conn(0x55d60c650c90 msgr2=0x55d60c68b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 -- 172.21.15.79:0/4073251546 shutdown_connections 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 -- 172.21.15.79:0/4073251546 wait complete. 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 -- start start 2024-04-22T06:54:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.813+0000 7fa38599e740 1 Processor -- start 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c6934c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c693a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c693f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1aed40 con 0x55d60c689b60 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1b01b0 con 0x55d60c6930e0 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1af410 con 0x55d60c68a8d0 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 10 client.0 _ll_register_callbacks cb 0x7fa370004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c6934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c6934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57706/0 (socket says 172.21.15.79:57706) 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 -- 172.21.15.79:0/3189203627 learned_addr learned my addr 172.21.15.79:0/3189203627 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38126f640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c693f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa37bfff640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c693a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 -- 172.21.15.79:0/3189203627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 msgr2=0x55d60c693a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c693a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 -- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 msgr2=0x55d60c693f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c693f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 -- 172.21.15.79:0/3189203627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d60c69a270 con 0x55d60c689b60 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38126f640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c693f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa380a6e640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c6934c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa3700986f0 tx=0x7fa37009b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891fd82feed76613 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa35bfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa35bfff640 1 -- 172.21.15.79:0/3189203627 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370014030 con 0x55d60c689b60 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa35bfff640 1 -- 172.21.15.79:0/3189203627 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fa3700cdcc0 con 0x55d60c689b60 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3700a4f00 con 0x55d60c689b60 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa35bfff640 1 -- 172.21.15.79:0/3189203627 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fa3700ad050 con 0x55d60c689b60 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 10 client.17475 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.814+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d60c69b4b0 con 0x55d60c689b60 2024-04-22T06:54:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa35bfff640 1 -- 172.21.15.79:0/3189203627 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3700a4f00 con 0x55d60c689b60 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa35bfff640 1 -- 172.21.15.79:0/3189203627 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fa370098180 con 0x55d60c689b60 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa38599e740 10 client.17475 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa38599e740 10 client.17475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa38599e740 20 client.17475 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa38599e740 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c709700 0x55d60c70baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa38599e740 4 client.17475 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d60c58e3d0 con 0x55d60c709700 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa37bfff640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c709700 0x55d60c70baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa37bfff640 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c709700 0x55d60c70baf0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac58c3123d7d572d server_cookie=50d0d6c7ba4785a9 in_seq=0 out_seq=0 2024-04-22T06:54:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.815+0000 7fa35bfff640 10 client.17475 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa35bfff640 1 -- 172.21.15.79:0/3189203627 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d60c58e3d0 con 0x55d60c709700 2024-04-22T06:54:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa35bfff640 10 client.17475 handle_command_reply: tid=0 2024-04-22T06:54:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 client.17475 shutdown 2024-04-22T06:54:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 msgr2=0x55d60c6934c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c6934c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa3700986f0 tx=0x7fa37009b6e0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 shutdown_connections 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d60c709700 0x55d60c70baf0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d60c6930e0 0x55d60c693f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa35bfff640 0 client.17475 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d60c68a8d0 0x55d60c693a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 --2- 172.21.15.79:0/3189203627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d60c689b60 0x55d60c6934c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 >> 172.21.15.79:0/3189203627 conn(0x55d60c650c90 msgr2=0x55d60c68e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 shutdown_connections 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 1 -- 172.21.15.79:0/3189203627 wait complete. 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.820+0000 7fa38599e740 20 client.17475 trim_cache size 0 max 16384 2024-04-22T06:54:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:54:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:54:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:54:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa36c0a3dc0 msgr2=0x7fa36c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa36c0a3dc0 0x7fa36c0a6280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa374006010 tx=0x7fa374005d80 comp rx=0 tx=0).stop 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 msgr2=0x7fa37c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 0x7fa37c102b30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa37000d950 tx=0x7fa37000de10 comp rx=0 tx=0).stop 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 shutdown_connections 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa36c0a3dc0 0x7fa36c0a6280 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa37c1035b0 0x7fa37c1a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa37c106170 0x7fa37c103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 --2- 172.21.15.79:0/3126814050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa37c1010f0 0x7fa37c102b30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 >> 172.21.15.79:0/3126814050 conn(0x7fa37c077fd0 msgr2=0x7fa37c0698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 shutdown_connections 2024-04-22T06:54:42.823 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:42.821+0000 7fa382cd2640 1 -- 172.21.15.79:0/3126814050 wait complete. 2024-04-22T06:54:42.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:42 smithi193 ceph-mon[31815]: pgmap v334: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:44.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:44 smithi190 ceph-mon[31421]: pgmap v335: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:54:44.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:44 smithi079 ceph-mon[27946]: pgmap v335: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:54:44.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:44 smithi193 ceph-mon[31815]: pgmap v335: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:54:46.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:46 smithi190 ceph-mon[31421]: pgmap v336: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:54:46.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:46 smithi079 ceph-mon[27946]: pgmap v336: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:54:46.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:46 smithi193 ceph-mon[31815]: pgmap v336: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:54:47.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:47 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:54:47.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:47 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:54:47.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:47 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:54:48.531 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:48 smithi193 ceph-mon[31815]: pgmap v337: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:48.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:48 smithi190 ceph-mon[31421]: pgmap v337: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:48.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:48 smithi079 ceph-mon[27946]: pgmap v337: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:54:50.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:54:50.441 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:54:50.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:54:50.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:50.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:54:50.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:54:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: pgmap v338: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-22T06:54:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-04-22T06:54:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: Adjusting osd_memory_target on smithi079 to 176.2M 2024-04-22T06:54:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: Unable to set osd_memory_target on smithi190 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:51.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:51 smithi193 ceph-mon[31815]: Unable to set osd_memory_target on smithi079 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:54:51.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: pgmap v338: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:51.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-22T06:54:51.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-04-22T06:54:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: Adjusting osd_memory_target on smithi079 to 176.2M 2024-04-22T06:54:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: Unable to set osd_memory_target on smithi190 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:51.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:51 smithi190 ceph-mon[31421]: Unable to set osd_memory_target on smithi079 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: pgmap v338: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi190 to 176.2M 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: Adjusting osd_memory_target on smithi079 to 176.2M 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: Unable to set osd_memory_target on smithi190 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:51.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:51 smithi079 ceph-mon[27946]: Unable to set osd_memory_target on smithi079 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:54:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:53 smithi193 ceph-mon[31815]: pgmap v339: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:53.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:53 smithi190 ceph-mon[31421]: pgmap v339: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:53.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:53 smithi079 ceph-mon[27946]: pgmap v339: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:55.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:55 smithi193 ceph-mon[31815]: pgmap v340: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:55.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:55 smithi190 ceph-mon[31421]: pgmap v340: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:55.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:55 smithi079 ceph-mon[27946]: pgmap v340: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 -- 172.21.15.79:0/2492017518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c0101aa0 msgr2=0x7fd3c0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2492017518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c0101aa0 0x7fd3c0105a90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009900 tx=0x7fd3b402bc30 comp rx=0 tx=0).stop 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 -- 172.21.15.79:0/2492017518 shutdown_connections 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2492017518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3c0106170 0x7fd3c010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2492017518 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c0101aa0 0x7fd3c0105a90 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2492017518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3c01010f0 0x7fd3c01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 -- 172.21.15.79:0/2492017518 >> 172.21.15.79:0/2492017518 conn(0x7fd3c0077fd0 msgr2=0x7fd3c00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 -- 172.21.15.79:0/2492017518 shutdown_connections 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.916+0000 7fd3c543f640 1 -- 172.21.15.79:0/2492017518 wait complete. 2024-04-22T06:54:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 Processor -- start 2024-04-22T06:54:55.917 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 -- start start 2024-04-22T06:54:55.917 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3c01010f0 0x7fd3c0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.917 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3c0106170 0x7fd3c0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 0x7fd3c01a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0111110 con 0x7fd3c019a020 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0110f90 con 0x7fd3c01010f0 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3c543f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0111290 con 0x7fd3c0106170 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 0x7fd3c01a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 0x7fd3c01a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38302/0 (socket says 172.21.15.79:38302) 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 -- 172.21.15.79:0/2727255233 learned_addr learned my addr 172.21.15.79:0/2727255233 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3beffd640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3c01010f0 0x7fd3c0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3c0106170 0x7fd3c0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 -- 172.21.15.79:0/2727255233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3c0106170 msgr2=0x7fd3c0199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3c0106170 0x7fd3c0199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 -- 172.21.15.79:0/2727255233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3c01010f0 msgr2=0x7fd3c0199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.918 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3c01010f0 0x7fd3c0199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.917+0000 7fd3bf7fe640 1 -- 172.21.15.79:0/2727255233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c01a4da0 con 0x7fd3c019a020 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.918+0000 7fd3bf7fe640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 0x7fd3c01a45d0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009fd0 tx=0x7fd3b800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.918+0000 7fd3a7fff640 1 -- 172.21.15.79:0/2727255233 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8011830 con 0x7fd3c019a020 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.918+0000 7fd3a7fff640 1 -- 172.21.15.79:0/2727255233 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b8016070 con 0x7fd3c019a020 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.918+0000 7fd3a7fff640 1 -- 172.21.15.79:0/2727255233 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8010de0 con 0x7fd3c019a020 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.918+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c01a5090 con 0x7fd3c019a020 2024-04-22T06:54:55.919 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.918+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c01a5560 con 0x7fd3c019a020 2024-04-22T06:54:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3a7fff640 1 -- 172.21.15.79:0/2727255233 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd3b8010380 con 0x7fd3c019a020 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 Processor -- start 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 -- start start 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5ac8f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac8ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac9b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558d5abf48d0 con 0x558d5ac990e0 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558d5ab91e80 con 0x558d5ac8fb60 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3c802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d5abf0890 con 0x558d5ac908d0 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.921+0000 7fd3a7fff640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd3a80a3cb0 0x7fd3a80a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac9b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac9b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38304/0 (socket says 172.21.15.79:38304) 2024-04-22T06:54:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 -- 172.21.15.79:0/3801386313 learned_addr learned my addr 172.21.15.79:0/3801386313 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3bf7fe640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd3a80a3cb0 0x7fd3a80a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5ac8f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3beffd640 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac8ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3a7fff640 1 -- 172.21.15.79:0/2727255233 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fd3b80cfb90 con 0x7fd3c019a020 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 -- 172.21.15.79:0/3801386313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 msgr2=0x558d5ac8f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5ac8f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 -- 172.21.15.79:0/3801386313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 msgr2=0x558d5ac8ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac8ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 -- 172.21.15.79:0/3801386313 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d5abb3320 con 0x558d5ac990e0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac9b4d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4005ec0 tx=0x7fd3b4005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1cfca06bac024 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3bf7fe640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd3a80a3cb0 0x7fd3a80a6170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8097cb0 tx=0x7fd3b80c68d0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3a57fa640 1 -- 172.21.15.79:0/3801386313 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4049070 con 0x558d5ac990e0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3a57fa640 1 -- 172.21.15.79:0/3801386313 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b4005500 con 0x558d5ac990e0 2024-04-22T06:54:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.922+0000 7fd3a57fa640 1 -- 172.21.15.79:0/3801386313 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4004ab0 con 0x558d5ac990e0 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 -- 172.21.15.79:0/3801386313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 msgr2=0x558d5ac9b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac9b4d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4005ec0 tx=0x7fd3b4005e30 comp rx=0 tx=0).stop 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 -- 172.21.15.79:0/3801386313 shutdown_connections 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac8ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac9b4d0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3801386313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5ac8f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 -- 172.21.15.79:0/3801386313 >> 172.21.15.79:0/3801386313 conn(0x558d5ac56c90 msgr2=0x558d5ac8a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 -- 172.21.15.79:0/3801386313 shutdown_connections 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 -- 172.21.15.79:0/3801386313 wait complete. 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.923+0000 7fd3c802b740 1 -- start start 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 Processor -- start 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac8fb60 0x558d5ac98b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac908d0 0x558d5ac95ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac96600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x558d5ac9bdb0 con 0x558d5ac990e0 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x558d5ab7ceb0 con 0x558d5ac908d0 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d5ac9bf30 con 0x558d5ac8fb60 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3c802b740 10 client.0 _ll_register_callbacks cb 0x558d5aba9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac96600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac96600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38308/0 (socket says 172.21.15.79:38308) 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3be7fc640 1 -- 172.21.15.79:0/3340063918 learned_addr learned my addr 172.21.15.79:0/3340063918 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3beffd640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac908d0 0x558d5ac95ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.924+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac8fb60 0x558d5ac98b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3beffd640 1 -- 172.21.15.79:0/3340063918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac8fb60 msgr2=0x558d5ac98b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3beffd640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac8fb60 0x558d5ac98b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3beffd640 1 -- 172.21.15.79:0/3340063918 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 msgr2=0x558d5ac96600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3beffd640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac96600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3beffd640 1 -- 172.21.15.79:0/3340063918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d5a907210 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac8fb60 0x558d5ac98b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3beffd640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac908d0 0x558d5ac95ec0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005c40 tx=0x7fd3b0011ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ece293506d6bcd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/3340063918 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b000fd20 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b4005500 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/3340063918 <== mon.2 v2:172.21.15.190:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fd3b001b3d0 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/3340063918 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fd3b001b610 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3c802b740 10 client.26162 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.925+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x558d5a9d3620 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/3340063918 <== mon.2 v2:172.21.15.190:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b4005500 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/3340063918 <== mon.2 v2:172.21.15.190:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fd3b001c820 con 0x558d5ac908d0 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3c802b740 10 client.26162 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3c802b740 10 client.26162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3c802b740 20 client.26162 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5aca9210 0x558d5acab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3c802b740 4 client.26162 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d5ac08880 con 0x558d5aca9210 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5aca9210 0x558d5acab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3a2ff5640 10 client.26162 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.926+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5aca9210 0x558d5acab620 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f60b83ca03c844e8 server_cookie=68708b96c7f81060 in_seq=0 out_seq=0 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/3340063918 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d5ac08880 con 0x558d5aca9210 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3a2ff5640 10 client.26162 handle_command_reply: tid=0 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 client.26162 shutdown 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac908d0 msgr2=0x558d5ac95ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac908d0 0x558d5ac95ec0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0005c40 tx=0x7fd3b0011ec0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 shutdown_connections 2024-04-22T06:54:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5aca9210 0x558d5acab620 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac908d0 0x558d5ac95ec0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3a2ff5640 0 client.26162 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5ac96600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 --2- 172.21.15.79:0/3340063918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac8fb60 0x558d5ac98b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 >> 172.21.15.79:0/3340063918 conn(0x558d5ac56c90 msgr2=0x558d5ac8c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 shutdown_connections 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 1 -- 172.21.15.79:0/3340063918 wait complete. 2024-04-22T06:54:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.927+0000 7fd3c802b740 20 client.26162 trim_cache size 0 max 16384 2024-04-22T06:54:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 Processor -- start 2024-04-22T06:54:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 -- start start 2024-04-22T06:54:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac95e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5acb0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01b0ba0 con 0x558d5ac990e0 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01ae470 con 0x558d5ac8fb60 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3c802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01ac2e0 con 0x558d5ac908d0 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5acb0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.942+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5acb0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38314/0 (socket says 172.21.15.79:38314) 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 -- 172.21.15.79:0/2912721880 learned_addr learned my addr 172.21.15.79:0/2912721880 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3beffd640 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 -- 172.21.15.79:0/2912721880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 msgr2=0x558d5aca4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 -- 172.21.15.79:0/2912721880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 msgr2=0x558d5ac95e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac95e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 -- 172.21.15.79:0/2912721880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d5ac8ee30 con 0x558d5ac990e0 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3beffd640 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5acb0f30 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4036040 tx=0x7fd3b4004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa5f07d9009673 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3a4ff9640 1 -- 172.21.15.79:0/2912721880 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4049040 con 0x558d5ac990e0 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3a4ff9640 1 -- 172.21.15.79:0/2912721880 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b404b040 con 0x558d5ac990e0 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.943+0000 7fd3a4ff9640 1 -- 172.21.15.79:0/2912721880 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4003850 con 0x558d5ac990e0 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- 172.21.15.79:0/2912721880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 msgr2=0x558d5acb0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5acb0f30 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4036040 tx=0x7fd3b4004560 comp rx=0 tx=0).stop 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- 172.21.15.79:0/2912721880 shutdown_connections 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac990e0 0x558d5acb0f30 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- 172.21.15.79:0/2912721880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac8fb60 0x558d5ac95e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- 172.21.15.79:0/2912721880 >> 172.21.15.79:0/2912721880 conn(0x558d5ac56c90 msgr2=0x558d5ac95750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- 172.21.15.79:0/2912721880 shutdown_connections 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- 172.21.15.79:0/2912721880 wait complete. 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- start start 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 Processor -- start 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 0x558d5aca11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac990e0 0x558d5ac93160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01ac930 con 0x558d5ac8fb60 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01adda0 con 0x558d5ac990e0 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01ad000 con 0x558d5ac908d0 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3c802b740 10 client.0 _ll_register_callbacks cb 0x7fd3b000a990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 0x558d5aca11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3be7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 0x558d5aca11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38324/0 (socket says 172.21.15.79:38324) 2024-04-22T06:54:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.944+0000 7fd3be7fc640 1 -- 172.21.15.79:0/1244470265 learned_addr learned my addr 172.21.15.79:0/1244470265 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3beffd640 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3be7fc640 1 -- 172.21.15.79:0/1244470265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 msgr2=0x558d5aca1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3be7fc640 1 -- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac990e0 msgr2=0x558d5ac93160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac990e0 0x558d5ac93160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3be7fc640 1 -- 172.21.15.79:0/1244470265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d5ac936a0 con 0x558d5ac8fb60 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3be7fc640 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 0x558d5aca11c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000c00 tx=0x7fd3b4009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0624fa7eb4a75f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/1244470265 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4065060 con 0x558d5ac8fb60 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/1244470265 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fd3b4057320 con 0x558d5ac8fb60 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b404b040 con 0x558d5ac8fb60 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/1244470265 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fd3b4049040 con 0x558d5ac8fb60 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3c802b740 10 client.17493 fetch_fsmap learned FSMap version 82 2024-04-22T06:54:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x558d5ac99d20 con 0x558d5ac8fb60 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.945+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/1244470265 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b404b040 con 0x558d5ac8fb60 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/1244470265 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fd3b4040040 con 0x558d5ac8fb60 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3c802b740 10 client.17493 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3c802b740 10 client.17493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3c802b740 20 client.17493 populate_metadata read hostname 'smithi079' 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3c802b740 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5ac9a010 0x558d5ad117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3c802b740 4 client.17493 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d5ab943d0 con 0x558d5ac9a010 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3beffd640 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5ac9a010 0x558d5ad117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3a2ff5640 10 client.17493 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.946+0000 7fd3beffd640 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5ac9a010 0x558d5ad117b0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94add190ec7e2af7 server_cookie=9e7a674f7241f995 in_seq=0 out_seq=0 2024-04-22T06:54:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3a2ff5640 1 -- 172.21.15.79:0/1244470265 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d5ab943d0 con 0x558d5ac9a010 2024-04-22T06:54:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3a2ff5640 10 client.17493 handle_command_reply: tid=0 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 client.17493 shutdown 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 msgr2=0x558d5aca11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 0x558d5aca11c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000c00 tx=0x7fd3b4009810 comp rx=0 tx=0).stop 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 shutdown_connections 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x558d5ac9a010 0x558d5ad117b0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558d5ac990e0 0x558d5ac93160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d5ac908d0 0x558d5aca1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 --2- 172.21.15.79:0/1244470265 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x558d5ac8fb60 0x558d5aca11c0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3a2ff5640 0 client.17493 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 >> 172.21.15.79:0/1244470265 conn(0x558d5ac56c90 msgr2=0x558d5ac90f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 shutdown_connections 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 1 -- 172.21.15.79:0/1244470265 wait complete. 2024-04-22T06:54:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.952+0000 7fd3c802b740 20 client.17493 trim_cache size 0 max 16384 2024-04-22T06:54:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:54:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:54:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:54:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd3a80a3cb0 msgr2=0x7fd3a80a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd3a80a3cb0 0x7fd3a80a6170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8097cb0 tx=0x7fd3b80c68d0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 msgr2=0x7fd3c01a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 0x7fd3c01a45d0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009fd0 tx=0x7fd3b800b8e0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 shutdown_connections 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3c019a020 0x7fd3c01a45d0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3c0106170 0x7fd3c0199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd3a80a3cb0 0x7fd3a80a6170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 --2- 172.21.15.79:0/2727255233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3c01010f0 0x7fd3c0199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 >> 172.21.15.79:0/2727255233 conn(0x7fd3c0077fd0 msgr2=0x7fd3c00fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 shutdown_connections 2024-04-22T06:54:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:54:55.953+0000 7fd3c543f640 1 -- 172.21.15.79:0/2727255233 wait complete. 2024-04-22T06:54:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:57 smithi193 ceph-mon[31815]: pgmap v341: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:57 smithi190 ceph-mon[31421]: pgmap v341: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:57.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:57 smithi079 ceph-mon[27946]: pgmap v341: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:59.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:54:59 smithi190 ceph-mon[31421]: pgmap v342: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:54:59.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:54:59 smithi079 ceph-mon[27946]: pgmap v342: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:54:59.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:54:59 smithi193 ceph-mon[31815]: pgmap v342: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:55:00.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:00 smithi193 ceph-mon[31815]: pgmap v343: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:00.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:00 smithi190 ceph-mon[31421]: pgmap v343: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:00.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:00 smithi079 ceph-mon[27946]: pgmap v343: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:02.001 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T06:55:02.001 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-22T06:55:02.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:55:02.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 -- 172.21.15.79:0/376887615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc107260 msgr2=0x7ff9fc10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 --2- 172.21.15.79:0/376887615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc107260 0x7ff9fc10ee30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009ca0 tx=0x7ff9f402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 -- 172.21.15.79:0/376887615 shutdown_connections 2024-04-22T06:55:02.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 --2- 172.21.15.79:0/376887615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc107260 0x7ff9fc10ee30 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 --2- 172.21.15.79:0/376887615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc102b40 0x7ff9fc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 --2- 172.21.15.79:0/376887615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc102190 0x7ff9fc102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 -- 172.21.15.79:0/376887615 >> 172.21.15.79:0/376887615 conn(0x7ff9fc0793e0 msgr2=0x7ff9fc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 -- 172.21.15.79:0/376887615 shutdown_connections 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.117+0000 7ffa03e14640 1 -- 172.21.15.79:0/376887615 wait complete. 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 Processor -- start 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 -- start start 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 0x7ff9fc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc107260 0x7ff9fc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc19cfd0 0x7ff9fc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1121d0 con 0x7ff9fc102190 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc112050 con 0x7ff9fc19cfd0 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa03e14640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc112350 con 0x7ff9fc107260 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa01bb0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 0x7ff9fc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa01bb0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 0x7ff9fc198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38356/0 (socket says 172.21.15.79:38356) 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa01bb0640 1 -- 172.21.15.79:0/2991180056 learned_addr learned my addr 172.21.15.79:0/2991180056 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:02.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.118+0000 7ffa023b1640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc19cfd0 0x7ff9fc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa01bb0640 1 -- 172.21.15.79:0/2991180056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc107260 msgr2=0x7ff9fc198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa01bb0640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc107260 0x7ff9fc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa01bb0640 1 -- 172.21.15.79:0/2991180056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc19cfd0 msgr2=0x7ff9fc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa01bb0640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc19cfd0 0x7ff9fc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa01bb0640 1 -- 172.21.15.79:0/2991180056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc19dac0 con 0x7ff9fc102190 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa023b1640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc19cfd0 0x7ff9fc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa01bb0640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 0x7ff9fc198710 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800d950 tx=0x7ff9f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8017070 con 0x7ff9fc102190 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9f8009c30 con 0x7ff9fc102190 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc19ddb0 con 0x7ff9fc102190 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f800f620 con 0x7ff9fc102190 2024-04-22T06:55:02.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.119+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc1a55a0 con 0x7ff9fc102190 2024-04-22T06:55:02.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.120+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc069230 con 0x7ff9fc102190 2024-04-22T06:55:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.121+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff9f8010460 con 0x7ff9fc102190 2024-04-22T06:55:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.122+0000 7ff9f2ffd640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff9e80a3d20 0x7ff9e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.122+0000 7ffa013af640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff9e80a3d20 0x7ff9e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.122+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7ff9f80d0970 con 0x7ff9fc102190 2024-04-22T06:55:02.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.122+0000 7ffa013af640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff9e80a3d20 0x7ff9e80a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc101820 tx=0x7ff9ec009590 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.127+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9f809cb90 con 0x7ff9fc102190 2024-04-22T06:55:02.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.345+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9fc102b40 con 0x7ff9fc102190 2024-04-22T06:55:02.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.346+0000 7ff9f2ffd640 1 -- 172.21.15.79:0/2991180056 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+11883 (secure 0 0 0) 0x7ff9f8009de0 con 0x7ff9fc102190 2024-04-22T06:55:02.344 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:55:02.345 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:54:15.404622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":26003,"mds_2":26000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26000":{"gid":26000,"name":"e","rank":2,"incarnation":73,"state":"up:active","state_seq":21,"addr":"172.21.15.190:6835/1769517674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1769517674},{"type":"v1","addr":"172.21.15.190:6835","nonce":1769517674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26003":{"gid":26003,"name":"f","rank":1,"incarnation":75,"state":"up:active","state_seq":21,"addr":"172.21.15.193:6835/2674386425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2674386425},{"type":"v1","addr":"172.21.15.193:6835","nonce":2674386425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26095":{"gid":26095,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/3177981466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3177981466},{"type":"v1","addr":"172.21.15.193:6839","nonce":3177981466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26096":{"gid":26096,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/246451352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":246451352},{"type":"v1","addr":"172.21.15.190:6837","nonce":246451352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26113":{"gid":26113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/4272953642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":4272953642},{"type":"v1","addr":"172.21.15.193:6837","nonce":4272953642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[26003,24467,26000]},"id":1}]} 2024-04-22T06:55:02.345 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 82 2024-04-22T06:55:02.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff9e80a3d20 msgr2=0x7ff9e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff9e80a3d20 0x7ff9e80a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc101820 tx=0x7ff9ec009590 comp rx=0 tx=0).stop 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 msgr2=0x7ff9fc198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 0x7ff9fc198710 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800d950 tx=0x7ff9f800de10 comp rx=0 tx=0).stop 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 shutdown_connections 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff9e80a3d20 0x7ff9e80a61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff9fc19cfd0 0x7ff9fc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc107260 0x7ff9fc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 --2- 172.21.15.79:0/2991180056 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9fc102190 0x7ff9fc198710 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 >> 172.21.15.79:0/2991180056 conn(0x7ff9fc0793e0 msgr2=0x7ff9fc0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 shutdown_connections 2024-04-22T06:55:02.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.348+0000 7ffa03e14640 1 -- 172.21.15.79:0/2991180056 wait complete. 2024-04-22T06:55:02.358 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 90.0 --await' 2024-04-22T06:55:02.358 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 90.0 --await 2024-04-22T06:55:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 -- 172.21.15.79:0/1490423673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd810c580 msgr2=0x7fadd810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 --2- 172.21.15.79:0/1490423673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd810c580 0x7fadd810e9b0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009ca0 tx=0x7fadd002c180 comp rx=0 tx=0).stop 2024-04-22T06:55:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 -- 172.21.15.79:0/1490423673 shutdown_connections 2024-04-22T06:55:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 --2- 172.21.15.79:0/1490423673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 0x7fadd810bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 --2- 172.21.15.79:0/1490423673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd810c580 0x7fadd810e9b0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 --2- 172.21.15.79:0/1490423673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadd81031b0 0x7fadd8103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 -- 172.21.15.79:0/1490423673 >> 172.21.15.79:0/1490423673 conn(0x7fadd80771d0 msgr2=0x7fadd80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 -- 172.21.15.79:0/1490423673 shutdown_connections 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.467+0000 7fade07b2640 1 -- 172.21.15.79:0/1490423673 wait complete. 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 Processor -- start 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 -- start start 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 0x7fadd8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 0x7fadd8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadd810c580 0x7fadd819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fadd8112200 con 0x7fadd81031b0 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fadd8112080 con 0x7fadd8103ad0 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fade07b2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fadd8112380 con 0x7fadd810c580 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 0x7fadd8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 0x7fadd8198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38378/0 (socket says 172.21.15.79:38378) 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 -- 172.21.15.79:0/237593120 learned_addr learned my addr 172.21.15.79:0/237593120 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadded4f640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadd810c580 0x7fadd819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadddd4d640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 0x7fadd8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 -- 172.21.15.79:0/237593120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadd810c580 msgr2=0x7fadd819d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadd810c580 0x7fadd819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 -- 172.21.15.79:0/237593120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 msgr2=0x7fadd8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 0x7fadd8198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 -- 172.21.15.79:0/237593120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd819d730 con 0x7fadd81031b0 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadddd4d640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 0x7fadd8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.468+0000 7fadde54e640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 0x7fadd8198740 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fadc800b4a0 tx=0x7fadc800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.469+0000 7fadcf7fe640 1 -- 172.21.15.79:0/237593120 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8014070 con 0x7fadd81031b0 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.469+0000 7fadcf7fe640 1 -- 172.21.15.79:0/237593120 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc8010ac0 con 0x7fadd81031b0 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.469+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd819da20 con 0x7fadd81031b0 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.469+0000 7fadcf7fe640 1 -- 172.21.15.79:0/237593120 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8017c30 con 0x7fadd81031b0 2024-04-22T06:55:02.472 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.469+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd81a52b0 con 0x7fadd81031b0 2024-04-22T06:55:02.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fadcf7fe640 1 -- 172.21.15.79:0/237593120 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fadc8017070 con 0x7fadd81031b0 2024-04-22T06:55:02.473 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 Processor -- start 2024-04-22T06:55:02.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 -- start start 2024-04-22T06:55:02.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a29c0 0x557a605a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.480 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605abec0 0x557a605ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557a605030b0 con 0x557a605a32e0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557a604b4680 con 0x557a605a29c0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fade33a1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557a604c0c60 con 0x557a605abec0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fadcf7fe640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fadc40a3d20 0x7fadc40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fadde54e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fadde54e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38384/0 (socket says 172.21.15.79:38384) 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.471+0000 7fadde54e640 1 -- 172.21.15.79:0/3799539347 learned_addr learned my addr 172.21.15.79:0/3799539347 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadded4f640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fadc40a3d20 0x7fadc40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadded4f640 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a29c0 0x557a605a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadcf7fe640 1 -- 172.21.15.79:0/237593120 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fadc80cffb0 con 0x7fadd81031b0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadddd4d640 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605abec0 0x557a605ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.481 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadde54e640 1 -- 172.21.15.79:0/3799539347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605abec0 msgr2=0x557a605ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadde54e640 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605abec0 0x557a605ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadde54e640 1 -- 172.21.15.79:0/3799539347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a29c0 msgr2=0x557a605a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadde54e640 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a29c0 0x557a605a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadde54e640 1 -- 172.21.15.79:0/3799539347 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a602e5c90 con 0x557a605a32e0 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadded4f640 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a29c0 0x557a605a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadde54e640 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605ab980 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fadc80a4320 tx=0x7fadc8017650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2d32f25c252be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadded4f640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fadc40a3d20 0x7fadc40a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fadd0005e30 tx=0x7fadd0005da0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadccff9640 1 -- 172.21.15.79:0/3799539347 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc801e070 con 0x557a605a32e0 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadccff9640 1 -- 172.21.15.79:0/3799539347 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc809c2e0 con 0x557a605a32e0 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.472+0000 7fadccff9640 1 -- 172.21.15.79:0/3799539347 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80cd500 con 0x557a605a32e0 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 -- 172.21.15.79:0/3799539347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 msgr2=0x557a605ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605ab980 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fadc80a4320 tx=0x7fadc8017650 comp rx=0 tx=0).stop 2024-04-22T06:55:02.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 -- 172.21.15.79:0/3799539347 shutdown_connections 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605abec0 0x557a605ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605ab980 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 --2- 172.21.15.79:0/3799539347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a29c0 0x557a605a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 -- 172.21.15.79:0/3799539347 >> 172.21.15.79:0/3799539347 conn(0x557a6059d0f0 msgr2=0x557a6059d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 -- 172.21.15.79:0/3799539347 shutdown_connections 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 -- 172.21.15.79:0/3799539347 wait complete. 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 -- start start 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.473+0000 7fade33a1740 1 Processor -- start 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a32e0 0x557a605a9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 0x557a605a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x557a60506400 con 0x557a605abec0 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x557a604c2840 con 0x557a605a32e0 2024-04-22T06:55:02.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557a6050b720 con 0x557a605a29c0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 10 client.0 _ll_register_callbacks cb 0x557a60513ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 0x557a605a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 0x557a605a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38394/0 (socket says 172.21.15.79:38394) 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 -- 172.21.15.79:0/2514666015 learned_addr learned my addr 172.21.15.79:0/2514666015 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadde54e640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadddd4d640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a32e0 0x557a605a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 -- 172.21.15.79:0/2514666015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 msgr2=0x557a605a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605a8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 -- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a32e0 msgr2=0x557a605a9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a32e0 0x557a605a9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 -- 172.21.15.79:0/2514666015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a60346420 con 0x557a605abec0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadddd4d640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a32e0 0x557a605a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadded4f640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 0x557a605a9a40 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fadd0036000 tx=0x7fadd002c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bf0897c09fa177 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadc1ffb640 1 -- 172.21.15.79:0/2514666015 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0050020 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadc1ffb640 1 -- 172.21.15.79:0/2514666015 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fadd00046c0 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc809c2e0 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fadc1ffb640 1 -- 172.21.15.79:0/2514666015 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fadd0048040 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 10 client.17511 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.474+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x557a601d9580 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fadc1ffb640 1 -- 172.21.15.79:0/2514666015 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc809c2e0 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fadc1ffb640 1 -- 172.21.15.79:0/2514666015 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fadd004b030 con 0x557a605abec0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fade33a1740 10 client.17511 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fade33a1740 10 client.17511 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fade33a1740 20 client.17511 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fade33a1740 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a605be210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fade33a1740 4 client.17511 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a60514b10 con 0x557a605bbdb0 2024-04-22T06:55:02.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fadde54e640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a605be210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fadc1ffb640 10 client.17511 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.475+0000 7fadde54e640 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a605be210 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0f2a3514fb8acc1 server_cookie=47418fd8a590adc5 in_seq=0 out_seq=0 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fadc1ffb640 1 -- 172.21.15.79:0/2514666015 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a60514b10 con 0x557a605bbdb0 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fadc1ffb640 10 client.17511 handle_command_reply: tid=0 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 client.17511 shutdown 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 msgr2=0x557a605a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 0x557a605a9a40 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fadd0036000 tx=0x7fadd002c180 comp rx=0 tx=0).stop 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 shutdown_connections 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a605be210 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605a32e0 0x557a605a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fadc1ffb640 0 client.17511 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605abec0 0x557a605a9a40 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 --2- 172.21.15.79:0/2514666015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 >> 172.21.15.79:0/2514666015 conn(0x557a6059d0f0 msgr2=0x557a6059ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 shutdown_connections 2024-04-22T06:55:02.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.476+0000 7fade33a1740 1 -- 172.21.15.79:0/2514666015 wait complete. 2024-04-22T06:55:02.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.477+0000 7fade33a1740 20 client.17511 trim_cache size 0 max 16384 2024-04-22T06:55:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 Processor -- start 2024-04-22T06:55:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 -- start start 2024-04-22T06:55:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 0x557a605ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a32e0 0x557a605b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81ae740 con 0x557a605a29c0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81ac590 con 0x557a605abec0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fade33a1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81ae070 con 0x557a605a32e0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fadde54e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 0x557a605ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fadde54e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 0x557a605ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40512/0 (socket says 172.21.15.79:40512) 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.493+0000 7fadde54e640 1 -- 172.21.15.79:0/3080790855 learned_addr learned my addr 172.21.15.79:0/3080790855 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadddd4d640 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a32e0 0x557a605b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadde54e640 1 -- 172.21.15.79:0/3080790855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a32e0 msgr2=0x557a605b7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadde54e640 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a32e0 0x557a605b7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadde54e640 1 -- 172.21.15.79:0/3080790855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 msgr2=0x557a605c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadde54e640 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadde54e640 1 -- 172.21.15.79:0/3080790855 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a605a7f90 con 0x557a605a29c0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadddd4d640 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a32e0 0x557a605b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadde54e640 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 0x557a605ab6f0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fadc8098040 tx=0x7fadc80b5210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b165f752f50bba26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadc3fff640 1 -- 172.21.15.79:0/3080790855 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc801e040 con 0x557a605a29c0 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadc3fff640 1 -- 172.21.15.79:0/3080790855 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc80a4850 con 0x557a605a29c0 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fadc3fff640 1 -- 172.21.15.79:0/3080790855 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80cd470 con 0x557a605a29c0 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fade33a1740 1 -- 172.21.15.79:0/3080790855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 msgr2=0x557a605ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fade33a1740 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 0x557a605ab6f0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fadc8098040 tx=0x7fadc80b5210 comp rx=0 tx=0).stop 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fade33a1740 1 -- 172.21.15.79:0/3080790855 shutdown_connections 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fade33a1740 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fade33a1740 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a32e0 0x557a605b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.494+0000 7fade33a1740 1 --2- 172.21.15.79:0/3080790855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a29c0 0x557a605ab6f0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.498 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- 172.21.15.79:0/3080790855 >> 172.21.15.79:0/3080790855 conn(0x557a6059d0f0 msgr2=0x557a6059c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- 172.21.15.79:0/3080790855 shutdown_connections 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- 172.21.15.79:0/3080790855 wait complete. 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- start start 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 Processor -- start 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81ad2d0 con 0x557a605a32e0 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81acc00 con 0x557a605abec0 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fadd81ad9a0 con 0x557a605a29c0 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fade33a1740 10 client.0 _ll_register_callbacks cb 0x7fadd0037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:02.499 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40524/0 (socket says 172.21.15.79:40524) 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 -- 172.21.15.79:0/684080150 learned_addr learned my addr 172.21.15.79:0/684080150 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadde54e640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 -- 172.21.15.79:0/684080150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 msgr2=0x557a605b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadded4f640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 -- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 msgr2=0x557a605b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadddd4d640 1 -- 172.21.15.79:0/684080150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a605b1410 con 0x557a605a32e0 2024-04-22T06:55:02.500 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.495+0000 7fadded4f640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadddd4d640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605b42c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005fd0 tx=0x7fadd40137d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a107265534d195b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadc1ffb640 1 -- 172.21.15.79:0/684080150 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4020060 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadc1ffb640 1 -- 172.21.15.79:0/684080150 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fadd4011670 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc80a4850 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadc1ffb640 1 -- 172.21.15.79:0/684080150 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fadd4027050 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 10 client.17517 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x557a60622d00 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadc1ffb640 1 -- 172.21.15.79:0/684080150 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc80a4850 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadc1ffb640 1 -- 172.21.15.79:0/684080150 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fadd400f040 con 0x557a605a32e0 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 10 client.17517 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 10 client.17517 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:55:02.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 20 client.17517 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a60625590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 4 client.17517 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}] 2024-04-22T06:55:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}) -- 0x557a6050e980 con 0x557a605bbdb0 2024-04-22T06:55:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.496+0000 7fadde54e640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a60625590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.497+0000 7fadc1ffb640 10 client.17517 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:02.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.497+0000 7fadde54e640 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a60625590 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d92591b900f87b server_cookie=ae42aa09fc4b04d4 in_seq=0 out_seq=0 2024-04-22T06:55:02.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fadc1ffb640 1 -- 172.21.15.79:0/684080150 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557a6050e980 con 0x557a605bbdb0 2024-04-22T06:55:02.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fadc1ffb640 10 client.17517 handle_command_reply: tid=0 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 client.17517 shutdown 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 msgr2=0x557a605b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605b42c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005fd0 tx=0x7fadd40137d0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 shutdown_connections 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x557a605bbdb0 0x557a60625590 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557a605abec0 0x557a605b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x557a605a32e0 0x557a605b42c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 --2- 172.21.15.79:0/684080150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a605a29c0 0x557a605b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fadc1ffb640 0 client.17517 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 >> 172.21.15.79:0/684080150 conn(0x557a6059d0f0 msgr2=0x557a6059b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 shutdown_connections 2024-04-22T06:55:02.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 1 -- 172.21.15.79:0/684080150 wait complete. 2024-04-22T06:55:02.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade33a1740 20 client.17517 trim_cache size 0 max 16384 2024-04-22T06:55:02.507 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:55:02.507 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 82, 2024-04-22T06:55:02.507 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:55:02.507 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 34, 2024-04-22T06:55:02.507 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "13ba45d6": { 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 34, 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 90.0, 2024-04-22T06:55:02.508 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCED", 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:02.509 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:55:02.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fadc40a3d20 msgr2=0x7fadc40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fadc40a3d20 0x7fadc40a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fadd0005e30 tx=0x7fadd0005da0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 msgr2=0x7fadd8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 0x7fadd8198740 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fadc800b4a0 tx=0x7fadc800b960 comp rx=0 tx=0).stop 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 shutdown_connections 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fadc40a3d20 0x7fadc40a61e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadd8103ad0 0x7fadd8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadd810c580 0x7fadd819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 --2- 172.21.15.79:0/237593120 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadd81031b0 0x7fadd8198740 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 >> 172.21.15.79:0/237593120 conn(0x7fadd80771d0 msgr2=0x7fadd80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 shutdown_connections 2024-04-22T06:55:02.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:02.507+0000 7fade07b2640 1 -- 172.21.15.79:0/237593120 wait complete. 2024-04-22T06:55:02.516 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 13ba45d6, quiesce took 0.5 seconds. Will release after: 30.0 2024-04-22T06:55:02.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:02 smithi193 ceph-mon[31815]: pgmap v344: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:55:02.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:02 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2991180056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:02.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:02 smithi190 ceph-mon[31421]: pgmap v344: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:55:02.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:02 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2991180056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:02.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:02 smithi079 ceph-mon[27946]: pgmap v344: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:55:02.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:02 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2991180056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:04.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:04 smithi193 ceph-mon[31815]: pgmap v345: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:04.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:04 smithi190 ceph-mon[31421]: pgmap v345: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:04.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:04 smithi079 ceph-mon[27946]: pgmap v345: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:06.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:06 smithi193 ceph-mon[31815]: pgmap v346: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:06.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:06 smithi190 ceph-mon[31421]: pgmap v346: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:06.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:06 smithi079 ceph-mon[27946]: pgmap v346: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:08.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:08 smithi193 ceph-mon[31815]: pgmap v347: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T06:55:08.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:08 smithi190 ceph-mon[31421]: pgmap v347: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T06:55:08.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:08 smithi079 ceph-mon[27946]: pgmap v347: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 -- 172.21.15.79:0/1258338723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a01010a0 msgr2=0x7fd8a0101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1258338723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a01010a0 0x7fd8a0101520 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fd890009930 tx=0x7fd89002bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 -- 172.21.15.79:0/1258338723 shutdown_connections 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1258338723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0101a60 0x7fd8a010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1258338723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a01010a0 0x7fd8a0101520 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1258338723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a0105e80 0x7fd8a0106260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 -- 172.21.15.79:0/1258338723 >> 172.21.15.79:0/1258338723 conn(0x7fd8a0077cd0 msgr2=0x7fd8a00780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 -- 172.21.15.79:0/1258338723 shutdown_connections 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.049+0000 7fd8a7398640 1 -- 172.21.15.79:0/1258338723 wait complete. 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 Processor -- start 2024-04-22T06:55:09.048 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 -- start start 2024-04-22T06:55:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a01010a0 0x7fd8a019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 0x7fd8a019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0105e80 0x7fd8a01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a0112300 con 0x7fd8a0101a60 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a0112180 con 0x7fd8a0105e80 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a7398640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a0112480 con 0x7fd8a01010a0 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a4933640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 0x7fd8a019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a01010a0 0x7fd8a019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a4933640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 0x7fd8a019e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40544/0 (socket says 172.21.15.79:40544) 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a4933640 1 -- 172.21.15.79:0/1956879714 learned_addr learned my addr 172.21.15.79:0/1956879714 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.050+0000 7fd8a5935640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0105e80 0x7fd8a01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a4933640 1 -- 172.21.15.79:0/1956879714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a01010a0 msgr2=0x7fd8a019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a4933640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a01010a0 0x7fd8a019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a4933640 1 -- 172.21.15.79:0/1956879714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0105e80 msgr2=0x7fd8a01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a4933640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0105e80 0x7fd8a01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a4933640 1 -- 172.21.15.79:0/1956879714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a0198d40 con 0x7fd8a0101a60 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a5935640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0105e80 0x7fd8a01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a5134640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a01010a0 0x7fd8a019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a4933640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 0x7fd8a019e950 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fd890009230 tx=0x7fd89002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8967fc640 1 -- 172.21.15.79:0/1956879714 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890043020 con 0x7fd8a0101a60 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8967fc640 1 -- 172.21.15.79:0/1956879714 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89002caf0 con 0x7fd8a0101a60 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a0198fd0 con 0x7fd8a0101a60 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8967fc640 1 -- 172.21.15.79:0/1956879714 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890003af0 con 0x7fd8a0101a60 2024-04-22T06:55:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.051+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a01a52d0 con 0x7fd8a0101a60 2024-04-22T06:55:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.053+0000 7fd8967fc640 1 -- 172.21.15.79:0/1956879714 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd890046020 con 0x7fd8a0101a60 2024-04-22T06:55:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 Processor -- start 2024-04-22T06:55:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 -- start start 2024-04-22T06:55:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da511a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da511afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.063 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da51264d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.064 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563da507f8d0 con 0x563da511b8d0 2024-04-22T06:55:09.064 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563da501ce80 con 0x563da51240e0 2024-04-22T06:55:09.064 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563da507b890 con 0x563da511ab60 2024-04-22T06:55:09.064 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8967fc640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd88c0a3dc0 0x7fd88c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.064 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a5935640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da511a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.064 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a5935640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da511a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40550/0 (socket says 172.21.15.79:40550) 2024-04-22T06:55:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a5935640 1 -- 172.21.15.79:0/347487534 learned_addr learned my addr 172.21.15.79:0/347487534 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a5935640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd88c0a3dc0 0x7fd88c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a5134640 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da511afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8967fc640 1 -- 172.21.15.79:0/1956879714 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fd8900f2c60 con 0x7fd8a0101a60 2024-04-22T06:55:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.054+0000 7fd8a4933640 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da51264d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a4933640 1 -- 172.21.15.79:0/347487534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 msgr2=0x563da511afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a4933640 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da511afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a4933640 1 -- 172.21.15.79:0/347487534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 msgr2=0x563da511a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a4933640 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da511a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.066 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a4933640 1 -- 172.21.15.79:0/347487534 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da503e320 con 0x563da51240e0 2024-04-22T06:55:09.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a5935640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd88c0a3dc0 0x7fd88c0a6280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd898013c30 tx=0x7fd89800a9e0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:09.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd8a4933640 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da51264d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd8900f0040 tx=0x7fd8900035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ffe24a8cc397652 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:09.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd88b7fe640 1 -- 172.21.15.79:0/347487534 <== mon.2 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8900be890 con 0x563da51240e0 2024-04-22T06:55:09.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd88b7fe640 1 -- 172.21.15.79:0/347487534 <== mon.2 v2:172.21.15.190:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8900c2c40 con 0x563da51240e0 2024-04-22T06:55:09.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.055+0000 7fd88b7fe640 1 -- 172.21.15.79:0/347487534 <== mon.2 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8900a90e0 con 0x563da51240e0 2024-04-22T06:55:09.067 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 -- 172.21.15.79:0/347487534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 msgr2=0x563da51264d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da51264d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd8900f0040 tx=0x7fd8900035f0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 -- 172.21.15.79:0/347487534 shutdown_connections 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da51264d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da511afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/347487534 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da511a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 -- 172.21.15.79:0/347487534 >> 172.21.15.79:0/347487534 conn(0x563da50e1c90 msgr2=0x563da5115850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 -- 172.21.15.79:0/347487534 shutdown_connections 2024-04-22T06:55:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 -- 172.21.15.79:0/347487534 wait complete. 2024-04-22T06:55:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 -- start start 2024-04-22T06:55:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.056+0000 7fd8a9f81740 1 Processor -- start 2024-04-22T06:55:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da51239a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da5120d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da5121270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x563da5126bd0 con 0x563da511b8d0 2024-04-22T06:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563da5007eb0 con 0x563da51240e0 2024-04-22T06:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563da5126d50 con 0x563da511ab60 2024-04-22T06:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a9f81740 10 client.0 _ll_register_callbacks cb 0x563da5034b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da5120d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da51239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da51239a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.79:46926/0 (socket says 172.21.15.79:46926) 2024-04-22T06:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a5134640 1 -- 172.21.15.79:0/566846294 learned_addr learned my addr 172.21.15.79:0/566846294 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da5120d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40566/0 (socket says 172.21.15.79:40566) 2024-04-22T06:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a5935640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da5121270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 -- 172.21.15.79:0/566846294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 msgr2=0x563da51239a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da51239a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 -- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 msgr2=0x563da5121270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da5121270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a4933640 1 -- 172.21.15.79:0/566846294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da4d92210 con 0x563da511b8d0 2024-04-22T06:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a5134640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da51239a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.057+0000 7fd8a5935640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da5121270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:09.072 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a4933640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da5120d30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd8900be590 tx=0x7fd890005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc9d574389629b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd888ff9640 1 -- 172.21.15.79:0/566846294 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8900be890 con 0x563da511b8d0 2024-04-22T06:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd888ff9640 1 -- 172.21.15.79:0/566846294 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fd8900a90b0 con 0x563da511b8d0 2024-04-22T06:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8900c2c40 con 0x563da511b8d0 2024-04-22T06:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd888ff9640 1 -- 172.21.15.79:0/566846294 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fd8900c7500 con 0x563da511b8d0 2024-04-22T06:55:09.073 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 10 client.17529 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x563da4e5e620 con 0x563da511b8d0 2024-04-22T06:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd888ff9640 1 -- 172.21.15.79:0/566846294 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8900c2c40 con 0x563da511b8d0 2024-04-22T06:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd888ff9640 1 -- 172.21.15.79:0/566846294 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fd8900c7070 con 0x563da511b8d0 2024-04-22T06:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 10 client.17529 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 10 client.17529 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:09.074 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 20 client.17529 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:09.075 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da5133ba0 0x563da5135f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.075 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 4 client.17529 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:09.075 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.058+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563da5093880 con 0x563da5133ba0 2024-04-22T06:55:09.075 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.059+0000 7fd8a5134640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da5133ba0 0x563da5135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.059+0000 7fd888ff9640 10 client.17529 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.059+0000 7fd8a5134640 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da5133ba0 0x563da5135f90 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a85fa4ab9cf178a server_cookie=14719c090bb00c39 in_seq=0 out_seq=0 2024-04-22T06:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd888ff9640 1 -- 172.21.15.79:0/566846294 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563da5093880 con 0x563da5133ba0 2024-04-22T06:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd888ff9640 10 client.17529 handle_command_reply: tid=0 2024-04-22T06:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 client.17529 shutdown 2024-04-22T06:55:09.076 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 msgr2=0x563da5120d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da5120d30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd8900be590 tx=0x7fd890005b70 comp rx=0 tx=0).stop 2024-04-22T06:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 shutdown_connections 2024-04-22T06:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da5133ba0 0x563da5135f90 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da5121270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd888ff9640 0 client.17529 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:09.077 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511b8d0 0x563da5120d30 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/566846294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511ab60 0x563da51239a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 >> 172.21.15.79:0/566846294 conn(0x563da50e1c90 msgr2=0x563da51175a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 shutdown_connections 2024-04-22T06:55:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 1 -- 172.21.15.79:0/566846294 wait complete. 2024-04-22T06:55:09.078 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.060+0000 7fd8a9f81740 20 client.17529 trim_cache size 0 max 16384 2024-04-22T06:55:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 Processor -- start 2024-04-22T06:55:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 -- start start 2024-04-22T06:55:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da5123460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da511b8d0 0x563da51239a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.091 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da51240e0 0x563da513b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01b0c60 con 0x563da511ab60 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01ae530 con 0x563da511b8d0 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01ac380 con 0x563da51240e0 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da5123460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da5123460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40582/0 (socket says 172.21.15.79:40582) 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a5134640 1 -- 172.21.15.79:0/3347796093 learned_addr learned my addr 172.21.15.79:0/3347796093 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.075+0000 7fd8a4933640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da511b8d0 0x563da51239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5935640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da51240e0 0x563da513b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5134640 1 -- 172.21.15.79:0/3347796093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da51240e0 msgr2=0x563da513b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5134640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da51240e0 0x563da513b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5134640 1 -- 172.21.15.79:0/3347796093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da511b8d0 msgr2=0x563da51239a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.092 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5134640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da511b8d0 0x563da51239a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5134640 1 -- 172.21.15.79:0/3347796093 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da5136570 con 0x563da511ab60 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a4933640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da511b8d0 0x563da51239a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5935640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da51240e0 0x563da513b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a5134640 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da5123460 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x563da5136770 tx=0x7fd89c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3563905861a5bd25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd88affd640 1 -- 172.21.15.79:0/3347796093 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c003b20 con 0x563da511ab60 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd88affd640 1 -- 172.21.15.79:0/3347796093 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89c0054d0 con 0x563da511ab60 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd88affd640 1 -- 172.21.15.79:0/3347796093 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c004c30 con 0x563da511ab60 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a9f81740 1 -- 172.21.15.79:0/3347796093 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 msgr2=0x563da5123460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.076+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da5123460 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x563da5136770 tx=0x7fd89c005f90 comp rx=0 tx=0).stop 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- 172.21.15.79:0/3347796093 shutdown_connections 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da511b8d0 0x563da51239a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da51240e0 0x563da513b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/3347796093 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da5123460 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.093 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- 172.21.15.79:0/3347796093 >> 172.21.15.79:0/3347796093 conn(0x563da50e1c90 msgr2=0x563da511bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- 172.21.15.79:0/3347796093 shutdown_connections 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- 172.21.15.79:0/3347796093 wait complete. 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- start start 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 Processor -- start 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da512c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511b8d0 0x563da5125760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da511dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01ac9f0 con 0x563da511ab60 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01ade60 con 0x563da51240e0 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a01ad0c0 con 0x563da511b8d0 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a9f81740 10 client.0 _ll_register_callbacks cb 0x7fd8900be660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da512c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a5134640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da512c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40592/0 (socket says 172.21.15.79:40592) 2024-04-22T06:55:09.094 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.077+0000 7fd8a5134640 1 -- 172.21.15.79:0/2588364560 learned_addr learned my addr 172.21.15.79:0/2588364560 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a4933640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511b8d0 0x563da5125760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5935640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da511dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5134640 1 -- 172.21.15.79:0/2588364560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511b8d0 msgr2=0x563da5125760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5134640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511b8d0 0x563da5125760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5134640 1 -- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 msgr2=0x563da511dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5134640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da511dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5134640 1 -- 172.21.15.79:0/2588364560 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da5119e30 con 0x563da511ab60 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5935640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da511dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a5134640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da512c510 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd89c004650 tx=0x7fd89c005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367ea06a0cb157d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd888ff9640 1 -- 172.21.15.79:0/2588364560 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c030020 con 0x563da511ab60 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd888ff9640 1 -- 172.21.15.79:0/2588364560 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9673+0+0 (secure 0 0 0) 0x7fd89c003b20 con 0x563da511ab60 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd89c0054d0 con 0x563da511ab60 2024-04-22T06:55:09.095 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd888ff9640 1 -- 172.21.15.79:0/2588364560 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fd89c03b050 con 0x563da511ab60 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a9f81740 10 client.17535 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x563da5124900 con 0x563da511ab60 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.078+0000 7fd888ff9640 1 -- 172.21.15.79:0/2588364560 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd89c0054d0 con 0x563da511ab60 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd888ff9640 1 -- 172.21.15.79:0/2588364560 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fd89c02c580 con 0x563da511ab60 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a9f81740 10 client.17535 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a9f81740 10 client.17535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a9f81740 20 client.17535 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da51250d0 0x563da519c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a9f81740 4 client.17535 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563da501f3d0 con 0x563da51250d0 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a4933640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da51250d0 0x563da519c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd888ff9640 10 client.17535 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.079+0000 7fd8a4933640 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da51250d0 0x563da519c080 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e48855b1d352223 server_cookie=1360b11c94951d44 in_seq=0 out_seq=0 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.086+0000 7fd888ff9640 1 -- 172.21.15.79:0/2588364560 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563da501f3d0 con 0x563da51250d0 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.086+0000 7fd888ff9640 10 client.17535 handle_command_reply: tid=0 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.086+0000 7fd8a9f81740 1 client.17535 shutdown 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 msgr2=0x563da512c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da512c510 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd89c004650 tx=0x7fd89c005700 comp rx=0 tx=0).stop 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 shutdown_connections 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x563da51250d0 0x563da519c080 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563da51240e0 0x563da511dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563da511b8d0 0x563da5125760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 --2- 172.21.15.79:0/2588364560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x563da511ab60 0x563da512c510 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd888ff9640 0 client.17535 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 >> 172.21.15.79:0/2588364560 conn(0x563da50e1c90 msgr2=0x563da511f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 shutdown_connections 2024-04-22T06:55:09.097 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 1 -- 172.21.15.79:0/2588364560 wait complete. 2024-04-22T06:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a9f81740 20 client.17535 trim_cache size 0 max 16384 2024-04-22T06:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:55:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd88c0a3dc0 msgr2=0x7fd88c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd88c0a3dc0 0x7fd88c0a6280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd898013c30 tx=0x7fd89800a9e0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 msgr2=0x7fd8a019e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 0x7fd8a019e950 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fd890009230 tx=0x7fd89002c710 comp rx=0 tx=0).stop 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 shutdown_connections 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fd88c0a3dc0 0x7fd88c0a6280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8a0105e80 0x7fd8a01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8a0101a60 0x7fd8a019e950 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 --2- 172.21.15.79:0/1956879714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8a01010a0 0x7fd8a019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 >> 172.21.15.79:0/1956879714 conn(0x7fd8a0077cd0 msgr2=0x7fd8a00fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 shutdown_connections 2024-04-22T06:55:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:09.087+0000 7fd8a7398640 1 -- 172.21.15.79:0/1956879714 wait complete. 2024-04-22T06:55:10.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:10 smithi193 ceph-mon[31815]: pgmap v348: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:10.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:10 smithi193 ceph-mon[31815]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:55:10.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:10 smithi190 ceph-mon[31421]: pgmap v348: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:10.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:10 smithi190 ceph-mon[31421]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:55:10.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:10 smithi079 ceph-mon[27946]: pgmap v348: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-22T06:55:10.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:10 smithi079 ceph-mon[27946]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:55:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:12 smithi193 ceph-mon[31815]: pgmap v350: 97 pgs: 97 active+clean; 35 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:55:12.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:12 smithi190 ceph-mon[31421]: pgmap v350: 97 pgs: 97 active+clean; 35 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:55:12.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:12 smithi079 ceph-mon[27946]: pgmap v350: 97 pgs: 97 active+clean; 35 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:55:14.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:14 smithi193 ceph-mon[31815]: pgmap v351: 97 pgs: 97 active+clean; 31 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:14.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:14 smithi190 ceph-mon[31421]: pgmap v351: 97 pgs: 97 active+clean; 31 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:14.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:14 smithi079 ceph-mon[27946]: pgmap v351: 97 pgs: 97 active+clean; 31 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:16 smithi193 ceph-mon[31815]: pgmap v352: 97 pgs: 97 active+clean; 30 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:16.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:16 smithi190 ceph-mon[31421]: pgmap v352: 97 pgs: 97 active+clean; 30 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:16 smithi079 ceph-mon[27946]: pgmap v352: 97 pgs: 97 active+clean; 30 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:18.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:18 smithi193 ceph-mon[31815]: pgmap v353: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:18.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:18 smithi190 ceph-mon[31421]: pgmap v353: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:18.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:18 smithi079 ceph-mon[27946]: pgmap v353: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:20 smithi193 ceph-mon[31815]: pgmap v354: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:20 smithi193 ceph-mon[31815]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:55:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:20 smithi190 ceph-mon[31421]: pgmap v354: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:20 smithi190 ceph-mon[31421]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:55:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:20 smithi079 ceph-mon[27946]: pgmap v354: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:20.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:20 smithi079 ceph-mon[27946]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:55:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3635135723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c101aa0 msgr2=0x7fc69c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3635135723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c101aa0 0x7fc69c105a90 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc688009900 tx=0x7fc68802bc50 comp rx=0 tx=0).stop 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3635135723 shutdown_connections 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3635135723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c106170 0x7fc69c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3635135723 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c101aa0 0x7fc69c105a90 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3635135723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc69c1010f0 0x7fc69c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3635135723 >> 172.21.15.79:0/3635135723 conn(0x7fc69c077fd0 msgr2=0x7fc69c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3635135723 shutdown_connections 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3635135723 wait complete. 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 Processor -- start 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.184+0000 7fc6a2e3a640 1 -- start start 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc69c1010f0 0x7fc69c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c106170 0x7fc69c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 0x7fc69c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c111090 con 0x7fc69c19cf40 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c110f10 con 0x7fc69c1010f0 2024-04-22T06:55:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c111210 con 0x7fc69c106170 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 0x7fc69c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 0x7fc69c19d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44514/0 (socket says 172.21.15.79:44514) 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 -- 172.21.15.79:0/3435429714 learned_addr learned my addr 172.21.15.79:0/3435429714 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc69bfff640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c106170 0x7fc69c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc69c1010f0 0x7fc69c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 -- 172.21.15.79:0/3435429714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c106170 msgr2=0x7fc69c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c106170 0x7fc69c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 -- 172.21.15.79:0/3435429714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc69c1010f0 msgr2=0x7fc69c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc69c1010f0 0x7fc69c198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 -- 172.21.15.79:0/3435429714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c19da30 con 0x7fc69c19cf40 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 0x7fc69c19d3f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc69400cce0 tx=0x7fc69400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc699ffb640 1 -- 172.21.15.79:0/3435429714 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69400f450 con 0x7fc69c19cf40 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc699ffb640 1 -- 172.21.15.79:0/3435429714 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc69400faa0 con 0x7fc69c19cf40 2024-04-22T06:55:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.185+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c19dd20 con 0x7fc69c19cf40 2024-04-22T06:55:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.186+0000 7fc699ffb640 1 -- 172.21.15.79:0/3435429714 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694018b70 con 0x7fc69c19cf40 2024-04-22T06:55:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.186+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1a5490 con 0x7fc69c19cf40 2024-04-22T06:55:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.188+0000 7fc699ffb640 1 -- 172.21.15.79:0/3435429714 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc694018d10 con 0x7fc69c19cf40 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.188+0000 7fc699ffb640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc68c0a3dc0 0x7fc68c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.188+0000 7fc6a5a21740 1 Processor -- start 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.188+0000 7fc6a5a21740 1 -- start start 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.188+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc68c0a3dc0 0x7fc68c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.188+0000 7fc699ffb640 1 -- 172.21.15.79:0/3435429714 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7fc6940d10a0 con 0x7fc69c19cf40 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf2928d0 0x5585bf291620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 0x5585bf291fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf29d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5585bf1f68d0 con 0x5585bf291b60 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5585bf193e80 con 0x5585bf2928d0 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585bf1f2890 con 0x5585bf29b0e0 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 0x5585bf291fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 0x5585bf291fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44530/0 (socket says 172.21.15.79:44530) 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 -- 172.21.15.79:0/432183143 learned_addr learned my addr 172.21.15.79:0/432183143 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf29d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc68c0a3dc0 0x7fc68c0a6280 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc69000ba50 tx=0x7fc69000be50 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf2928d0 0x5585bf291620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 -- 172.21.15.79:0/432183143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 msgr2=0x5585bf29d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf29d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 -- 172.21.15.79:0/432183143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf2928d0 msgr2=0x5585bf291620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf2928d0 0x5585bf291620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 -- 172.21.15.79:0/432183143 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585bf1b5320 con 0x5585bf291b60 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf29d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc69bfff640 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 0x5585bf291fe0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc69c06b730 tx=0x7fc6880040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3702c61854cdb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc686ffd640 1 -- 172.21.15.79:0/432183143 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688045020 con 0x5585bf291b60 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc686ffd640 1 -- 172.21.15.79:0/432183143 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc68802ccb0 con 0x5585bf291b60 2024-04-22T06:55:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.189+0000 7fc686ffd640 1 -- 172.21.15.79:0/432183143 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6880048c0 con 0x5585bf291b60 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 -- 172.21.15.79:0/432183143 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 msgr2=0x5585bf291fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 0x5585bf291fe0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc69c06b730 tx=0x7fc6880040f0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 -- 172.21.15.79:0/432183143 shutdown_connections 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf29d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf291b60 0x5585bf291fe0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/432183143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf2928d0 0x5585bf291620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 -- 172.21.15.79:0/432183143 >> 172.21.15.79:0/432183143 conn(0x5585bf258c90 msgr2=0x5585bf28c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 -- 172.21.15.79:0/432183143 shutdown_connections 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 -- 172.21.15.79:0/432183143 wait complete. 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.190+0000 7fc6a5a21740 1 -- start start 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 Processor -- start 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf29a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf297d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf298260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5585bf29ddb0 con 0x5585bf2928d0 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5585bf17eeb0 con 0x5585bf291b60 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585bf29df30 con 0x5585bf29b0e0 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a5a21740 10 client.0 _ll_register_callbacks cb 0x5585bf1abb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf297d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf297d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44532/0 (socket says 172.21.15.79:44532) 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 -- 172.21.15.79:0/2605604660 learned_addr learned my addr 172.21.15.79:0/2605604660 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf29a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 -- 172.21.15.79:0/2605604660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 msgr2=0x5585bf298260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf298260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 -- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 msgr2=0x5585bf29a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf29a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.191+0000 7fc69bfff640 1 -- 172.21.15.79:0/2605604660 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585bef09210 con 0x5585bf2928d0 2024-04-22T06:55:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc69bfff640 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf297d20 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc688009900 tx=0x7fc68802cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f682a5b09c910583 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc67bfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc67bfff640 1 -- 172.21.15.79:0/2605604660 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688003b40 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc67bfff640 1 -- 172.21.15.79:0/2605604660 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7fc688003ce0 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc68802ccb0 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc67bfff640 1 -- 172.21.15.79:0/2605604660 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fc68805b050 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 10 client.17547 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x5585befd5620 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc67bfff640 1 -- 172.21.15.79:0/2605604660 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc68802ccb0 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc67bfff640 1 -- 172.21.15.79:0/2605604660 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fc68804a450 con 0x5585bf2928d0 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 10 client.17547 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 10 client.17547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 20 client.17547 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf2aad90 0x5585bf2ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 4 client.17547 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.192+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585bf20a880 con 0x5585bf2aad90 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.193+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf2aad90 0x5585bf2ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.193+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf2aad90 0x5585bf2ad180 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff841405f270b3f7 server_cookie=2e54f0d15bc983e2 in_seq=0 out_seq=0 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.193+0000 7fc67bfff640 10 client.17547 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc67bfff640 1 -- 172.21.15.79:0/2605604660 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585bf20a880 con 0x5585bf2aad90 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc67bfff640 10 client.17547 handle_command_reply: tid=0 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 client.17547 shutdown 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 msgr2=0x5585bf297d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf297d20 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc688009900 tx=0x7fc68802cab0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 shutdown_connections 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf2aad90 0x5585bf2ad180 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf29b0e0 0x5585bf298260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf297d20 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc67bfff640 0 client.17547 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2605604660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf29a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 >> 172.21.15.79:0/2605604660 conn(0x5585bf258c90 msgr2=0x5585bf28e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 shutdown_connections 2024-04-22T06:55:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2605604660 wait complete. 2024-04-22T06:55:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.194+0000 7fc6a5a21740 20 client.17547 trim_cache size 0 max 16384 2024-04-22T06:55:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 Processor -- start 2024-04-22T06:55:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 -- start start 2024-04-22T06:55:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf2a6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf2928d0 0x5585bf296d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 0x5585bf29a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1b0cb0 con 0x5585bf29b0e0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1ae580 con 0x5585bf291b60 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1ac3f0 con 0x5585bf2928d0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a13d7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 0x5585bf29a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a13d7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 0x5585bf29a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44542/0 (socket says 172.21.15.79:44542) 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a13d7640 1 -- 172.21.15.79:0/2932096961 learned_addr learned my addr 172.21.15.79:0/2932096961 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.209+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf2a6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc69bfff640 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf2928d0 0x5585bf296d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6a13d7640 1 -- 172.21.15.79:0/2932096961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf2928d0 msgr2=0x5585bf296d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf2928d0 0x5585bf296d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6a13d7640 1 -- 172.21.15.79:0/2932096961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 msgr2=0x5585bf2a6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf2a6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6a13d7640 1 -- 172.21.15.79:0/2932096961 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585bf29df20 con 0x5585bf29b0e0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 0x5585bf29a450 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc694098ff0 tx=0x7fc694098f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74357996f4da80b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6867fc640 1 -- 172.21.15.79:0/2932096961 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940a0f40 con 0x5585bf29b0e0 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6867fc640 1 -- 172.21.15.79:0/2932096961 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6940b1070 con 0x5585bf29b0e0 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.210+0000 7fc6867fc640 1 -- 172.21.15.79:0/2932096961 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940bd070 con 0x5585bf29b0e0 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2932096961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 msgr2=0x5585bf29a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 0x5585bf29a450 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc694098ff0 tx=0x7fc694098f60 comp rx=0 tx=0).stop 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2932096961 shutdown_connections 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf29b0e0 0x5585bf29a450 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf2928d0 0x5585bf296d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/2932096961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf291b60 0x5585bf2a6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2932096961 >> 172.21.15.79:0/2932096961 conn(0x5585bf258c90 msgr2=0x5585bf2965f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2932096961 shutdown_connections 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- 172.21.15.79:0/2932096961 wait complete. 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- start start 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 Processor -- start 2024-04-22T06:55:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf291b60 0x5585bf2a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf2a3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf29b0e0 0x5585bf294090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1aca40 con 0x5585bf2928d0 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1adeb0 con 0x5585bf29b0e0 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1ad110 con 0x5585bf291b60 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.211+0000 7fc6a5a21740 10 client.0 _ll_register_callbacks cb 0x7fc6880056a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf2a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf2a3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44544/0 (socket says 172.21.15.79:44544) 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 -- 172.21.15.79:0/3413216833 learned_addr learned my addr 172.21.15.79:0/3413216833 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf291b60 0x5585bf2a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc6a13d7640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf29b0e0 0x5585bf294090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 -- 172.21.15.79:0/3413216833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf291b60 msgr2=0x5585bf2a31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf291b60 0x5585bf2a31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 -- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf29b0e0 msgr2=0x5585bf294090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf29b0e0 0x5585bf294090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 -- 172.21.15.79:0/3413216833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585bf290e30 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc69bfff640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf2a3700 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc68804aa10 tx=0x7fc688005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976c39e2a6ea4b7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc67bfff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc67bfff640 1 -- 172.21.15.79:0/3413216833 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68804ae60 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc67bfff640 1 -- 172.21.15.79:0/3413216833 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7fc68804aa90 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6940b1070 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc67bfff640 1 -- 172.21.15.79:0/3413216833 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7fc688068050 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc6a5a21740 10 client.17553 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x5585bf310870 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.212+0000 7fc67bfff640 1 -- 172.21.15.79:0/3413216833 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6940b1070 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc67bfff640 1 -- 172.21.15.79:0/3413216833 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7fc688041480 con 0x5585bf2928d0 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a5a21740 10 client.17553 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a5a21740 10 client.17553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a5a21740 20 client.17553 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf311020 0x5585bf313440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a5a21740 4 client.17553 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585bf1963d0 con 0x5585bf311020 2024-04-22T06:55:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf311020 0x5585bf313440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc67bfff640 10 client.17553 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:22.217 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.213+0000 7fc6a0bd6640 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf311020 0x5585bf313440 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0bc0c0c5088a397 server_cookie=91494108fb76c82c in_seq=0 out_seq=0 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc67bfff640 1 -- 172.21.15.79:0/3413216833 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5585bf1963d0 con 0x5585bf311020 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc67bfff640 10 client.17553 handle_command_reply: tid=0 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 client.17553 shutdown 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 msgr2=0x5585bf2a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf2a3700 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc68804aa10 tx=0x7fc688005410 comp rx=0 tx=0).stop 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 shutdown_connections 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5585bf311020 0x5585bf313440 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585bf29b0e0 0x5585bf294090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5585bf2928d0 0x5585bf2a3700 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc67bfff640 0 client.17553 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 --2- 172.21.15.79:0/3413216833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf291b60 0x5585bf2a31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 >> 172.21.15.79:0/3413216833 conn(0x5585bf258c90 msgr2=0x5585bf28b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 shutdown_connections 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.220+0000 7fc6a5a21740 1 -- 172.21.15.79:0/3413216833 wait complete. 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a5a21740 20 client.17553 trim_cache size 0 max 16384 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:55:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:55:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc68c0a3dc0 msgr2=0x7fc68c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc68c0a3dc0 0x7fc68c0a6280 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc69000ba50 tx=0x7fc69000be50 comp rx=0 tx=0).stop 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 msgr2=0x7fc69c19d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 0x7fc69c19d3f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc69400cce0 tx=0x7fc69400a620 comp rx=0 tx=0).stop 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 shutdown_connections 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc69c19cf40 0x7fc69c19d3f0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc69c106170 0x7fc69c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc68c0a3dc0 0x7fc68c0a6280 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 --2- 172.21.15.79:0/3435429714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc69c1010f0 0x7fc69c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 >> 172.21.15.79:0/3435429714 conn(0x7fc69c077fd0 msgr2=0x7fc69c069910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 shutdown_connections 2024-04-22T06:55:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:22.221+0000 7fc6a2e3a640 1 -- 172.21.15.79:0/3435429714 wait complete. 2024-04-22T06:55:22.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:22 smithi193 ceph-mon[31815]: pgmap v356: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:22.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:22 smithi190 ceph-mon[31421]: pgmap v356: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:22.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:22 smithi079 ceph-mon[27946]: pgmap v356: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:24.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:24 smithi193 ceph-mon[31815]: pgmap v357: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:24.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:24 smithi190 ceph-mon[31421]: pgmap v357: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:24.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:24 smithi079 ceph-mon[27946]: pgmap v357: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:26.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:26 smithi193 ceph-mon[31815]: pgmap v358: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:26.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:26 smithi190 ceph-mon[31421]: pgmap v358: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:26.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:26 smithi079 ceph-mon[27946]: pgmap v358: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:55:28.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:28 smithi193 ceph-mon[31815]: pgmap v359: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:55:28.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:28 smithi190 ceph-mon[31421]: pgmap v359: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:55:28.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:28 smithi079 ceph-mon[27946]: pgmap v359: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:55:30.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:30 smithi193 ceph-mon[31815]: pgmap v360: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:55:30.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:30 smithi190 ceph-mon[31421]: pgmap v360: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:55:30.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:30 smithi079 ceph-mon[27946]: pgmap v360: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:55:32.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 -- 172.21.15.79:0/2364131271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0101ad0 msgr2=0x7ffbd010dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 --2- 172.21.15.79:0/2364131271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0101ad0 0x7ffbd010dca0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009900 tx=0x7ffbc802bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 -- 172.21.15.79:0/2364131271 shutdown_connections 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 --2- 172.21.15.79:0/2364131271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffbd0101130 0x7ffbd0101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 --2- 172.21.15.79:0/2364131271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0101ad0 0x7ffbd010dca0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 --2- 172.21.15.79:0/2364131271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd0106f00 0x7ffbd01072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 -- 172.21.15.79:0/2364131271 >> 172.21.15.79:0/2364131271 conn(0x7ffbd0077e30 msgr2=0x7ffbd0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:32.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 -- 172.21.15.79:0/2364131271 shutdown_connections 2024-04-22T06:55:32.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.632+0000 7ffbd5248640 1 -- 172.21.15.79:0/2364131271 wait complete. 2024-04-22T06:55:32.632 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 Processor -- start 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 -- start start 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffbd0101130 0x7ffbd01a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd0101ad0 0x7ffbd01a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 0x7ffbd019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01110e0 con 0x7ffbd0106f00 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd0110f60 con 0x7ffbd0101130 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.633+0000 7ffbd5248640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd0111260 con 0x7ffbd0101ad0 2024-04-22T06:55:32.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 0x7ffbd019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 0x7ffbd019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48188/0 (socket says 172.21.15.79:48188) 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 -- 172.21.15.79:0/1006079871 learned_addr learned my addr 172.21.15.79:0/1006079871 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbce59c640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd0101ad0 0x7ffbd01a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbced9d640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffbd0101130 0x7ffbd01a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 -- 172.21.15.79:0/1006079871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd0101ad0 msgr2=0x7ffbd01a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd0101ad0 0x7ffbd01a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 -- 172.21.15.79:0/1006079871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffbd0101130 msgr2=0x7ffbd01a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffbd0101130 0x7ffbd01a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 -- 172.21.15.79:0/1006079871 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd019afe0 con 0x7ffbd0106f00 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbcf59e640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 0x7ffbd019a720 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009230 tx=0x7ffbc80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8043020 con 0x7ffbd0106f00 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbc802cb70 con 0x7ffbd0106f00 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd019b270 con 0x7ffbd0106f00 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8003ac0 con 0x7ffbd0106f00 2024-04-22T06:55:32.635 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.634+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd01a75e0 con 0x7ffbd0106f00 2024-04-22T06:55:32.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.635+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd019b5e0 con 0x7ffbd0106f00 2024-04-22T06:55:32.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.636+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffbc8046020 con 0x7ffbd0106f00 2024-04-22T06:55:32.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.637+0000 7ffbb7fff640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffbb80a3da0 0x7ffbb80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.637+0000 7ffbced9d640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffbb80a3da0 0x7ffbb80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.637+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7ffbc80f2e90 con 0x7ffbd0106f00 2024-04-22T06:55:32.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.637+0000 7ffbced9d640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffbb80a3da0 0x7ffbb80a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000a2e0 tx=0x7ffbc000a270 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:32.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.642+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffbc804d020 con 0x7ffbd0106f00 2024-04-22T06:55:32.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:32 smithi193 ceph-mon[31815]: pgmap v361: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:55:32.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:32 smithi190 ceph-mon[31421]: pgmap v361: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:55:32.743 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:32 smithi079 ceph-mon[27946]: pgmap v361: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 344 B/s wr, 7 op/s 2024-04-22T06:55:32.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.860+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbd01a77b0 con 0x7ffbd0106f00 2024-04-22T06:55:32.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.860+0000 7ffbb7fff640 1 -- 172.21.15.79:0/1006079871 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+11883 (secure 0 0 0) 0x7ffbc80bf130 con 0x7ffbd0106f00 2024-04-22T06:55:32.859 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:55:32.859 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:54:15.404622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":26003,"mds_2":26000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26000":{"gid":26000,"name":"e","rank":2,"incarnation":73,"state":"up:active","state_seq":21,"addr":"172.21.15.190:6835/1769517674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1769517674},{"type":"v1","addr":"172.21.15.190:6835","nonce":1769517674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26003":{"gid":26003,"name":"f","rank":1,"incarnation":75,"state":"up:active","state_seq":21,"addr":"172.21.15.193:6835/2674386425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2674386425},{"type":"v1","addr":"172.21.15.193:6835","nonce":2674386425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26095":{"gid":26095,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/3177981466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3177981466},{"type":"v1","addr":"172.21.15.193:6839","nonce":3177981466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26096":{"gid":26096,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/246451352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":246451352},{"type":"v1","addr":"172.21.15.190:6837","nonce":246451352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26113":{"gid":26113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/4272953642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":4272953642},{"type":"v1","addr":"172.21.15.193:6837","nonce":4272953642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[26003,24467,26000]},"id":1}]} 2024-04-22T06:55:32.859 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 82 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffbb80a3da0 msgr2=0x7ffbb80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffbb80a3da0 0x7ffbb80a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000a2e0 tx=0x7ffbc000a270 comp rx=0 tx=0).stop 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 msgr2=0x7ffbd019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 0x7ffbd019a720 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009230 tx=0x7ffbc80045e0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 shutdown_connections 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffbd0106f00 0x7ffbd019a720 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd0101ad0 0x7ffbd01a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ffbb80a3da0 0x7ffbb80a6260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 --2- 172.21.15.79:0/1006079871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffbd0101130 0x7ffbd01a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.862+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 >> 172.21.15.79:0/1006079871 conn(0x7ffbd0077e30 msgr2=0x7ffbd006a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.863+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 shutdown_connections 2024-04-22T06:55:32.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.863+0000 7ffbd5248640 1 -- 172.21.15.79:0/1006079871 wait complete. 2024-04-22T06:55:32.872 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --set-id 13ba45d6 --release --await' 2024-04-22T06:55:32.872 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --set-id 13ba45d6 --release --await 2024-04-22T06:55:32.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 -- 172.21.15.79:0/2046320463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 msgr2=0x7f40a010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 --2- 172.21.15.79:0/2046320463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a010dd30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4098009ca0 tx=0x7f409802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 -- 172.21.15.79:0/2046320463 shutdown_connections 2024-04-22T06:55:32.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 --2- 172.21.15.79:0/2046320463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40a0101a40 0x7f40a0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 --2- 172.21.15.79:0/2046320463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a010dd30 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 --2- 172.21.15.79:0/2046320463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40a0101090 0x7f40a0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 -- 172.21.15.79:0/2046320463 >> 172.21.15.79:0/2046320463 conn(0x7f40a0077f50 msgr2=0x7f40a0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 -- 172.21.15.79:0/2046320463 shutdown_connections 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 -- 172.21.15.79:0/2046320463 wait complete. 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 Processor -- start 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 -- start start 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40a0101090 0x7f40a0197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.979+0000 7f40a704a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a0197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a704a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40a019bdd0 0x7f40a019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a704a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01110b0 con 0x7f40a0106160 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a704a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40a0110f30 con 0x7f40a0101090 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a704a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40a0111230 con 0x7f40a019bdd0 2024-04-22T06:55:32.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a0197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a0197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48198/0 (socket says 172.21.15.79:48198) 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 -- 172.21.15.79:0/484843058 learned_addr learned my addr 172.21.15.79:0/484843058 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a4de6640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40a0101090 0x7f40a0197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a55e7640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40a019bdd0 0x7f40a019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 -- 172.21.15.79:0/484843058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40a019bdd0 msgr2=0x7f40a019c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40a019bdd0 0x7f40a019c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 -- 172.21.15.79:0/484843058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40a0101090 msgr2=0x7f40a0197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40a0101090 0x7f40a0197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 -- 172.21.15.79:0/484843058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a019c8c0 con 0x7f40a0106160 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409ffff640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a0197a50 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c970 tx=0x7f408c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409dffb640 1 -- 172.21.15.79:0/484843058 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c014070 con 0x7f40a0106160 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409dffb640 1 -- 172.21.15.79:0/484843058 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f408c004440 con 0x7f40a0106160 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f409dffb640 1 -- 172.21.15.79:0/484843058 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c003d30 con 0x7f40a0106160 2024-04-22T06:55:32.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a019cbb0 con 0x7f40a0106160 2024-04-22T06:55:32.982 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.980+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a01a43a0 con 0x7f40a0106160 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.982+0000 7f409dffb640 1 -- 172.21.15.79:0/484843058 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f408c017070 con 0x7f40a0106160 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 Processor -- start 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f409dffb640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f40900a3d20 0x7f40900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 -- start start 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a55e7640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f40900a3d20 0x7f40900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.988 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f409dffb640 1 -- 172.21.15.79:0/484843058 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f408c0d0a60 con 0x7f40a0106160 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 0x55d9070cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d907025500 con 0x55d9070cba20 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d906fcb6a0 con 0x55d9070c2520 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d9070214c0 con 0x55d9070c2e40 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a55e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 0x55d9070cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a55e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 0x55d9070cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48204/0 (socket says 172.21.15.79:48204) 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a55e7640 1 -- 172.21.15.79:0/2402821558 learned_addr learned my addr 172.21.15.79:0/2402821558 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f409ffff640 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a4de6640 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.983+0000 7f40a55e7640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f40900a3d20 0x7f40900a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4098037040 tx=0x7f4098032ca0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a55e7640 1 -- 172.21.15.79:0/2402821558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 msgr2=0x55d9070cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a55e7640 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a55e7640 1 -- 172.21.15.79:0/2402821558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 msgr2=0x55d9070c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a55e7640 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070c2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a55e7640 1 -- 172.21.15.79:0/2402821558 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d906fea5e0 con 0x55d9070cba20 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a55e7640 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 0x55d9070cde10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4098009ca0 tx=0x7f4098048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2cde410eb5df2b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f408affd640 1 -- 172.21.15.79:0/2402821558 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098051070 con 0x55d9070cba20 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f408affd640 1 -- 172.21.15.79:0/2402821558 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4098003b70 con 0x55d9070cba20 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f408affd640 1 -- 172.21.15.79:0/2402821558 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098039040 con 0x55d9070cba20 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2402821558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 msgr2=0x55d9070cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.984+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 0x55d9070cde10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4098009ca0 tx=0x7f4098048000 comp rx=0 tx=0).stop 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2402821558 shutdown_connections 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070cba20 0x55d9070cde10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2402821558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2402821558 >> 172.21.15.79:0/2402821558 conn(0x55d9070bcc50 msgr2=0x55d9070bd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2402821558 shutdown_connections 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2402821558 wait complete. 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- start start 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 Processor -- start 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 0x55d9070cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070c8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070cba20 0x55d9070c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55d9070ce7b0 con 0x55d9070c2520 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d906fdea60 con 0x55d9070cba20 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d9070ce930 con 0x55d9070c2e40 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a9c3f740 10 client.0 _ll_register_callbacks cb 0x55d906d37110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a4de6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 0x55d9070cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a4de6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 0x55d9070cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48212/0 (socket says 172.21.15.79:48212) 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.985+0000 7f40a4de6640 1 -- 172.21.15.79:0/4243706410 learned_addr learned my addr 172.21.15.79:0/4243706410 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f409ffff640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a4de6640 1 -- 172.21.15.79:0/4243706410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 msgr2=0x55d9070c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a4de6640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a4de6640 1 -- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070cba20 msgr2=0x55d9070c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a4de6640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070cba20 0x55d9070c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a4de6640 1 -- 172.21.15.79:0/4243706410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d906d37920 con 0x55d9070c2520 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f409ffff640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a4de6640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 0x55d9070cb490 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f4094012840 tx=0x7f4094012d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d62976a7226776 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f407ffff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f407ffff640 1 -- 172.21.15.79:0/4243706410 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409401f060 con 0x55d9070c2520 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f407ffff640 1 -- 172.21.15.79:0/4243706410 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f4094010dc0 con 0x55d9070c2520 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4098003b70 con 0x55d9070c2520 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f407ffff640 1 -- 172.21.15.79:0/4243706410 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f4094026050 con 0x55d9070c2520 2024-04-22T06:55:32.992 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a9c3f740 10 client.17571 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d906e03d30 con 0x55d9070c2520 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f407ffff640 1 -- 172.21.15.79:0/4243706410 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4098003b70 con 0x55d9070c2520 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f407ffff640 1 -- 172.21.15.79:0/4243706410 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f409400d460 con 0x55d9070c2520 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a9c3f740 10 client.17571 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a9c3f740 10 client.17571 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.986+0000 7f40a9c3f740 20 client.17571 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.987+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9070dbb90 0x55d9070de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.987+0000 7f40a9c3f740 4 client.17571 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.987+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d906ff8a10 con 0x55d9070dbb90 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.987+0000 7f409ffff640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9070dbb90 0x55d9070de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.987+0000 7f407ffff640 10 client.17571 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.987+0000 7f409ffff640 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9070dbb90 0x55d9070de000 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7311168f04d842 server_cookie=bc837548c30710a9 in_seq=0 out_seq=0 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f407ffff640 1 -- 172.21.15.79:0/4243706410 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d906ff8a10 con 0x55d9070dbb90 2024-04-22T06:55:32.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f407ffff640 10 client.17571 handle_command_reply: tid=0 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 client.17571 shutdown 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 msgr2=0x55d9070cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 0x55d9070cb490 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f4094012840 tx=0x7f4094012d00 comp rx=0 tx=0).stop 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 shutdown_connections 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9070dbb90 0x55d9070de000 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070cba20 0x55d9070c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070c2e40 0x55d9070c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f407ffff640 0 client.17571 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4243706410 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2520 0x55d9070cb490 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 >> 172.21.15.79:0/4243706410 conn(0x55d9070bcc50 msgr2=0x55d9070be9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 shutdown_connections 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4243706410 wait complete. 2024-04-22T06:55:32.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:32.988+0000 7f40a9c3f740 20 client.17571 trim_cache size 0 max 16384 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 Processor -- start 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 -- start start 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 0x55d9070cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01aec60 con 0x55d9070c2e40 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ac530 con 0x55d9070c2520 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01aa3a0 con 0x55d9070cba20 2024-04-22T06:55:33.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f409ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f409ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48216/0 (socket says 172.21.15.79:48216) 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.005+0000 7f409ffff640 1 -- 172.21.15.79:0/4117841198 learned_addr learned my addr 172.21.15.79:0/4117841198 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f409ffff640 1 -- 172.21.15.79:0/4117841198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 msgr2=0x55d9070cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a4de6640 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f409ffff640 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 0x55d9070cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f409ffff640 1 -- 172.21.15.79:0/4117841198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 msgr2=0x55d9070b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f409ffff640 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070b9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f409ffff640 1 -- 172.21.15.79:0/4117841198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9070c2c70 con 0x55d9070c2e40 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f409ffff640 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070caf50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f408c09c250 tx=0x7f408c0d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c500278a0670ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f408a7fc640 1 -- 172.21.15.79:0/4117841198 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c0a0900 con 0x55d9070c2e40 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f408a7fc640 1 -- 172.21.15.79:0/4117841198 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f408c0a51c0 con 0x55d9070c2e40 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f408a7fc640 1 -- 172.21.15.79:0/4117841198 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c00ec00 con 0x55d9070c2e40 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4117841198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 msgr2=0x55d9070caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070caf50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f408c09c250 tx=0x7f408c0d0040 comp rx=0 tx=0).stop 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4117841198 shutdown_connections 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 0x55d9070cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070caf50 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/4117841198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4117841198 >> 172.21.15.79:0/4117841198 conn(0x55d9070bcc50 msgr2=0x55d9070b9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.006+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4117841198 shutdown_connections 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 -- 172.21.15.79:0/4117841198 wait complete. 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 -- start start 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 Processor -- start 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 0x55d9070ba3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.010 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01aa9f0 con 0x55d9070c2e40 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01abe60 con 0x55d9070c2520 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40a01ab0c0 con 0x55d9070cba20 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a9c3f740 10 client.0 _ll_register_callbacks cb 0x7f409400a6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f409ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f409ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48220/0 (socket says 172.21.15.79:48220) 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f409ffff640 1 -- 172.21.15.79:0/2448883679 learned_addr learned my addr 172.21.15.79:0/2448883679 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f40a4de6640 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f409ffff640 1 -- 172.21.15.79:0/2448883679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 msgr2=0x55d9070ba3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.007+0000 7f409ffff640 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 0x55d9070ba3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f409ffff640 1 -- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 msgr2=0x55d9070d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f409ffff640 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070d3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f409ffff640 1 -- 172.21.15.79:0/2448883679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9070bac10 con 0x55d9070c2e40 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f409ffff640 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070d40c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f408c000c00 tx=0x7f408c00a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df4cd0aedb0468f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f407ffff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:33.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f407ffff640 1 -- 172.21.15.79:0/2448883679 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c0cdf80 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f407ffff640 1 -- 172.21.15.79:0/2448883679 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f408c00ec00 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f408c0a51c0 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f407ffff640 1 -- 172.21.15.79:0/2448883679 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f408c00ee00 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 10 client.17577 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x55d9071451a0 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f407ffff640 1 -- 172.21.15.79:0/2448883679 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f408c0a51c0 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f407ffff640 1 -- 172.21.15.79:0/2448883679 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f408c09b410 con 0x55d9070c2e40 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 10 client.17577 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 10 client.17577 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 20 client.17577 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9071457f0 0x55d907147c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 4 client.17577 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "13ba45d6", "release": true, "await": true}] 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.008+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "set_id": "13ba45d6", "release": true, "await": true}) -- 0x55d907032f20 con 0x55d9071457f0 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.009+0000 7f40a4de6640 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9071457f0 0x55d907147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.009+0000 7f40a4de6640 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9071457f0 0x55d907147c10 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a65e83a974b48357 server_cookie=7fc7aafe1c397462 in_seq=0 out_seq=0 2024-04-22T06:55:33.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.009+0000 7f407ffff640 10 client.17577 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:33.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f407ffff640 1 -- 172.21.15.79:0/2448883679 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d907032f20 con 0x55d9071457f0 2024-04-22T06:55:33.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f407ffff640 10 client.17577 handle_command_reply: tid=0 2024-04-22T06:55:33.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 client.17577 shutdown 2024-04-22T06:55:33.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 msgr2=0x55d9070d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070d40c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f408c000c00 tx=0x7f408c00a5e0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 shutdown_connections 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55d9071457f0 0x55d907147c10 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9070cba20 0x55d9070ba3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55d9070c2e40 0x55d9070d40c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f407ffff640 0 client.17577 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 --2- 172.21.15.79:0/2448883679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9070c2520 0x55d9070d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 >> 172.21.15.79:0/2448883679 conn(0x55d9070bcc50 msgr2=0x55d9070b9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.012+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 shutdown_connections 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a9c3f740 1 -- 172.21.15.79:0/2448883679 wait complete. 2024-04-22T06:55:33.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a9c3f740 20 client.17577 trim_cache size 0 max 16384 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 82, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 36, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "13ba45d6": { 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 36, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:55:33.016 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 90.0, 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": false, 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RELEASED", 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:33.017 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:55:33.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f40900a3d20 msgr2=0x7f40900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f40900a3d20 0x7f40900a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4098037040 tx=0x7f4098032ca0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 msgr2=0x7f40a0197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a0197a50 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c970 tx=0x7f408c00cd70 comp rx=0 tx=0).stop 2024-04-22T06:55:33.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 shutdown_connections 2024-04-22T06:55:33.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40a019bdd0 0x7f40a019c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a0106160 0x7f40a0197a50 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f40900a3d20 0x7f40900a61e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 --2- 172.21.15.79:0/484843058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40a0101090 0x7f40a0197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 >> 172.21.15.79:0/484843058 conn(0x7f40a0077f50 msgr2=0x7f40a00fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 shutdown_connections 2024-04-22T06:55:33.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.013+0000 7f40a704a640 1 -- 172.21.15.79:0/484843058 wait complete. 2024-04-22T06:55:33.022 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 13ba45d6, seconds elapsed: 31.0 2024-04-22T06:55:33.022 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-04-22T06:55:33.022 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:55:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 -- 172.21.15.79:0/648469850 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 msgr2=0x7f7cd410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/648469850 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd410e9b0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009ca0 tx=0x7f7ccc02c180 comp rx=0 tx=0).stop 2024-04-22T06:55:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 -- 172.21.15.79:0/648469850 shutdown_connections 2024-04-22T06:55:33.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/648469850 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd410e9b0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/648469850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cd4103b10 0x7f7cd410bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/648469850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd41031f0 0x7f7cd41035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 -- 172.21.15.79:0/648469850 >> 172.21.15.79:0/648469850 conn(0x7f7cd4077fe0 msgr2=0x7f7cd40783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 -- 172.21.15.79:0/648469850 shutdown_connections 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.123+0000 7f7cd91a4640 1 -- 172.21.15.79:0/648469850 wait complete. 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 Processor -- start 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 -- start start 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cd41031f0 0x7f7cd41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd4103b10 0x7f7cd4198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd41121a0 con 0x7f7cd410c530 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd4112020 con 0x7f7cd4103b10 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd91a4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd4112320 con 0x7f7cd41031f0 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48230/0 (socket says 172.21.15.79:48230) 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 -- 172.21.15.79:0/3503311043 learned_addr learned my addr 172.21.15.79:0/3503311043 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd2d9d640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cd41031f0 0x7f7cd41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7ccbfff640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd4103b10 0x7f7cd4198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 -- 172.21.15.79:0/3503311043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cd41031f0 msgr2=0x7f7cd41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cd41031f0 0x7f7cd41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 -- 172.21.15.79:0/3503311043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd4103b10 msgr2=0x7f7cd4198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd4103b10 0x7f7cd4198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7cd359e640 1 -- 172.21.15.79:0/3503311043 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd419d6b0 con 0x7f7cd410c530 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.124+0000 7f7ccbfff640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd4103b10 0x7f7cd4198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.125+0000 7f7cd359e640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd419cf80 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc02c790 tx=0x7f7ccc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.125+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc039040 con 0x7f7cd410c530 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.125+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd419d940 con 0x7f7cd410c530 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.125+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ccc044070 con 0x7f7cd410c530 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.125+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc032850 con 0x7f7cd410c530 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.125+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd41a5230 con 0x7f7cd410c530 2024-04-22T06:55:33.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.126+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd4069230 con 0x7f7cd410c530 2024-04-22T06:55:33.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.127+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7ccc004240 con 0x7f7cd410c530 2024-04-22T06:55:33.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.128+0000 7f7cd0d99640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7cc00a3d20 0x7f7cc00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.128+0000 7f7cd2d9d640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7cc00a3d20 0x7f7cc00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.128+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f7ccc0f37d0 con 0x7f7cd410c530 2024-04-22T06:55:33.128 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.128+0000 7f7cd2d9d640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7cc00a3d20 0x7f7cc00a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40058c0 tx=0x7f7cc400a2d0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.131+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ccc0bfa30 con 0x7f7cd410c530 2024-04-22T06:55:33.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.352+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cd4107710 con 0x7f7cd410c530 2024-04-22T06:55:33.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.353+0000 7f7cd0d99640 1 -- 172.21.15.79:0/3503311043 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+11883 (secure 0 0 0) 0x7f7ccc0bf330 con 0x7f7cd410c530 2024-04-22T06:55:33.351 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:55:33.351 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:54:15.404622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":26003,"mds_2":26000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26000":{"gid":26000,"name":"e","rank":2,"incarnation":73,"state":"up:active","state_seq":21,"addr":"172.21.15.190:6835/1769517674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1769517674},{"type":"v1","addr":"172.21.15.190:6835","nonce":1769517674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26003":{"gid":26003,"name":"f","rank":1,"incarnation":75,"state":"up:active","state_seq":21,"addr":"172.21.15.193:6835/2674386425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2674386425},{"type":"v1","addr":"172.21.15.193:6835","nonce":2674386425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26095":{"gid":26095,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/3177981466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3177981466},{"type":"v1","addr":"172.21.15.193:6839","nonce":3177981466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26096":{"gid":26096,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6837/246451352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":246451352},{"type":"v1","addr":"172.21.15.190:6837","nonce":246451352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26113":{"gid":26113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/4272953642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":4272953642},{"type":"v1","addr":"172.21.15.193:6837","nonce":4272953642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[26003,24467,26000]},"id":1}]} 2024-04-22T06:55:33.351 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 82 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7cc00a3d20 msgr2=0x7f7cc00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7cc00a3d20 0x7f7cc00a61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40058c0 tx=0x7f7cc400a2d0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 msgr2=0x7f7cd419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd419cf80 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc02c790 tx=0x7f7ccc003c20 comp rx=0 tx=0).stop 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 shutdown_connections 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7cc00a3d20 0x7f7cc00a61e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cd4103b10 0x7f7cd4198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7cd410c530 0x7f7cd419cf80 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 --2- 172.21.15.79:0/3503311043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cd41031f0 0x7f7cd41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 >> 172.21.15.79:0/3503311043 conn(0x7f7cd4077fe0 msgr2=0x7f7cd40fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 shutdown_connections 2024-04-22T06:55:33.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.355+0000 7f7cd91a4640 1 -- 172.21.15.79:0/3503311043 wait complete. 2024-04-22T06:55:33.364 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db / --timeout 60 --expiration 90.0 --await' 2024-04-22T06:55:33.364 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db / --timeout 60 --expiration 90.0 --await 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f425a1c2700 time 2024-04-22T06:55:33.010973+0000 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f426d7fd3dd] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.368 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: clone() 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.011+0000 7f425a1c2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f425a1c2700 time 2024-04-22T06:55:33.010973+0000 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f426d7fd3dd] 2024-04-22T06:55:33.369 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: clone() 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.370 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: *** Caught signal (Aborted) ** 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: in thread 7f425a1c2700 thread_name:quiesce.agt 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f426c585d20] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: gsignal() 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: abort() 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f426d7fd437] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.371 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 13: clone() 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.012+0000 7f425a1c2700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: in thread 7f425a1c2700 thread_name:quiesce.agt 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f426c585d20] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: gsignal() 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: abort() 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f426d7fd437] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.372 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 13: clone() 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: -2> 2024-04-22T06:55:33.011+0000 7f425a1c2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f425a1c2700 time 2024-04-22T06:55:33.010973+0000 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f426d7fd3dd] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.373 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: clone() 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 0> 2024-04-22T06:55:33.012+0000 7f425a1c2700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: in thread 7f425a1c2700 thread_name:quiesce.agt 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f426c585d20] 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: gsignal() 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: abort() 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f426d7fd437] 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.374 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 13: clone() 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: -6> 2024-04-22T06:55:33.011+0000 7f425a1c2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f425a1c2700 time 2024-04-22T06:55:33.010973+0000 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.375 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f426d7fd3dd] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: clone() 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: -4> 2024-04-22T06:55:33.012+0000 7f425a1c2700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: in thread 7f425a1c2700 thread_name:quiesce.agt 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.376 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f426c585d20] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2: gsignal() 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 3: abort() 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f426d7fd437] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f426d7fd5a3] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562060ecfe9c] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562060ecff9f] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562060dfe0d3] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562060dfe03c] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56206107dd3f] 2024-04-22T06:55:33.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5620610e6560] 2024-04-22T06:55:33.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f426c57b1ca] 2024-04-22T06:55:33.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 13: clone() 2024-04-22T06:55:33.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:33.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[68416]: 2024-04-22T06:55:33.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 podman[69399]: 2024-04-22 06:55:33.072896651 +0000 UTC m=+0.022746679 container died 0f7958e8bb8ecb0f035dd7d232f4ffc11100caf23584cf8e39215787bdf723ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416) 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f04d5c25700 time 2024-04-22T06:55:33.010843+0000 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04e92603dd] 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.422 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: clone() 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.010+0000 7f04d5c25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f04d5c25700 time 2024-04-22T06:55:33.010843+0000 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04e92603dd] 2024-04-22T06:55:33.423 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: clone() 2024-04-22T06:55:33.424 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: *** Caught signal (Aborted) ** 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: in thread 7f04d5c25700 thread_name:quiesce.agt 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e7fe8d20] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: gsignal() 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: abort() 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04e9260437] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.425 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 13: clone() 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.012+0000 7f04d5c25700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: in thread 7f04d5c25700 thread_name:quiesce.agt 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e7fe8d20] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: gsignal() 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: abort() 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04e9260437] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.426 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 13: clone() 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: -5> 2024-04-22T06:55:33.010+0000 7f04d5c25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f04d5c25700 time 2024-04-22T06:55:33.010843+0000 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04e92603dd] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.427 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: clone() 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 0> 2024-04-22T06:55:33.012+0000 7f04d5c25700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: in thread 7f04d5c25700 thread_name:quiesce.agt 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e7fe8d20] 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: gsignal() 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: abort() 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04e9260437] 2024-04-22T06:55:33.428 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 13: clone() 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:33.429 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: -6> 2024-04-22T06:55:33.010+0000 7f04d5c25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f04d5c25700 time 2024-04-22T06:55:33.010843+0000 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f04e92603dd] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: clone() 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: -1> 2024-04-22T06:55:33.012+0000 7f04d5c25700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: in thread 7f04d5c25700 thread_name:quiesce.agt 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e7fe8d20] 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2: gsignal() 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 3: abort() 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f04e9260437] 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f04e92605a3] 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x564beecb0e9c] 2024-04-22T06:55:33.431 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x564beecb0f9f] 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x564beebdf0d3] 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x564beebdf03c] 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x564beee5ed3f] 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x564beeec7560] 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f04e7fde1ca] 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 13: clone() 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[67192]: 2024-04-22T06:55:33.432 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 podman[68003]: 2024-04-22 06:55:33.071471014 +0000 UTC m=+0.022149524 container died 69e64f8300348a194a64c242960c1cb0603aa9c811d0bdbe1f67678ebaee382c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:55:33.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 -- 172.21.15.79:0/2484865690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103b50 msgr2=0x7f631010c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 --2- 172.21.15.79:0/2484865690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103b50 0x7f631010c030 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009900 tx=0x7f62f802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 -- 172.21.15.79:0/2484865690 shutdown_connections 2024-04-22T06:55:33.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 --2- 172.21.15.79:0/2484865690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f631010c570 0x7f631010e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.474 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 --2- 172.21.15.79:0/2484865690 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103b50 0x7f631010c030 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 --2- 172.21.15.79:0/2484865690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6310103230 0x7f6310103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 -- 172.21.15.79:0/2484865690 >> 172.21.15.79:0/2484865690 conn(0x7f6310077fb0 msgr2=0x7f63100783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 -- 172.21.15.79:0/2484865690 shutdown_connections 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.474+0000 7f631574a640 1 -- 172.21.15.79:0/2484865690 wait complete. 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 Processor -- start 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 -- start start 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103230 0x7f63101a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6310103b50 0x7f63101a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f631010c570 0x7f631019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6310112230 con 0x7f6310103230 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63101120b0 con 0x7f6310103b50 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f631574a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63101123b0 con 0x7f631010c570 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103230 0x7f63101a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.475 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103230 0x7f63101a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48260/0 (socket says 172.21.15.79:48260) 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 -- 172.21.15.79:0/2082530262 learned_addr learned my addr 172.21.15.79:0/2082530262 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f631010c570 0x7f631019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f6307fff640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6310103b50 0x7f63101a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 -- 172.21.15.79:0/2082530262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f631010c570 msgr2=0x7f631019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f631010c570 0x7f631019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 -- 172.21.15.79:0/2082530262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6310103b50 msgr2=0x7f63101a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6310103b50 0x7f63101a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 -- 172.21.15.79:0/2082530262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631019af60 con 0x7f6310103230 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630effd640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6310103230 0x7f63101a0620 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f630000b4a0 tx=0x7f630000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.475+0000 7f630cff9640 1 -- 172.21.15.79:0/2082530262 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300014070 con 0x7f6310103230 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.476+0000 7f630cff9640 1 -- 172.21.15.79:0/2082530262 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6300010ac0 con 0x7f6310103230 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.476+0000 7f631574a640 1 -- 172.21.15.79:0/2082530262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631019b250 con 0x7f6310103230 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.476+0000 7f630cff9640 1 -- 172.21.15.79:0/2082530262 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300017c30 con 0x7f6310103230 2024-04-22T06:55:33.476 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.476+0000 7f631574a640 1 -- 172.21.15.79:0/2082530262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63101a7470 con 0x7f6310103230 2024-04-22T06:55:33.483 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f630cff9640 1 -- 172.21.15.79:0/2082530262 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6300017070 con 0x7f6310103230 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 Processor -- start 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 -- start start 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e2924cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29255980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e29255ec0 0x562e292582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562e291ad0b0 con 0x562e2924d2e0 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562e2915e680 con 0x562e29255ec0 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f631840e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562e2916ac60 con 0x562e2924c9c0 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f630effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29255980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f630cff9640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f62fc0a3d20 0x7f62fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f630effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29255980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48270/0 (socket says 172.21.15.79:48270) 2024-04-22T06:55:33.484 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f630effd640 1 -- 172.21.15.79:0/2467241483 learned_addr learned my addr 172.21.15.79:0/2467241483 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.478+0000 7f6307fff640 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e29255ec0 0x562e292582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e2924cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f62fc0a3d20 0x7f62fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630cff9640 1 -- 172.21.15.79:0/2082530262 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f63000d0000 con 0x7f6310103230 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630effd640 1 -- 172.21.15.79:0/2467241483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 msgr2=0x562e2924cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630effd640 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e2924cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630effd640 1 -- 172.21.15.79:0/2467241483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e29255ec0 msgr2=0x562e292582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630effd640 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e29255ec0 0x562e292582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630effd640 1 -- 172.21.15.79:0/2467241483 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e28f8fc90 con 0x562e2924d2e0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630effd640 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29255980 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f63000985d0 tx=0x7f630000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7f25c0c31680e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2082530262 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f62fc0a3d20 0x7f62fc0a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6308007310 tx=0x7f6308004a40 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f63057fa640 1 -- 172.21.15.79:0/2467241483 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630001e070 con 0x562e2924d2e0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f63057fa640 1 -- 172.21.15.79:0/2467241483 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6300017360 con 0x562e2924d2e0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.479+0000 7f63057fa640 1 -- 172.21.15.79:0/2467241483 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300010c70 con 0x562e2924d2e0 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 -- 172.21.15.79:0/2467241483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 msgr2=0x562e29255980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.485 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29255980 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f63000985d0 tx=0x7f630000bf90 comp rx=0 tx=0).stop 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 -- 172.21.15.79:0/2467241483 shutdown_connections 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e29255ec0 0x562e292582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29255980 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 --2- 172.21.15.79:0/2467241483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e2924cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 -- 172.21.15.79:0/2467241483 >> 172.21.15.79:0/2467241483 conn(0x562e292470f0 msgr2=0x562e292474d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 -- 172.21.15.79:0/2467241483 shutdown_connections 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 -- 172.21.15.79:0/2467241483 wait complete. 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 -- start start 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.480+0000 7f631840e740 1 Processor -- start 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29252f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29253470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e292539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x562e291b0400 con 0x562e29255ec0 2024-04-22T06:55:33.486 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562e2916c840 con 0x562e2924d2e0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562e291b5720 con 0x562e2924c9c0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 10 client.0 _ll_register_callbacks cb 0x562e291bdea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e292539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e292539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48274/0 (socket says 172.21.15.79:48274) 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 -- 172.21.15.79:0/775427926 learned_addr learned my addr 172.21.15.79:0/775427926 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f6307fff640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29253470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630effd640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29252f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 -- 172.21.15.79:0/775427926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 msgr2=0x562e29252f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29252f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 -- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 msgr2=0x562e29253470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29253470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 -- 172.21.15.79:0/775427926 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e28ff0420 con 0x562e29255ec0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f630f7fe640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e292539b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f630800e6b0 tx=0x7f6308004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b36670d1487f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f62f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:33.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f62f67fc640 1 -- 172.21.15.79:0/775427926 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308013230 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e291beb10 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.481+0000 7f62f67fc640 1 -- 172.21.15.79:0/775427926 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f630801f660 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f62f67fc640 1 -- 172.21.15.79:0/775427926 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f6308030050 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 10 client.17595 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x562e28e83580 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f62f67fc640 1 -- 172.21.15.79:0/775427926 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562e291beb10 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f62f67fc640 1 -- 172.21.15.79:0/775427926 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f6308012590 con 0x562e29255ec0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 10 client.17595 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 10 client.17595 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 20 client.17595 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e29265ca0 0x562e292680d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 4 client.17595 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e291b8980 con 0x562e29265ca0 2024-04-22T06:55:33.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.482+0000 7f630effd640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e29265ca0 0x562e292680d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f62f67fc640 10 client.17595 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f630effd640 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e29265ca0 0x562e292680d0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5138d4288eff671e server_cookie=b568bc188889afda in_seq=0 out_seq=0 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f62f67fc640 1 -- 172.21.15.79:0/775427926 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e291b8980 con 0x562e29265ca0 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f62f67fc640 10 client.17595 handle_command_reply: tid=0 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 client.17595 shutdown 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 msgr2=0x562e292539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e292539b0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f630800e6b0 tx=0x7f6308004ab0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 shutdown_connections 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e29265ca0 0x562e292680d0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29253470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e292539b0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f62f67fc640 0 client.17595 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 --2- 172.21.15.79:0/775427926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29252f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.483+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 >> 172.21.15.79:0/775427926 conn(0x562e292470f0 msgr2=0x562e29248e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.484+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 shutdown_connections 2024-04-22T06:55:33.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.484+0000 7f631840e740 1 -- 172.21.15.79:0/775427926 wait complete. 2024-04-22T06:55:33.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.484+0000 7f631840e740 20 client.17595 trim_cache size 0 max 16384 2024-04-22T06:55:33.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 Processor -- start 2024-04-22T06:55:33.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 -- start start 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29255660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29261870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e2926d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f63101b0a50 con 0x562e29255ec0 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63101ae8c0 con 0x562e2924d2e0 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f631840e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63101b0380 con 0x562e2924c9c0 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e2926d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e2926d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48284/0 (socket says 172.21.15.79:48284) 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 -- 172.21.15.79:0/2809344909 learned_addr learned my addr 172.21.15.79:0/2809344909 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f6307fff640 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29261870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 -- 172.21.15.79:0/2809344909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 msgr2=0x562e29255660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29255660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 -- 172.21.15.79:0/2809344909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 msgr2=0x562e29261870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29261870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.501+0000 7f630f7fe640 1 -- 172.21.15.79:0/2809344909 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e292582b0 con 0x562e29255ec0 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.502+0000 7f630f7fe640 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e2926d9a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6308010860 tx=0x7f6308007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dffad8661b552fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.502+0000 7f6304ff9640 1 -- 172.21.15.79:0/2809344909 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630803a020 con 0x562e29255ec0 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.502+0000 7f6304ff9640 1 -- 172.21.15.79:0/2809344909 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6308012650 con 0x562e29255ec0 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.502+0000 7f6304ff9640 1 -- 172.21.15.79:0/2809344909 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308012e80 con 0x562e29255ec0 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.502+0000 7f631840e740 1 -- 172.21.15.79:0/2809344909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 msgr2=0x562e2926d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.502+0000 7f631840e740 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e2926d9a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6308010860 tx=0x7f6308007920 comp rx=0 tx=0).stop 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- 172.21.15.79:0/2809344909 shutdown_connections 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924d2e0 0x562e29261870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e29255ec0 0x562e2926d9a0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 --2- 172.21.15.79:0/2809344909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e2924c9c0 0x562e29255660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- 172.21.15.79:0/2809344909 >> 172.21.15.79:0/2809344909 conn(0x562e292470f0 msgr2=0x562e29251800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:33.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- 172.21.15.79:0/2809344909 shutdown_connections 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- 172.21.15.79:0/2809344909 wait complete. 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- start start 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 Processor -- start 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924c9c0 0x562e29259a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29259fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e29255ec0 0x562e2925a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f63101af5e0 con 0x562e2924d2e0 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63101aef10 con 0x562e2924c9c0 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63101afcb0 con 0x562e29255ec0 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f631840e740 10 client.0 _ll_register_callbacks cb 0x7f6308010db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29259fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29259fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48296/0 (socket says 172.21.15.79:48296) 2024-04-22T06:55:33.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 -- 172.21.15.79:0/1389702415 learned_addr learned my addr 172.21.15.79:0/1389702415 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f630effd640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924c9c0 0x562e29259a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f630f7fe640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e29255ec0 0x562e2925a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 -- 172.21.15.79:0/1389702415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e29255ec0 msgr2=0x562e2925a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e29255ec0 0x562e2925a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 -- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924c9c0 msgr2=0x562e29259a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.503+0000 7f6307fff640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924c9c0 0x562e29259a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f6307fff640 1 -- 172.21.15.79:0/1389702415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e292cb4e0 con 0x562e2924d2e0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f6307fff640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29259fb0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f62f8005ab0 tx=0x7f62f8005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99811df271a82899 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f62f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f62f67fc640 1 -- 172.21.15.79:0/1389702415 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f802c880 con 0x562e2924d2e0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f62f67fc640 1 -- 172.21.15.79:0/1389702415 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f62f802ca20 con 0x562e2924d2e0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6308012650 con 0x562e2924d2e0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f62f67fc640 1 -- 172.21.15.79:0/1389702415 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f62f802cc20 con 0x562e2924d2e0 2024-04-22T06:55:33.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 10 client.17601 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x562e292cc7c0 con 0x562e2924d2e0 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f62f67fc640 1 -- 172.21.15.79:0/1389702415 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6308012650 con 0x562e2924d2e0 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f62f67fc640 1 -- 172.21.15.79:0/1389702415 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f62f8003710 con 0x562e2924d2e0 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 10 client.17601 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 10 client.17601 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 20 client.17601 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e292cd030 0x562e292cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 4 client.17601 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}] 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.504+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.0, "await": true}) -- 0x562e29134d30 con 0x562e292cd030 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.505+0000 7f630effd640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e292cd030 0x562e292cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.505+0000 7f630effd640 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e292cd030 0x562e292cf450 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9d9ba14c32ba6f6 server_cookie=959cf7ef92419ebd in_seq=0 out_seq=0 2024-04-22T06:55:33.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:33.505+0000 7f62f67fc640 10 client.17601 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:33.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1006079871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:33.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:33 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3503311043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:33.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1006079871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:33.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:33 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3503311043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:33 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1006079871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:33.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:33 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3503311043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:34.051 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 podman[69399]: 2024-04-22 06:55:33.797226215 +0000 UTC m=+0.747076243 container remove 0f7958e8bb8ecb0f035dd7d232f4ffc11100caf23584cf8e39215787bdf723ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T06:55:34.051 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:33 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:55:34.161 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 podman[68003]: 2024-04-22 06:55:33.908417775 +0000 UTC m=+0.859096328 container remove 69e64f8300348a194a64c242960c1cb0603aa9c811d0bdbe1f67678ebaee382c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:55:34.162 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:33 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:55:34.430 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:34 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-22T06:55:34.431 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:34 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-22T06:55:34.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:34 smithi193 ceph-mon[31815]: pgmap v362: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:55:34.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:34 smithi190 ceph-mon[31421]: pgmap v362: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:55:34.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:34 smithi079 ceph-mon[27946]: pgmap v362: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:55:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 -- 172.21.15.79:0/627023568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 msgr2=0x7f229810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 --2- 172.21.15.79:0/627023568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f229810ede0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f2290009cd0 tx=0x7f229002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 -- 172.21.15.79:0/627023568 shutdown_connections 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 --2- 172.21.15.79:0/627023568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2298100450 0x7f22981008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 --2- 172.21.15.79:0/627023568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f229810ede0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 --2- 172.21.15.79:0/627023568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2298106f30 0x7f2298107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 -- 172.21.15.79:0/627023568 >> 172.21.15.79:0/627023568 conn(0x7f22980773e0 msgr2=0x7f22980fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 -- 172.21.15.79:0/627023568 shutdown_connections 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 -- 172.21.15.79:0/627023568 wait complete. 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 Processor -- start 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.318+0000 7f229f23a640 1 -- start start 2024-04-22T06:55:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2298100450 0x7f22981986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f2298198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2298106f30 0x7f229819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f2298112290 con 0x7f2298100e10 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2298112110 con 0x7f2298100450 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2298112410 con 0x7f2298106f30 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f2298198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f2298198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48306/0 (socket says 172.21.15.79:48306) 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 -- 172.21.15.79:0/1264626061 learned_addr learned my addr 172.21.15.79:0/1264626061 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229d7d7640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2298106f30 0x7f229819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229cfd6640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2298100450 0x7f22981986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 -- 172.21.15.79:0/1264626061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2298106f30 msgr2=0x7f229819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2298106f30 0x7f229819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 -- 172.21.15.79:0/1264626061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2298100450 msgr2=0x7f22981986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2298100450 0x7f22981986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 -- 172.21.15.79:0/1264626061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229819d6c0 con 0x7f2298100e10 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2297fff640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f2298198c10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f228400c890 tx=0x7f228400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2295ffb640 1 -- 172.21.15.79:0/1264626061 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284004220 con 0x7f2298100e10 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229819d9b0 con 0x7f2298100e10 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2295ffb640 1 -- 172.21.15.79:0/1264626061 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22840043c0 con 0x7f2298100e10 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.319+0000 7f2295ffb640 1 -- 172.21.15.79:0/1264626061 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284004a00 con 0x7f2298100e10 2024-04-22T06:55:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.320+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22981a5240 con 0x7f2298100e10 2024-04-22T06:55:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f2295ffb640 1 -- 172.21.15.79:0/1264626061 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f22840079b0 con 0x7f2298100e10 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 Processor -- start 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 -- start start 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f2295ffb640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f22880a3dc0 0x7f22880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbed9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbed9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f229d7d7640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f22880a3dc0 0x7f22880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5613cbe3e8d0 con 0x5613cbeda8d0 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5613cbddbe80 con 0x5613cbee30e0 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.322+0000 7f22a1e20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613cbe3a890 con 0x5613cbed9b60 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f2295ffb640 1 -- 172.21.15.79:0/1264626061 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f22840d0920 con 0x7f2298100e10 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbed9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbed9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48308/0 (socket says 172.21.15.79:48308) 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 -- 172.21.15.79:0/1257023319 learned_addr learned my addr 172.21.15.79:0/1257023319 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f2297fff640 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbed9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 -- 172.21.15.79:0/1257023319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 msgr2=0x5613cbed9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbed9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 -- 172.21.15.79:0/1257023319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 msgr2=0x5613cbee54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 -- 172.21.15.79:0/1257023319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613cbdfd320 con 0x5613cbeda8d0 2024-04-22T06:55:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f2297fff640 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbed9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229d7d7640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f22880a3dc0 0x7f22880a6280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2290005cc0 tx=0x7f2290032360 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f229cfd6640 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbed9620 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f228c00b760 tx=0x7f228c00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdcb2f0f55d358da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f2282ffd640 1 -- 172.21.15.79:0/1257023319 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c003dd0 con 0x5613cbeda8d0 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f2282ffd640 1 -- 172.21.15.79:0/1257023319 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f228c015070 con 0x5613cbeda8d0 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.323+0000 7f2282ffd640 1 -- 172.21.15.79:0/1257023319 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c004d20 con 0x5613cbeda8d0 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 -- 172.21.15.79:0/1257023319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 msgr2=0x5613cbed9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbed9620 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f228c00b760 tx=0x7f228c00bc20 comp rx=0 tx=0).stop 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 -- 172.21.15.79:0/1257023319 shutdown_connections 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbed9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1257023319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbed9620 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 -- 172.21.15.79:0/1257023319 >> 172.21.15.79:0/1257023319 conn(0x5613cbea0c90 msgr2=0x5613cbed4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 -- 172.21.15.79:0/1257023319 shutdown_connections 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 -- 172.21.15.79:0/1257023319 wait complete. 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 -- start start 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.324+0000 7f22a1e20740 1 Processor -- start 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbee2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbedfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5613cbee5e10 con 0x5613cbeda8d0 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5613cbdc6eb0 con 0x5613cbee30e0 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613cbee5f90 con 0x5613cbed9b60 2024-04-22T06:55:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f22a1e20740 10 client.0 _ll_register_callbacks cb 0x5613cbdf3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbedfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbedfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48324/0 (socket says 172.21.15.79:48324) 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 -- 172.21.15.79:0/919132057 learned_addr learned my addr 172.21.15.79:0/919132057 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f229cfd6640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbee2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f229d7d7640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 -- 172.21.15.79:0/919132057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 msgr2=0x5613cbee2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbee2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 -- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 msgr2=0x5613cbee0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 -- 172.21.15.79:0/919132057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613cbb51210 con 0x5613cbeda8d0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2297fff640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbedfdd0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f2284005fd0 tx=0x7f228409c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a276afe3b4c3e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.325+0000 7f2277fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f2277fff640 1 -- 172.21.15.79:0/919132057 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228401d040 con 0x5613cbeda8d0 2024-04-22T06:55:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f2277fff640 1 -- 172.21.15.79:0/919132057 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f228404aac0 con 0x5613cbeda8d0 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f228c015070 con 0x5613cbeda8d0 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f2277fff640 1 -- 172.21.15.79:0/919132057 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f2284005290 con 0x5613cbeda8d0 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 10 client.17613 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x5613cbc1d620 con 0x5613cbeda8d0 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f2277fff640 1 -- 172.21.15.79:0/919132057 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f228c015070 con 0x5613cbeda8d0 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f2277fff640 1 -- 172.21.15.79:0/919132057 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f2284051320 con 0x5613cbeda8d0 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 10 client.17613 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 10 client.17613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 20 client.17613 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbef30b0 0x5613cbef54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 4 client.17613 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.326+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613cbe52880 con 0x5613cbef30b0 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.327+0000 7f229cfd6640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbef30b0 0x5613cbef54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.327+0000 7f2277fff640 10 client.17613 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.327+0000 7f229cfd6640 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbef30b0 0x5613cbef54a0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e8a04cd745aece1 server_cookie=47704f4b1f0b9c8e in_seq=0 out_seq=0 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f2277fff640 1 -- 172.21.15.79:0/919132057 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613cbe52880 con 0x5613cbef30b0 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f2277fff640 10 client.17613 handle_command_reply: tid=0 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 client.17613 shutdown 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 msgr2=0x5613cbedfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbedfdd0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f2284005fd0 tx=0x7f228409c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 shutdown_connections 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbef30b0 0x5613cbef54a0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbee30e0 0x5613cbee0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbedfdd0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 --2- 172.21.15.79:0/919132057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbee2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 >> 172.21.15.79:0/919132057 conn(0x5613cbea0c90 msgr2=0x5613cbed65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f2277fff640 0 client.17613 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 shutdown_connections 2024-04-22T06:55:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 1 -- 172.21.15.79:0/919132057 wait complete. 2024-04-22T06:55:35.335 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.328+0000 7f22a1e20740 20 client.17613 trim_cache size 0 max 16384 2024-04-22T06:55:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 Processor -- start 2024-04-22T06:55:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 -- start start 2024-04-22T06:55:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbeeed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbeda8d0 0x5613cbee2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 0x5613cbee2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f22981b0cd0 con 0x5613cbee30e0 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f22981ae5a0 con 0x5613cbeda8d0 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f22a1e20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22981ac410 con 0x5613cbed9b60 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 0x5613cbee2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 0x5613cbee2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48336/0 (socket says 172.21.15.79:48336) 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 -- 172.21.15.79:0/1102121739 learned_addr learned my addr 172.21.15.79:0/1102121739 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229cfd6640 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbeeed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f2297fff640 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbeda8d0 0x5613cbee2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 -- 172.21.15.79:0/1102121739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 msgr2=0x5613cbeeed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbeeed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 -- 172.21.15.79:0/1102121739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbeda8d0 msgr2=0x5613cbee2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbeda8d0 0x5613cbee2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 -- 172.21.15.79:0/1102121739 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613cbecff30 con 0x5613cbee30e0 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f2297fff640 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbeda8d0 0x5613cbee2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.343+0000 7f229d7d7640 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 0x5613cbee2a40 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f2290009cd0 tx=0x7f2290040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaa211e0c21a66e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22827fc640 1 -- 172.21.15.79:0/1102121739 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290051070 con 0x5613cbee30e0 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22827fc640 1 -- 172.21.15.79:0/1102121739 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2290032940 con 0x5613cbee30e0 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22827fc640 1 -- 172.21.15.79:0/1102121739 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290048040 con 0x5613cbee30e0 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 -- 172.21.15.79:0/1102121739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 msgr2=0x5613cbee2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 0x5613cbee2a40 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f2290009cd0 tx=0x7f2290040000 comp rx=0 tx=0).stop 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 -- 172.21.15.79:0/1102121739 shutdown_connections 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbeda8d0 0x5613cbee2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbee30e0 0x5613cbee2a40 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 --2- 172.21.15.79:0/1102121739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbed9b60 0x5613cbeeed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 -- 172.21.15.79:0/1102121739 >> 172.21.15.79:0/1102121739 conn(0x5613cbea0c90 msgr2=0x5613cbedb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 -- 172.21.15.79:0/1102121739 shutdown_connections 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.344+0000 7f22a1e20740 1 -- 172.21.15.79:0/1102121739 wait complete. 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 -- start start 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 Processor -- start 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbed9b60 0x5613cbf519d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbf51f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbee30e0 0x5613cbee6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f22981aca60 con 0x5613cbeda8d0 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f22981aded0 con 0x5613cbed9b60 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22981ad130 con 0x5613cbee30e0 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f22a1e20740 10 client.0 _ll_register_callbacks cb 0x7f228409c300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f2297fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbf51f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f2297fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbf51f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48348/0 (socket says 172.21.15.79:48348) 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.345+0000 7f2297fff640 1 -- 172.21.15.79:0/2446396412 learned_addr learned my addr 172.21.15.79:0/2446396412 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f229cfd6640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbed9b60 0x5613cbf519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2297fff640 1 -- 172.21.15.79:0/2446396412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbee30e0 msgr2=0x5613cbee6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f229d7d7640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbee30e0 0x5613cbee6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2297fff640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbee30e0 0x5613cbee6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2297fff640 1 -- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbed9b60 msgr2=0x5613cbf519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2297fff640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbed9b60 0x5613cbf519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2297fff640 1 -- 172.21.15.79:0/2446396412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613cbee6e50 con 0x5613cbeda8d0 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f229cfd6640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbed9b60 0x5613cbf519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f229d7d7640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbee30e0 0x5613cbee6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2297fff640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbf51f10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f228409a770 tx=0x7f2284098d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afa24912d06186b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2277fff640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2277fff640 1 -- 172.21.15.79:0/2446396412 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22840b9070 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2277fff640 1 -- 172.21.15.79:0/2446396412 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9717+0+0 (secure 0 0 0) 0x7f22840bb2f0 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2290032940 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2277fff640 1 -- 172.21.15.79:0/2446396412 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f228401d070 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 10 client.17619 fetch_fsmap learned FSMap version 82 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=4+}) -- 0x5613cbf59040 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2277fff640 1 -- 172.21.15.79:0/2446396412 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2290032940 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f2277fff640 1 -- 172.21.15.79:0/2446396412 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 82) ==== 8338+0+0 (secure 0 0 0) 0x7f2284098210 con 0x5613cbeda8d0 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 10 client.17619 fetch_fsmap finished waiting for FSMap version 82 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 10 client.17619 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 20 client.17619 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbf59790 0x5613cbf5bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 4 client.17619 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.346+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613cbdde3d0 con 0x5613cbf59790 2024-04-22T06:55:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.347+0000 7f229cfd6640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbf59790 0x5613cbf5bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.347+0000 7f229cfd6640 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbf59790 0x5613cbf5bbb0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e48dd0c1a2d51902 server_cookie=4980b2c61ab0513a in_seq=0 out_seq=0 2024-04-22T06:55:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.347+0000 7f2277fff640 10 client.17619 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f2277fff640 1 -- 172.21.15.79:0/2446396412 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613cbdde3d0 con 0x5613cbf59790 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f2277fff640 10 client.17619 handle_command_reply: tid=0 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 client.17619 shutdown 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 msgr2=0x5613cbf51f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbf51f10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f228409a770 tx=0x7f2284098d60 comp rx=0 tx=0).stop 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 shutdown_connections 2024-04-22T06:55:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x5613cbf59790 0x5613cbf5bbb0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613cbee30e0 0x5613cbee6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x5613cbeda8d0 0x5613cbf51f10 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f2277fff640 0 client.17619 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 --2- 172.21.15.79:0/2446396412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5613cbed9b60 0x5613cbf519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 >> 172.21.15.79:0/2446396412 conn(0x5613cbea0c90 msgr2=0x5613cbede260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 shutdown_connections 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 1 -- 172.21.15.79:0/2446396412 wait complete. 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f22a1e20740 20 client.17619 trim_cache size 0 max 16384 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:55:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:55:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:55:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:55:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f22880a3dc0 msgr2=0x7f22880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f229f23a640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f22880a3dc0 0x7f22880a6280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2290005cc0 tx=0x7f2290032360 comp rx=0 tx=0).stop 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 msgr2=0x7f2298198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.352+0000 7f229f23a640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f2298198c10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f228400c890 tx=0x7f228400cd50 comp rx=0 tx=0).stop 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 shutdown_connections 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2298106f30 0x7f229819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2298100e10 0x7f2298198c10 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f22880a3dc0 0x7f22880a6280 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 --2- 172.21.15.79:0/1264626061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2298100450 0x7f22981986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 >> 172.21.15.79:0/1264626061 conn(0x7f22980773e0 msgr2=0x7f2298104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 shutdown_connections 2024-04-22T06:55:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:35.353+0000 7f229f23a640 1 -- 172.21.15.79:0/1264626061 wait complete. 2024-04-22T06:55:36.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:36 smithi193 ceph-mon[31815]: pgmap v363: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:55:36.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:36 smithi190 ceph-mon[31421]: pgmap v363: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:55:36.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:36 smithi079 ceph-mon[27946]: pgmap v363: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T06:55:38.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:38 smithi193 ceph-mon[31815]: pgmap v364: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:55:38.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:38 smithi190 ceph-mon[31421]: pgmap v364: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:55:38.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:38 smithi079 ceph-mon[27946]: pgmap v364: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-22T06:55:40.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:40 smithi190 ceph-mon[31421]: pgmap v365: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:40.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:40 smithi079 ceph-mon[27946]: pgmap v365: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:40.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:40 smithi193 ceph-mon[31815]: pgmap v365: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:42.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:42 smithi190 ceph-mon[31421]: pgmap v366: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:42.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:42 smithi079 ceph-mon[27946]: pgmap v366: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:42.942 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:42 smithi193 ceph-mon[31815]: pgmap v366: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:44.450 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-04-22T06:55:44.450 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 systemd[1]: Stopped Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:55:44.450 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 systemd[1]: Starting Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:55:44.450 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 podman[69512]: 2024-04-22 06:55:44.335616027 +0000 UTC m=+0.018854568 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:55:44.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:44 smithi190 ceph-mon[31421]: pgmap v367: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:44.465 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-04-22T06:55:44.466 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 systemd[1]: Stopped Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:55:44.466 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 systemd[1]: Starting Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:55:44.466 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 podman[68110]: 2024-04-22 06:55:44.350033108 +0000 UTC m=+0.018517744 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:55:44.707 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:44 smithi193 ceph-mon[31815]: pgmap v367: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:44.707 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 podman[69512]: 2024-04-22 06:55:44.456433057 +0000 UTC m=+0.139671582 container create c795b8988ff894629852104cd73c34143b817b975b9811096144a00de581786c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T06:55:44.707 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 podman[69512]: 2024-04-22 06:55:44.684536749 +0000 UTC m=+0.367775274 container init c795b8988ff894629852104cd73c34143b817b975b9811096144a00de581786c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:55:44.707 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 podman[69512]: 2024-04-22 06:55:44.68714498 +0000 UTC m=+0.370383509 container start c795b8988ff894629852104cd73c34143b817b975b9811096144a00de581786c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f62f67fc640 1 -- 172.21.15.79:0/1389702415 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: -125 ) ==== 8+0+591 (crc 0 0 0) 0x562e29134d30 con 0x562e292cd030 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f62f67fc640 10 client.17601 handle_command_reply: tid=0 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 client.17601 shutdown 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 msgr2=0x562e29259fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29259fb0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f62f8005ab0 tx=0x7f62f8005300 comp rx=0 tx=0).stop 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 shutdown_connections 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x562e292cd030 0x562e292cf450 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e29255ec0 0x562e2925a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x562e2924d2e0 0x562e29259fb0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:44.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 --2- 172.21.15.79:0/1389702415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e2924c9c0 0x562e29259a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 >> 172.21.15.79:0/1389702415 conn(0x562e292470f0 msgr2=0x562e29252ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f62f67fc640 0 client.17601 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 shutdown_connections 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 1 -- 172.21.15.79:0/1389702415 wait complete. 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:55:44.717+0000 7f631840e740 20 client.17601 trim_cache size 0 max 16384 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 83, 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:55:44.717 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 38, 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": { 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "29eac228": { 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "version": 38, 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "age_ref": 0.0, 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "CANCELED", 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 0.0 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "timeout": 60.0, 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "expiration": 90.0, 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "members": { 2024-04-22T06:55:44.718 INFO:teuthology.orchestra.run.smithi079.stdout: "file:/": { 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: "excluded": true, 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: "state": { 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "QUIESCING", 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: "age": 11.2 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:55:44.719 INFO:teuthology.orchestra.run.smithi079.stderr:Error ECANCELED: 2024-04-22T06:55:44.720 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:44 smithi079 ceph-mon[27946]: pgmap v367: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-22T06:55:44.722 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 8).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-04-22T06:55:44.892 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 podman[68110]: 2024-04-22 06:55:44.467312565 +0000 UTC m=+0.135797199 container create ea1358230f531fba5e6e823a4339b85959f89944b3f4e4b1ec56a11f186aafbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=) 2024-04-22T06:55:45.189 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 ceph-mds[69531]: -- 172.21.15.193:0/1136474981 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573ad323a40 con 0x5573ad2f7000 2024-04-22T06:55:45.189 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[69527]: starting mds.f at 2024-04-22T06:55:45.189 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 bash[69512]: c795b8988ff894629852104cd73c34143b817b975b9811096144a00de581786c 2024-04-22T06:55:45.189 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:55:44 smithi193 systemd[1]: Started Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:55:45.202 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 podman[68110]: 2024-04-22 06:55:44.895279158 +0000 UTC m=+0.563763792 container init ea1358230f531fba5e6e823a4339b85959f89944b3f4e4b1ec56a11f186aafbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:55:45.203 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 podman[68110]: 2024-04-22 06:55:44.898092559 +0000 UTC m=+0.566577194 container start ea1358230f531fba5e6e823a4339b85959f89944b3f4e4b1ec56a11f186aafbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:55:45.203 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 ceph-mds[68129]: -- 172.21.15.190:0/3226498016 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56527ceafa40 con 0x56527ce82800 2024-04-22T06:55:45.203 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[68125]: starting mds.e at 2024-04-22T06:55:45.203 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 bash[68110]: ea1358230f531fba5e6e823a4339b85959f89944b3f4e4b1ec56a11f186aafbd 2024-04-22T06:55:45.203 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:55:44 smithi190 systemd[1]: Started Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:55:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: daemon mds.f restarted 2024-04-22T06:55:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:55:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:55:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/755015132,v1:172.21.15.193:6835/755015132] up:boot 2024-04-22T06:55:45.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,2=e=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: daemon mds.e restarted 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1936736593,v1:172.21.15.190:6835/1936736593] up:boot 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:55:45.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:45 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: daemon mds.f restarted 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/755015132,v1:172.21.15.193:6835/755015132] up:boot 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,2=e=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:55:45.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:55:45.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:55:45.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: daemon mds.e restarted 2024-04-22T06:55:45.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:55:45.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1936736593,v1:172.21.15.190:6835/1936736593] up:boot 2024-04-22T06:55:45.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:55:45.741 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:55:45.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:55:45.742 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:45 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:45.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: daemon mds.f restarted 2024-04-22T06:55:45.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:55:45.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/755015132,v1:172.21.15.193:6835/755015132] up:boot 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,2=e=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: daemon mds.e restarted 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1936736593,v1:172.21.15.190:6835/1936736593] up:boot 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: fsmap cephfs:2/3 {0=b=up:active,1=h=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:55:45.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:45 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:replay,2=i=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:46 smithi079 ceph-mon[27946]: pgmap v370: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:55:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:46 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:resolve 2024-04-22T06:55:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:46 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:resolve 2024-04-22T06:55:47.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:46 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:46 smithi193 ceph-mon[31815]: pgmap v370: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:55:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:46 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:resolve 2024-04-22T06:55:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:46 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:resolve 2024-04-22T06:55:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:46 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:47.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:46 smithi190 ceph-mon[31421]: pgmap v370: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:55:47.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:46 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:resolve 2024-04-22T06:55:47.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:46 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:resolve 2024-04-22T06:55:47.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:46 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:resolve,2=i=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:55:48.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:55:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:55:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999974 2024-04-22T06:55:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:reconnect 2024-04-22T06:55:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:reconnect 2024-04-22T06:55:48.190 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:47 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999974 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:reconnect 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:reconnect 2024-04-22T06:55:48.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:47 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:48.445 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 -- 172.21.15.79:0/1592071472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2150102b80 msgr2=0x7f2150106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.445 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 --2- 172.21.15.79:0/1592071472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2150102b80 0x7f2150106bc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f2144009900 tx=0x7f214402bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 -- 172.21.15.79:0/1592071472 shutdown_connections 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 --2- 172.21.15.79:0/1592071472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21501072a0 0x7f215010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 --2- 172.21.15.79:0/1592071472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2150102b80 0x7f2150106bc0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 --2- 172.21.15.79:0/1592071472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21501021d0 0x7f21501025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 -- 172.21.15.79:0/1592071472 >> 172.21.15.79:0/1592071472 conn(0x7f21500793e0 msgr2=0x7f21500797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 -- 172.21.15.79:0/1592071472 shutdown_connections 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 -- 172.21.15.79:0/1592071472 wait complete. 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 Processor -- start 2024-04-22T06:55:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.447+0000 7f21556a7640 1 -- start start 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f21556a7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 0x7f21501986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f21556a7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21501072a0 0x7f2150198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f21556a7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f215019cf80 0x7f215019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f21556a7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f21501121c0 con 0x7f21501021d0 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f21556a7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2150112040 con 0x7f215019cf80 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f21556a7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2150112340 con 0x7f21501072a0 2024-04-22T06:55:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 0x7f21501986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 0x7f21501986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34110/0 (socket says 172.21.15.79:34110) 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 -- 172.21.15.79:0/2772970163 learned_addr learned my addr 172.21.15.79:0/2772970163 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214e7fc640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21501072a0 0x7f2150198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214f7fe640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f215019cf80 0x7f215019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 -- 172.21.15.79:0/2772970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21501072a0 msgr2=0x7f2150198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21501072a0 0x7f2150198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 -- 172.21.15.79:0/2772970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f215019cf80 msgr2=0x7f215019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f215019cf80 0x7f215019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 -- 172.21.15.79:0/2772970163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215019da70 con 0x7f21501021d0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214f7fe640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f215019cf80 0x7f215019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f214effd640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 0x7f21501986c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f214000eab0 tx=0x7f214000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.448+0000 7f2137fff640 1 -- 172.21.15.79:0/2772970163 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214000ce40 con 0x7f21501021d0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.449+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215019dd60 con 0x7f21501021d0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.449+0000 7f2137fff640 1 -- 172.21.15.79:0/2772970163 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2140011920 con 0x7f21501021d0 2024-04-22T06:55:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.449+0000 7f2137fff640 1 -- 172.21.15.79:0/2772970163 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140010a30 con 0x7f21501021d0 2024-04-22T06:55:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.449+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21501a5570 con 0x7f21501021d0 2024-04-22T06:55:48.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:55:48.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: reconnect by client.15747 172.21.15.79:0/4207854255 after 0 2024-04-22T06:55:48.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999984 2024-04-22T06:55:48.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: reconnect by client.24851 172.21.15.79:0/2094839437 after 0.000999974 2024-04-22T06:55:48.456 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:reconnect 2024-04-22T06:55:48.456 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:reconnect 2024-04-22T06:55:48.456 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:47 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:reconnect,2=i=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2137fff640 1 -- 172.21.15.79:0/2772970163 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2140010bd0 con 0x7f21501021d0 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 Processor -- start 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 -- start start 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff37620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff37fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff434d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b04fe9c8d0 con 0x55b04ff388d0 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b04fe39e80 con 0x55b04ff37b60 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2158370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b04fe98890 con 0x55b04ff410e0 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f2137fff640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f21380a3dc0 0x7f21380a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff37620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff37620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34114/0 (socket says 172.21.15.79:34114) 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f214f7fe640 1 -- 172.21.15.79:0/637555396 learned_addr learned my addr 172.21.15.79:0/637555396 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.451+0000 7f214effd640 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff37fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214e7fc640 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f21380a3dc0 0x7f21380a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 -- 172.21.15.79:0/637555396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 msgr2=0x55b04ff434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f2137fff640 1 -- 172.21.15.79:0/2772970163 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9845+0+0 (secure 0 0 0) 0x7f21400d1360 con 0x7f21501021d0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 -- 172.21.15.79:0/637555396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 msgr2=0x55b04ff37fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff37fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 -- 172.21.15.79:0/637555396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04fe5b320 con 0x55b04ff388d0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f214f7fe640 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff37620 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f21480145f0 tx=0x7f2148014ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd20aeca650eb4c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f21357fa640 1 -- 172.21.15.79:0/637555396 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21480129b0 con 0x55b04ff388d0 2024-04-22T06:55:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f21357fa640 1 -- 172.21.15.79:0/637555396 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21480035f0 con 0x55b04ff388d0 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.452+0000 7f21357fa640 1 -- 172.21.15.79:0/637555396 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214801e660 con 0x55b04ff388d0 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f214f7fe640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f21380a3dc0 0x7f21380a6280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2148021000 tx=0x7f214801f000 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 -- 172.21.15.79:0/637555396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 msgr2=0x55b04ff37620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff37620 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f21480145f0 tx=0x7f2148014ab0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 -- 172.21.15.79:0/637555396 shutdown_connections 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff37fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 --2- 172.21.15.79:0/637555396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff37620 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 -- 172.21.15.79:0/637555396 >> 172.21.15.79:0/637555396 conn(0x55b04fefec90 msgr2=0x55b04ff32850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 -- 172.21.15.79:0/637555396 shutdown_connections 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 -- 172.21.15.79:0/637555396 wait complete. 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 -- start start 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.453+0000 7f2158370740 1 Processor -- start 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff40a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.459 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff3dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff3e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55b04ff43e10 con 0x55b04ff388d0 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55b04fe24eb0 con 0x55b04ff37b60 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b04ff43f90 con 0x55b04ff410e0 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 10 client.0 _ll_register_callbacks cb 0x55b04fe51b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff3dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff3dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34116/0 (socket says 172.21.15.79:34116) 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 -- 172.21.15.79:0/1837620614 learned_addr learned my addr 172.21.15.79:0/1837620614 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214f7fe640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff3e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214effd640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff40a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 -- 172.21.15.79:0/1837620614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 msgr2=0x55b04ff3e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff3e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 -- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 msgr2=0x55b04ff40a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.460 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff40a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 -- 172.21.15.79:0/1837620614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04fbaf210 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f214e7fc640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff3dde0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2144009900 tx=0x7f214402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf160fd7a67f3fb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2132ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2132ff5640 1 -- 172.21.15.79:0/1837620614 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214402c610 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2132ff5640 1 -- 172.21.15.79:0/1837620614 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9845+0+0 (secure 0 0 0) 0x7f2144005030 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.454+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21480035f0 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2132ff5640 1 -- 172.21.15.79:0/1837620614 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2144052050 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 10 client.17634 fetch_fsmap learned FSMap version 89 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x55b04fc7b620 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2132ff5640 1 -- 172.21.15.79:0/1837620614 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21480035f0 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2132ff5640 1 -- 172.21.15.79:0/1837620614 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 89) ==== 8346+0+0 (secure 0 0 0) 0x7f21440035f0 con 0x55b04ff388d0 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 10 client.17634 fetch_fsmap finished waiting for FSMap version 89 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 10 client.17634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 20 client.17634 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ff51100 0x55b04ff53510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 4 client.17634 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b04feb0880 con 0x55b04ff51100 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.455+0000 7f214effd640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ff51100 0x55b04ff53510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f214effd640 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ff51100 0x55b04ff53510 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f02dc742da521b4 server_cookie=98e3554086590e83 in_seq=0 out_seq=0 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2132ff5640 10 client.17634 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2132ff5640 1 -- 172.21.15.79:0/1837620614 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b04feb0880 con 0x55b04ff51100 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2132ff5640 10 client.17634 handle_command_reply: tid=0 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2158370740 1 client.17634 shutdown 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 msgr2=0x55b04ff3dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2158370740 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff3dde0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2144009900 tx=0x7f214402ca10 comp rx=0 tx=0).stop 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 shutdown_connections 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.456+0000 7f2158370740 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ff51100 0x55b04ff53510 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff410e0 0x55b04ff3e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff3dde0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 1 --2- 172.21.15.79:0/1837620614 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff37b60 0x55b04ff40a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 >> 172.21.15.79:0/1837620614 conn(0x55b04fefec90 msgr2=0x55b04ff345a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2132ff5640 0 client.17634 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 shutdown_connections 2024-04-22T06:55:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 1 -- 172.21.15.79:0/1837620614 wait complete. 2024-04-22T06:55:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.457+0000 7f2158370740 20 client.17634 trim_cache size 0 max 16384 2024-04-22T06:55:48.474 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.471+0000 7f2158370740 1 Processor -- start 2024-04-22T06:55:48.474 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 -- start start 2024-04-22T06:55:48.474 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff4cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.474 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff32380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff410e0 0x55b04ff58e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ae830 con 0x55b04ff388d0 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21501aef00 con 0x55b04ff410e0 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2158370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ac680 con 0x55b04ff37b60 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff32380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff32380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34120/0 (socket says 172.21.15.79:34120) 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 -- 172.21.15.79:0/3446199734 learned_addr learned my addr 172.21.15.79:0/3446199734 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 -- 172.21.15.79:0/3446199734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 msgr2=0x55b04ff4cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214effd640 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff4cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff4cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 -- 172.21.15.79:0/3446199734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff410e0 msgr2=0x55b04ff58e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff410e0 0x55b04ff58e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.475 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 -- 172.21.15.79:0/3446199734 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04ff3e6b0 con 0x55b04ff388d0 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f214e7fc640 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff32380 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2144000c00 tx=0x7f214402cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3415462df45a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2134ff9640 1 -- 172.21.15.79:0/3446199734 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214405b070 con 0x55b04ff388d0 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2134ff9640 1 -- 172.21.15.79:0/3446199734 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21440054f0 con 0x55b04ff388d0 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.472+0000 7f2134ff9640 1 -- 172.21.15.79:0/3446199734 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21440046c0 con 0x55b04ff388d0 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 -- 172.21.15.79:0/3446199734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 msgr2=0x55b04ff32380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff32380 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f2144000c00 tx=0x7f214402cd90 comp rx=0 tx=0).stop 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 -- 172.21.15.79:0/3446199734 shutdown_connections 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff410e0 0x55b04ff58e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff388d0 0x55b04ff32380 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 --2- 172.21.15.79:0/3446199734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff4cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 -- 172.21.15.79:0/3446199734 >> 172.21.15.79:0/3446199734 conn(0x55b04fefec90 msgr2=0x55b04ff3c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 -- 172.21.15.79:0/3446199734 shutdown_connections 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 -- 172.21.15.79:0/3446199734 wait complete. 2024-04-22T06:55:48.476 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 -- start start 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.473+0000 7f2158370740 1 Processor -- start 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff3ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff388d0 0x55b04ff3b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 0x55b04ffb6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f21501accf0 con 0x55b04ff410e0 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ae160 con 0x55b04ff388d0 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ad3c0 con 0x55b04ff37b60 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 10 client.0 _ll_register_callbacks cb 0x7f2144009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 0x55b04ffb6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 0x55b04ffb6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34130/0 (socket says 172.21.15.79:34130) 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 -- 172.21.15.79:0/98460548 learned_addr learned my addr 172.21.15.79:0/98460548 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214effd640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff3ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214e7fc640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff388d0 0x55b04ff3b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 -- 172.21.15.79:0/98460548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 msgr2=0x55b04ff3ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff3ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 -- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff388d0 msgr2=0x55b04ff3b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff388d0 0x55b04ff3b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 -- 172.21.15.79:0/98460548 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04ff3bc50 con 0x55b04ff410e0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214e7fc640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff388d0 0x55b04ff3b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f214f7fe640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 0x55b04ffb6690 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f2148003f00 tx=0x7f2148005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64507e683020448 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2132ff5640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2132ff5640 1 -- 172.21.15.79:0/98460548 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148037070 con 0x55b04ff410e0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2132ff5640 1 -- 172.21.15.79:0/98460548 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9845+0+0 (secure 0 0 0) 0x7f21480129b0 con 0x55b04ff410e0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.474+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21440054f0 con 0x55b04ff410e0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2132ff5640 1 -- 172.21.15.79:0/98460548 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2148043050 con 0x55b04ff410e0 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 10 client.17640 fetch_fsmap learned FSMap version 89 2024-04-22T06:55:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x55b04ffb7ce0 con 0x55b04ff410e0 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2132ff5640 1 -- 172.21.15.79:0/98460548 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21440054f0 con 0x55b04ff410e0 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2132ff5640 1 -- 172.21.15.79:0/98460548 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 89) ==== 8346+0+0 (secure 0 0 0) 0x7f214800f040 con 0x55b04ff410e0 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 10 client.17640 fetch_fsmap finished waiting for FSMap version 89 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 10 client.17640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 20 client.17640 populate_metadata read hostname 'smithi079' 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ffb8120 0x55b04ffba540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 4 client.17640 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b04fe3c3d0 con 0x55b04ffb8120 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.475+0000 7f214effd640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ffb8120 0x55b04ffba540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.476+0000 7f214effd640 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ffb8120 0x55b04ffba540 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49300666bb5578b server_cookie=3d1eda0ef4817754 in_seq=0 out_seq=0 2024-04-22T06:55:48.479 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.476+0000 7f2132ff5640 10 client.17640 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2132ff5640 1 -- 172.21.15.79:0/98460548 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b04fe3c3d0 con 0x55b04ffb8120 2024-04-22T06:55:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2132ff5640 10 client.17640 handle_command_reply: tid=0 2024-04-22T06:55:48.480 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 client.17640 shutdown 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 msgr2=0x55b04ffb6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 0x55b04ffb6690 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f2148003f00 tx=0x7f2148005540 comp rx=0 tx=0).stop 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 shutdown_connections 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55b04ffb8120 0x55b04ffba540 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b04ff388d0 0x55b04ff3b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55b04ff410e0 0x55b04ffb6690 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2132ff5640 0 client.17640 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 --2- 172.21.15.79:0/98460548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b04ff37b60 0x55b04ff3ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 >> 172.21.15.79:0/98460548 conn(0x55b04fefec90 msgr2=0x55b04ff31040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 shutdown_connections 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 1 -- 172.21.15.79:0/98460548 wait complete. 2024-04-22T06:55:48.481 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.480+0000 7f2158370740 20 client.17640 trim_cache size 0 max 16384 2024-04-22T06:55:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:55:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:55:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:55:48.482 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f21380a3dc0 msgr2=0x7f21380a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f21380a3dc0 0x7f21380a6280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2148021000 tx=0x7f214801f000 comp rx=0 tx=0).stop 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 msgr2=0x7f21501986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 0x7f21501986c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f214000eab0 tx=0x7f214000ef70 comp rx=0 tx=0).stop 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 shutdown_connections 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f21380a3dc0 0x7f21380a6280 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f215019cf80 0x7f215019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21501072a0 0x7f2150198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 --2- 172.21.15.79:0/2772970163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f21501021d0 0x7f21501986c0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 >> 172.21.15.79:0/2772970163 conn(0x7f21500793e0 msgr2=0x7f21500ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 shutdown_connections 2024-04-22T06:55:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:55:48.481+0000 7f21556a7640 1 -- 172.21.15.79:0/2772970163 wait complete. 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: pgmap v371: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:rejoin 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:rejoin 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:active 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:active 2024-04-22T06:55:49.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:49.241 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:48 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:55:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: pgmap v371: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-22T06:55:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:rejoin 2024-04-22T06:55:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:rejoin 2024-04-22T06:55:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:49.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:active 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:active 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:49.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:48 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:55:49.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: pgmap v371: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-22T06:55:49.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:rejoin 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:rejoin 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: fsmap cephfs:3/3 {0=b=up:active,1=h=up:rejoin,2=i=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/3177981466,v1:172.21.15.193:6839/3177981466] up:active 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/246451352,v1:172.21.15.190:6837/246451352] up:active 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:55:49.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:48 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:55:50.239 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:49 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:55:50.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:49 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:55:50.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:49 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:55:51.239 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:50 smithi079 ceph-mon[27946]: pgmap v372: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-22T06:55:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:51.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:50 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:51.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:50 smithi190 ceph-mon[31421]: pgmap v372: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-22T06:55:51.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:51.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:50 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:50 smithi193 ceph-mon[31815]: pgmap v372: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-22T06:55:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:51.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:50 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:52.202 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:51 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:55:52.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:51 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:55:52.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:51 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:52.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:51 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:55:52.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:51 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:55:52.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:51 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:52.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:51 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:55:52.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:51 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:55:52.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:51 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:55:53.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:53 smithi193 ceph-mon[31815]: pgmap v373: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 767 B/s wr, 20 op/s 2024-04-22T06:55:53.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:53 smithi190 ceph-mon[31421]: pgmap v373: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 767 B/s wr, 20 op/s 2024-04-22T06:55:53.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:53 smithi079 ceph-mon[27946]: pgmap v373: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 767 B/s wr, 20 op/s 2024-04-22T06:55:55.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:55 smithi193 ceph-mon[31815]: pgmap v374: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 833 B/s wr, 23 op/s 2024-04-22T06:55:55.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:55 smithi190 ceph-mon[31421]: pgmap v374: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 833 B/s wr, 23 op/s 2024-04-22T06:55:55.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:55 smithi079 ceph-mon[27946]: pgmap v374: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 833 B/s wr, 23 op/s 2024-04-22T06:55:57.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:57 smithi193 ceph-mon[31815]: pgmap v375: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 690 B/s wr, 20 op/s 2024-04-22T06:55:57.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:57 smithi190 ceph-mon[31421]: pgmap v375: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 690 B/s wr, 20 op/s 2024-04-22T06:55:57.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:57 smithi079 ceph-mon[27946]: pgmap v375: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 690 B/s wr, 20 op/s 2024-04-22T06:55:59.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:55:59 smithi193 ceph-mon[31815]: pgmap v376: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 19 op/s 2024-04-22T06:55:59.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:55:59 smithi190 ceph-mon[31421]: pgmap v376: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 19 op/s 2024-04-22T06:55:59.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:55:59 smithi079 ceph-mon[27946]: pgmap v376: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 19 op/s 2024-04-22T06:56:01.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:01 smithi193 ceph-mon[31815]: pgmap v377: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 17 op/s 2024-04-22T06:56:01.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:01 smithi190 ceph-mon[31421]: pgmap v377: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 17 op/s 2024-04-22T06:56:01.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:01 smithi079 ceph-mon[27946]: pgmap v377: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 17 op/s 2024-04-22T06:56:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 -- 172.21.15.79:0/1653274377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c102b80 msgr2=0x7f722c106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 --2- 172.21.15.79:0/1653274377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c102b80 0x7f722c106bc0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f7214009900 tx=0x7f721402bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 -- 172.21.15.79:0/1653274377 shutdown_connections 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 --2- 172.21.15.79:0/1653274377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1072a0 0x7f722c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 --2- 172.21.15.79:0/1653274377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c102b80 0x7f722c106bc0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 --2- 172.21.15.79:0/1653274377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f722c1021d0 0x7f722c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 -- 172.21.15.79:0/1653274377 >> 172.21.15.79:0/1653274377 conn(0x7f722c0793e0 msgr2=0x7f722c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 -- 172.21.15.79:0/1653274377 shutdown_connections 2024-04-22T06:56:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.576+0000 7f723097e640 1 -- 172.21.15.79:0/1653274377 wait complete. 2024-04-22T06:56:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 Processor -- start 2024-04-22T06:56:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 -- start start 2024-04-22T06:56:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1021d0 0x7f722c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f722c1072a0 0x7f722c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 0x7f722c19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1121e0 con 0x7f722c19cfa0 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f722c112060 con 0x7f722c1021d0 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f723097e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f722c112360 con 0x7f722c1072a0 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f722ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 0x7f722c19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f722ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 0x7f722c19d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56862/0 (socket says 172.21.15.79:56862) 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.577+0000 7f722ad9d640 1 -- 172.21.15.79:0/1190291151 learned_addr learned my addr 172.21.15.79:0/1190291151 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722a59c640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1021d0 0x7f722c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722ad9d640 1 -- 172.21.15.79:0/1190291151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f722c1072a0 msgr2=0x7f722c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722ad9d640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f722c1072a0 0x7f722c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722ad9d640 1 -- 172.21.15.79:0/1190291151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1021d0 msgr2=0x7f722c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722ad9d640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1021d0 0x7f722c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722ad9d640 1 -- 172.21.15.79:0/1190291151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722c19da90 con 0x7f722c19cfa0 2024-04-22T06:56:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722a59c640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1021d0 0x7f722c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f722ad9d640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 0x7f722c19d450 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f722400e960 tx=0x7f722400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f72237fe640 1 -- 172.21.15.79:0/1190291151 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224004d20 con 0x7f722c19cfa0 2024-04-22T06:56:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f72237fe640 1 -- 172.21.15.79:0/1190291151 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7224005020 con 0x7f722c19cfa0 2024-04-22T06:56:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722c19dd80 con 0x7f722c19cfa0 2024-04-22T06:56:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f72237fe640 1 -- 172.21.15.79:0/1190291151 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72240116d0 con 0x7f722c19cfa0 2024-04-22T06:56:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.578+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f722c1a5520 con 0x7f722c19cfa0 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.580+0000 7f72237fe640 1 -- 172.21.15.79:0/1190291151 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f72240051d0 con 0x7f722c19cfa0 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.580+0000 7f723356a740 1 Processor -- start 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 -- start start 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a229a158d0 con 0x55a229aba0e0 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a2299b2e80 con 0x55a229ab0b60 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f723356a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a229a11890 con 0x55a229ab18d0 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f72237fe640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f72180a3dc0 0x7f72180a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56866/0 (socket says 172.21.15.79:56866) 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f7223fff640 1 -- 172.21.15.79:0/570411368 learned_addr learned my addr 172.21.15.79:0/570411368 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f722a59c640 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f722ad9d640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f72180a3dc0 0x7f72180a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.581+0000 7f7223fff640 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7223fff640 1 -- 172.21.15.79:0/570411368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 msgr2=0x55a229ab0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7223fff640 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7223fff640 1 -- 172.21.15.79:0/570411368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 msgr2=0x55a229ab0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7223fff640 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7223fff640 1 -- 172.21.15.79:0/570411368 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2299d4320 con 0x55a229aba0e0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f722a59c640 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f72237fe640 1 -- 172.21.15.79:0/1190291151 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9845+0+0 (secure 0 0 0) 0x7f72240d0ef0 con 0x7f722c19cfa0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7223fff640 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abc4d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7214005ab0 tx=0x7f7214005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a1e404267a47de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f722ad9d640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f72180a3dc0 0x7f72180a6280 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7224098dc0 tx=0x7f722407b040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7220ff9640 1 -- 172.21.15.79:0/570411368 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214046070 con 0x55a229aba0e0 2024-04-22T06:56:01.587 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7220ff9640 1 -- 172.21.15.79:0/570411368 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f721402ca60 con 0x55a229aba0e0 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.582+0000 7f7220ff9640 1 -- 172.21.15.79:0/570411368 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72140046c0 con 0x55a229aba0e0 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 -- 172.21.15.79:0/570411368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 msgr2=0x55a229abc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abc4d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7214005ab0 tx=0x7f7214005a20 comp rx=0 tx=0).stop 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 -- 172.21.15.79:0/570411368 shutdown_connections 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abc4d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 --2- 172.21.15.79:0/570411368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 -- 172.21.15.79:0/570411368 >> 172.21.15.79:0/570411368 conn(0x55a229a77c90 msgr2=0x55a229aab850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 -- 172.21.15.79:0/570411368 shutdown_connections 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 -- 172.21.15.79:0/570411368 wait complete. 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 -- start start 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.583+0000 7f723356a740 1 Processor -- start 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab0b60 0x55a229ab9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab18d0 0x55a229ab6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229ab74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x55a229abcdb0 con 0x55a229aba0e0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55a22999deb0 con 0x55a229ab18d0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a229abcf30 con 0x55a229ab0b60 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 10 client.0 _ll_register_callbacks cb 0x55a2299cab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229ab74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229ab74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56882/0 (socket says 172.21.15.79:56882) 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 -- 172.21.15.79:0/3700808091 learned_addr learned my addr 172.21.15.79:0/3700808091 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f722a59c640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab18d0 0x55a229ab6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab0b60 0x55a229ab9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 -- 172.21.15.79:0/3700808091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab0b60 msgr2=0x55a229ab9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab0b60 0x55a229ab9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 -- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab18d0 msgr2=0x55a229ab6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab18d0 0x55a229ab6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 -- 172.21.15.79:0/3700808091 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a229728210 con 0x55a229aba0e0 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f722a59c640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab18d0 0x55a229ab6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7223fff640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229ab74a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7214036040 tx=0x7f7214005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f56d12c3c50a61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7211ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7211ffb640 1 -- 172.21.15.79:0/3700808091 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214051060 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7211ffb640 1 -- 172.21.15.79:0/3700808091 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9845+0+0 (secure 0 0 0) 0x7f7214004830 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f721402ca60 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.584+0000 7f7211ffb640 1 -- 172.21.15.79:0/3700808091 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7214058050 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 10 client.17652 fetch_fsmap learned FSMap version 91 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x55a2297f4620 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f7211ffb640 1 -- 172.21.15.79:0/3700808091 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f721402ca60 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f7211ffb640 1 -- 172.21.15.79:0/3700808091 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 91) ==== 8338+0+0 (secure 0 0 0) 0x7f721403fa40 con 0x55a229aba0e0 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 10 client.17652 fetch_fsmap finished waiting for FSMap version 91 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 10 client.17652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 20 client.17652 populate_metadata read hostname 'smithi079' 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229ac9fd0 0x55a229acc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 4 client.17652 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:56:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a229a29880 con 0x55a229ac9fd0 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.585+0000 7f722a59c640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229ac9fd0 0x55a229acc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f722a59c640 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229ac9fd0 0x55a229acc3c0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4bd35bbb13edbe server_cookie=59486fbe60202abf in_seq=0 out_seq=0 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f7211ffb640 10 client.17652 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f7211ffb640 1 -- 172.21.15.79:0/3700808091 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a229a29880 con 0x55a229ac9fd0 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f7211ffb640 10 client.17652 handle_command_reply: tid=0 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f723356a740 1 client.17652 shutdown 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 msgr2=0x55a229ab74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.586+0000 7f723356a740 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229ab74a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7214036040 tx=0x7f7214005fd0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 shutdown_connections 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229ac9fd0 0x55a229acc3c0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab18d0 0x55a229ab6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229ab74a0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f7211ffb640 0 client.17652 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 --2- 172.21.15.79:0/3700808091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab0b60 0x55a229ab9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 >> 172.21.15.79:0/3700808091 conn(0x55a229a77c90 msgr2=0x55a229aad5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 shutdown_connections 2024-04-22T06:56:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 1 -- 172.21.15.79:0/3700808091 wait complete. 2024-04-22T06:56:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.587+0000 7f723356a740 20 client.17652 trim_cache size 0 max 16384 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.601+0000 7f723356a740 1 Processor -- start 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.601+0000 7f723356a740 1 -- start start 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.601+0000 7f723356a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.601+0000 7f723356a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f723356a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f723356a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1b0ec0 con 0x55a229aba0e0 2024-04-22T06:56:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f723356a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1ae790 con 0x55a229ab0b60 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f723356a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1ac5e0 con 0x55a229ab18d0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56886/0 (socket says 172.21.15.79:56886) 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 -- 172.21.15.79:0/4111952269 learned_addr learned my addr 172.21.15.79:0/4111952269 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 -- 172.21.15.79:0/4111952269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 msgr2=0x55a229ab9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 -- 172.21.15.79:0/4111952269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 msgr2=0x55a229ab95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 -- 172.21.15.79:0/4111952269 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a229aafe30 con 0x55a229aba0e0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7223fff640 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abb4d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f7214009900 tx=0x7f7214005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cd1c8ca4e7e41d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7213fff640 1 -- 172.21.15.79:0/4111952269 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214051070 con 0x55a229aba0e0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7213fff640 1 -- 172.21.15.79:0/4111952269 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f721402ca20 con 0x55a229aba0e0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.602+0000 7f7213fff640 1 -- 172.21.15.79:0/4111952269 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214046040 con 0x55a229aba0e0 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- 172.21.15.79:0/4111952269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 msgr2=0x55a229abb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abb4d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f7214009900 tx=0x7f7214005e60 comp rx=0 tx=0).stop 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- 172.21.15.79:0/4111952269 shutdown_connections 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229aba0e0 0x55a229abb4d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229ab9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- 172.21.15.79:0/4111952269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229ab0b60 0x55a229ab95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- 172.21.15.79:0/4111952269 >> 172.21.15.79:0/4111952269 conn(0x55a229a77c90 msgr2=0x55a229ab5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- 172.21.15.79:0/4111952269 shutdown_connections 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- 172.21.15.79:0/4111952269 wait complete. 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- start start 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 Processor -- start 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 0x55a229acae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229acb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229aba0e0 0x55a229acb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1acc50 con 0x55a229ab0b60 2024-04-22T06:56:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1ae0c0 con 0x55a229aba0e0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f722c1ad320 con 0x55a229ab18d0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f723356a740 10 client.0 _ll_register_callbacks cb 0x7f7214044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 0x55a229acae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f7223fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 0x55a229acae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56902/0 (socket says 172.21.15.79:56902) 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.603+0000 7f7223fff640 1 -- 172.21.15.79:0/1576818028 learned_addr learned my addr 172.21.15.79:0/1576818028 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f722a59c640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229acb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229aba0e0 0x55a229acb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 -- 172.21.15.79:0/1576818028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 msgr2=0x55a229acb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229acb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 -- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229aba0e0 msgr2=0x55a229acb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229aba0e0 0x55a229acb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 -- 172.21.15.79:0/1576818028 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a229b2f1d0 con 0x55a229ab0b60 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7223fff640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 0x55a229acae20 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7214000c00 tx=0x7f7214005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830d19c915fda112 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7211ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:56:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7211ffb640 1 -- 172.21.15.79:0/1576818028 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214051030 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7211ffb640 1 -- 172.21.15.79:0/1576818028 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9845+0+0 (secure 0 0 0) 0x7f7214034ab0 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f721402ca20 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7211ffb640 1 -- 172.21.15.79:0/1576818028 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7214046040 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f723356a740 10 client.17658 fetch_fsmap learned FSMap version 91 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x55a229b304a0 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7211ffb640 1 -- 172.21.15.79:0/1576818028 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f721402ca20 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.604+0000 7f7211ffb640 1 -- 172.21.15.79:0/1576818028 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 91) ==== 8338+0+0 (secure 0 0 0) 0x7f721403f6e0 con 0x55a229ab0b60 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f723356a740 10 client.17658 fetch_fsmap finished waiting for FSMap version 91 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f723356a740 10 client.17658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f723356a740 20 client.17658 populate_metadata read hostname 'smithi079' 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f723356a740 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229b30910 0x55a229b32d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f723356a740 4 client.17658 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2299b53d0 con 0x55a229b30910 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f722a59c640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229b30910 0x55a229b32d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f722a59c640 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229b30910 0x55a229b32d00 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36c9d1a49c9e828c server_cookie=541504580f4da648 in_seq=0 out_seq=0 2024-04-22T06:56:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:01.605+0000 7f7211ffb640 10 client.17658 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f7211ffb640 1 -- 172.21.15.79:0/1576818028 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2299b53d0 con 0x55a229b30910 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f7211ffb640 10 client.17658 handle_command_reply: tid=0 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 client.17658 shutdown 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 msgr2=0x55a229acae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 0x55a229acae20 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7214000c00 tx=0x7f7214005970 comp rx=0 tx=0).stop 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 shutdown_connections 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x55a229b30910 0x55a229b32d00 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a229aba0e0 0x55a229acb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a229ab18d0 0x55a229acb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.426 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f7211ffb640 0 client.17658 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 --2- 172.21.15.79:0/1576818028 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x55a229ab0b60 0x55a229acae20 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 >> 172.21.15.79:0/1576818028 conn(0x55a229a77c90 msgr2=0x55a229aaa040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 shutdown_connections 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 1 -- 172.21.15.79:0/1576818028 wait complete. 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.427+0000 7f723356a740 20 client.17658 trim_cache size 0 max 16384 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:{ 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "message": "", 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout: "return_code": 0 2024-04-22T06:56:03.427 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stdout:} 2024-04-22T06:56:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f72180a3dc0 msgr2=0x7f72180a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f72180a3dc0 0x7f72180a6280 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7224098dc0 tx=0x7f722407b040 comp rx=0 tx=0).stop 2024-04-22T06:56:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 msgr2=0x7f722c19d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 0x7f722c19d450 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f722400e960 tx=0x7f722400ee20 comp rx=0 tx=0).stop 2024-04-22T06:56:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 shutdown_connections 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f722c19cfa0 0x7f722c19d450 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f722c1072a0 0x7f722c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f72180a3dc0 0x7f72180a6280 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 --2- 172.21.15.79:0/1190291151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f722c1021d0 0x7f722c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 >> 172.21.15.79:0/1190291151 conn(0x7f722c0793e0 msgr2=0x7f722c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 shutdown_connections 2024-04-22T06:56:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi079.front.sepia.ceph.com.smithi079.stderr:2024-04-22T06:56:03.428+0000 7f723097e640 1 -- 172.21.15.79:0/1190291151 wait complete. 2024-04-22T06:56:03.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-mon[31815]: pgmap v378: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-22T06:56:03.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:03 smithi190 ceph-mon[31421]: pgmap v378: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-22T06:56:03.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:03 smithi079 ceph-mon[27946]: pgmap v378: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: *** Caught signal (Segmentation fault) ** 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: in thread 7ff9d85ec700 thread_name:md_log_replay 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff9e99add20] 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2: /usr/bin/ceph-mds(+0x549332) [0x557084c50332] 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557084c53ae3] 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 4: (MDLog::_replay_thread()+0x617) [0x557084cae607] 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 5: (MDLog::ReplayThread::entry()+0x11) [0x557084cb7cf1] 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff9e99a31ca] 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 7: clone() 2024-04-22T06:56:03.940 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.616+0000 7ff9d85ec700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: in thread 7ff9d85ec700 thread_name:md_log_replay 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff9e99add20] 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2: /usr/bin/ceph-mds(+0x549332) [0x557084c50332] 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557084c53ae3] 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 4: (MDLog::_replay_thread()+0x617) [0x557084cae607] 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 5: (MDLog::ReplayThread::entry()+0x11) [0x557084cb7cf1] 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff9e99a31ca] 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 7: clone() 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 0> 2024-04-22T06:56:03.616+0000 7ff9d85ec700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:56:03.941 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: in thread 7ff9d85ec700 thread_name:md_log_replay 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff9e99add20] 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2: /usr/bin/ceph-mds(+0x549332) [0x557084c50332] 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557084c53ae3] 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 4: (MDLog::_replay_thread()+0x617) [0x557084cae607] 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 5: (MDLog::ReplayThread::entry()+0x11) [0x557084cb7cf1] 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff9e99a31ca] 2024-04-22T06:56:03.942 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 7: clone() 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 0> 2024-04-22T06:56:03.616+0000 7ff9d85ec700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: in thread 7ff9d85ec700 thread_name:md_log_replay 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff9e99add20] 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2: /usr/bin/ceph-mds(+0x549332) [0x557084c50332] 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557084c53ae3] 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 4: (MDLog::_replay_thread()+0x617) [0x557084cae607] 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 5: (MDLog::ReplayThread::entry()+0x11) [0x557084cb7cf1] 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7ff9e99a31ca] 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 7: clone() 2024-04-22T06:56:03.943 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:56:03.944 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[68874]: 2024-04-22T06:56:03.944 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:03 smithi193 podman[69702]: 2024-04-22 06:56:03.671135075 +0000 UTC m=+0.022258036 container died 55e5a664d73d046f72f38b56fc671407a78d2d40e660c2b395a6075c47ab9e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T06:56:04.689 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:04 smithi193 podman[69702]: 2024-04-22 06:56:04.373114695 +0000 UTC m=+0.724237645 container remove 55e5a664d73d046f72f38b56fc671407a78d2d40e660c2b395a6075c47ab9e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:56:04.689 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:04 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-22T06:56:04.693 INFO:tasks.workunit.client.0.smithi079.stdout:stopping iogen 2024-04-22T06:56:04.693 INFO:tasks.workunit.client.0.smithi079.stderr:+ echo 'stopping iogen' 2024-04-22T06:56:04.693 INFO:tasks.workunit.client.0.smithi079.stderr:+ ./iogen -k 2024-04-22T06:56:04.698 INFO:tasks.workunit.client.0.smithi079.stdout:OK 2024-04-22T06:56:04.698 INFO:tasks.workunit.client.0.smithi079.stderr:+ echo OK 2024-04-22T06:56:05.189 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:04 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:56:05.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:05 smithi190 ceph-mon[31421]: pgmap v379: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-22T06:56:05.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:05 smithi190 ceph-mon[31421]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:56:05.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:05 smithi079 ceph-mon[27946]: pgmap v379: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-22T06:56:05.491 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:05 smithi079 ceph-mon[27946]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:56:05.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:05 smithi193 ceph-mon[31815]: pgmap v379: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-22T06:56:05.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:05 smithi193 ceph-mon[31815]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:56:06.717 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T06:56:06.717 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:56:06.814 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-04-22T06:56:06.814 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T06:56:07.125 DEBUG:teuthology.parallel:result is None 2024-04-22T06:56:07.126 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T06:56:07.155 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T06:56:07.156 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-22T06:56:07.167 INFO:tasks.quiescer:joining Quiescers 2024-04-22T06:56:07.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 -- 172.21.15.79:0/87979390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae241011a0 msgr2=0x7fae24101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/87979390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae241011a0 0x7fae24101620 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fae18009930 tx=0x7fae1802bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 -- 172.21.15.79:0/87979390 shutdown_connections 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/87979390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae24101b60 0x7fae2410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/87979390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae241011a0 0x7fae24101620 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/87979390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae24105e30 0x7fae24106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 -- 172.21.15.79:0/87979390 >> 172.21.15.79:0/87979390 conn(0x7fae24077e30 msgr2=0x7fae24078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 -- 172.21.15.79:0/87979390 shutdown_connections 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 -- 172.21.15.79:0/87979390 wait complete. 2024-04-22T06:56:07.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 Processor -- start 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.297+0000 7fae2bfbb640 1 -- start start 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae241011a0 0x7fae2410e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 0x7fae2410eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae24105e30 0x7fae2410f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fae24112250 con 0x7fae24101b60 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fae241120d0 con 0x7fae24105e30 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fae241123d0 con 0x7fae241011a0 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 0x7fae2410eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 0x7fae2410eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40774/0 (socket says 172.21.15.79:40774) 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 -- 172.21.15.79:0/497862852 learned_addr learned my addr 172.21.15.79:0/497862852 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2a558640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae24105e30 0x7fae2410f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29d57640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae241011a0 0x7fae2410e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 -- 172.21.15.79:0/497862852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae241011a0 msgr2=0x7fae2410e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae241011a0 0x7fae2410e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 -- 172.21.15.79:0/497862852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae24105e30 msgr2=0x7fae2410f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae24105e30 0x7fae2410f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 -- 172.21.15.79:0/497862852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae241a6bf0 con 0x7fae24101b60 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae29556640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 0x7fae2410eae0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fae18005940 tx=0x7fae180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18043020 con 0x7fae24101b60 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae241a6e80 con 0x7fae24101b60 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae180054c0 con 0x7fae24101b60 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18003d40 con 0x7fae24101b60 2024-04-22T06:56:07.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.298+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae241a7390 con 0x7fae24101b60 2024-04-22T06:56:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.299+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae24068de0 con 0x7fae24101b60 2024-04-22T06:56:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.300+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fae18046020 con 0x7fae24101b60 2024-04-22T06:56:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.301+0000 7fae16ffd640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fae100a3d50 0x7fae100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.301+0000 7fae29d57640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fae100a3d50 0x7fae100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.301+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7fae180f35d0 con 0x7fae24101b60 2024-04-22T06:56:07.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.301+0000 7fae29d57640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fae100a3d50 0x7fae100a6210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae20005e10 tx=0x7fae20005da0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.302 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.304+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae1804d020 con 0x7fae24101b60 2024-04-22T06:56:07.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:07 smithi190 ceph-mon[31421]: pgmap v381: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T06:56:07.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:07 smithi079 ceph-mon[27946]: pgmap v381: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T06:56:07.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:07 smithi193 ceph-mon[31815]: pgmap v381: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T06:56:07.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.521+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae24109350 con 0x7fae24101b60 2024-04-22T06:56:07.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.522+0000 7fae16ffd640 1 -- 172.21.15.79:0/497862852 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+11882 (secure 0 0 0) 0x7fae180bf800 con 0x7fae24101b60 2024-04-22T06:56:07.520 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:07.520 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":91,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:55:48.958280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":26096,"mds_2":26095},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26095":{"gid":26095,"name":"i","rank":2,"incarnation":86,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6839/3177981466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3177981466},{"type":"v1","addr":"172.21.15.193:6839","nonce":3177981466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26096":{"gid":26096,"name":"h","rank":1,"incarnation":84,"state":"up:active","state_seq":32,"addr":"172.21.15.190:6837/246451352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":246451352},{"type":"v1","addr":"172.21.15.190:6837","nonce":246451352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26113":{"gid":26113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/4272953642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":4272953642},{"type":"v1","addr":"172.21.15.193:6837","nonce":4272953642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26234":{"gid":26234,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1936736593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1936736593},{"type":"v1","addr":"172.21.15.190:6835","nonce":1936736593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26236":{"gid":26236,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/755015132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":755015132},{"type":"v1","addr":"172.21.15.193:6835","nonce":755015132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[24467,26095,26096]},"id":1}]} 2024-04-22T06:56:07.520 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 91 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.523+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fae100a3d50 msgr2=0x7fae100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.523+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fae100a3d50 0x7fae100a6210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae20005e10 tx=0x7fae20005da0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.523+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 msgr2=0x7fae2410eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.523+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 0x7fae2410eae0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fae18005940 tx=0x7fae180042f0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 shutdown_connections 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fae100a3d50 0x7fae100a6210 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae24105e30 0x7fae2410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fae24101b60 0x7fae2410eae0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 --2- 172.21.15.79:0/497862852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae241011a0 0x7fae2410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 >> 172.21.15.79:0/497862852 conn(0x7fae24077e30 msgr2=0x7fae240fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 shutdown_connections 2024-04-22T06:56:07.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.524+0000 7fae2bfbb640 1 -- 172.21.15.79:0/497862852 wait complete. 2024-04-22T06:56:07.533 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24467 quiesce db --cancel --all' 2024-04-22T06:56:07.533 DEBUG:teuthology.orchestra.run.smithi079:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24467 quiesce db --cancel --all 2024-04-22T06:56:07.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.637+0000 7fa878354640 1 -- 172.21.15.79:0/3472235341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 msgr2=0x7fa870101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.637+0000 7fa878354640 1 --2- 172.21.15.79:0/3472235341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870101470 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa860009900 tx=0x7fa86002bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- 172.21.15.79:0/3472235341 shutdown_connections 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 --2- 172.21.15.79:0/3472235341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 0x7fa87010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 --2- 172.21.15.79:0/3472235341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101a40 0x7fa870105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 --2- 172.21.15.79:0/3472235341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870101470 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- 172.21.15.79:0/3472235341 >> 172.21.15.79:0/3472235341 conn(0x7fa8700771d0 msgr2=0x7fa8700775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- 172.21.15.79:0/3472235341 shutdown_connections 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- 172.21.15.79:0/3472235341 wait complete. 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 Processor -- start 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- start start 2024-04-22T06:56:07.638 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 0x7fa870199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87019a040 0x7fa8701a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701110a0 con 0x7fa870101090 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa870110f20 con 0x7fa870106160 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa878354640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa870111220 con 0x7fa87019a040 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40800/0 (socket says 172.21.15.79:40800) 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.638+0000 7fa8760f0640 1 -- 172.21.15.79:0/1038525051 learned_addr learned my addr 172.21.15.79:0/1038525051 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8758ef640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 0x7fa870199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8768f1640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87019a040 0x7fa8701a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8760f0640 1 -- 172.21.15.79:0/1038525051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87019a040 msgr2=0x7fa8701a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8760f0640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87019a040 0x7fa8701a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8760f0640 1 -- 172.21.15.79:0/1038525051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 msgr2=0x7fa870199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8760f0640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 0x7fa870199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8760f0640 1 -- 172.21.15.79:0/1038525051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8701a4dc0 con 0x7fa870101090 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8758ef640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 0x7fa870199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8760f0640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870199520 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa860005940 tx=0x7fa86002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8677fe640 1 -- 172.21.15.79:0/1038525051 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860043020 con 0x7fa870101090 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8677fe640 1 -- 172.21.15.79:0/1038525051 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa86002cbe0 con 0x7fa870101090 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8701a5050 con 0x7fa870101090 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa8677fe640 1 -- 172.21.15.79:0/1038525051 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860003bf0 con 0x7fa870101090 2024-04-22T06:56:07.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.639+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8701a55a0 con 0x7fa870101090 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.642+0000 7fa8677fe640 1 -- 172.21.15.79:0/1038525051 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa860046020 con 0x7fa870101090 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.642+0000 7fa87b017740 1 Processor -- start 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.642+0000 7fa87b017740 1 -- start start 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.642+0000 7fa87b017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a6efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.642+0000 7fa87b017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 0x564535a780d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa87b017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a78b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa87b017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x5645359d4000 con 0x564535a6f5a0 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa87b017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x564535970b20 con 0x564535a6ebf0 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa87b017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5645359cffc0 con 0x564535a786a0 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 0x564535a780d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 0x564535a780d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40806/0 (socket says 172.21.15.79:40806) 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 -- 172.21.15.79:0/121100310 learned_addr learned my addr 172.21.15.79:0/121100310 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8758ef640 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a78b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8677fe640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa85c0a3d20 0x7fa85c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8768f1640 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a6efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8768f1640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa85c0a3d20 0x7fa85c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8677fe640 1 -- 172.21.15.79:0/1038525051 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7fa8600f2f80 con 0x7fa870101090 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 -- 172.21.15.79:0/121100310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 msgr2=0x564535a78b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a78b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 -- 172.21.15.79:0/121100310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 msgr2=0x564535a6efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a6efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 -- 172.21.15.79:0/121100310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56453598aea0 con 0x564535a6f5a0 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8768f1640 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a6efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8760f0640 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 0x564535a780d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa8600bb480 tx=0x7fa86004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c49bab5c7f8218 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa8768f1640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa85c0a3d20 0x7fa85c0a61e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa868009fd0 tx=0x7fa868005bd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa864bfe640 1 -- 172.21.15.79:0/121100310 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600be5f0 con 0x564535a6f5a0 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa864bfe640 1 -- 172.21.15.79:0/121100310 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa86003c350 con 0x564535a6f5a0 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.643+0000 7fa864bfe640 1 -- 172.21.15.79:0/121100310 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600be410 con 0x564535a6f5a0 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 -- 172.21.15.79:0/121100310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 msgr2=0x564535a780d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 0x564535a780d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fa8600bb480 tx=0x7fa86004f000 comp rx=0 tx=0).stop 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 -- 172.21.15.79:0/121100310 shutdown_connections 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a78b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6f5a0 0x564535a780d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 --2- 172.21.15.79:0/121100310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a6efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 -- 172.21.15.79:0/121100310 >> 172.21.15.79:0/121100310 conn(0x564535a693e0 msgr2=0x564535a697c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 -- 172.21.15.79:0/121100310 shutdown_connections 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.644+0000 7fa87b017740 1 -- 172.21.15.79:0/121100310 wait complete. 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 -- start start 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 Processor -- start 2024-04-22T06:56:07.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a7c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a7c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a7cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x564535a7f3a0 con 0x564535a6ebf0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x56453596e510 con 0x564535a6f5a0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564535a7f520 con 0x564535a786a0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa87b017740 10 client.0 _ll_register_callbacks cb 0x56453596df00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a7c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a7c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40808/0 (socket says 172.21.15.79:40808) 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 -- 172.21.15.79:0/2623873843 learned_addr learned my addr 172.21.15.79:0/2623873843 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8768f1640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a7cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8758ef640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a7c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 -- 172.21.15.79:0/2623873843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 msgr2=0x564535a7cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a7cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 -- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 msgr2=0x564535a7c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a7c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 -- 172.21.15.79:0/2623873843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645357b2a20 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa8760f0640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a7c0f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa8600bea40 tx=0x7fa8600036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add10ec5b9001512 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.645+0000 7fa859ffb640 1 -- 172.21.15.79:0/2623873843 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600be410 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa859ffb640 1 -- 172.21.15.79:0/2623873843 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7fa8600054d0 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa86003c350 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa859ffb640 1 -- 172.21.15.79:0/2623873843 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa8600be5f0 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 10 client.17676 fetch_fsmap learned FSMap version 91 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x5645358131b0 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa859ffb640 1 -- 172.21.15.79:0/2623873843 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa86003c350 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa859ffb640 1 -- 172.21.15.79:0/2623873843 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 91) ==== 8338+0+0 (secure 0 0 0) 0x7fa8600c7290 con 0x564535a6ebf0 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 10 client.17676 fetch_fsmap finished waiting for FSMap version 91 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 10 client.17676 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:56:07.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 20 client.17676 populate_metadata read hostname 'smithi079' 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535a8c6b0 0x564535a8eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 4 client.17676 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645359e1a20 con 0x564535a8c6b0 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.646+0000 7fa8758ef640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535a8c6b0 0x564535a8eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.647+0000 7fa859ffb640 10 client.17676 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.647+0000 7fa8758ef640 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535a8c6b0 0x564535a8eaa0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a698a128c2990748 server_cookie=a2ce5e962a10cd8c in_seq=0 out_seq=0 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.647+0000 7fa859ffb640 1 -- 172.21.15.79:0/2623873843 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645359e1a20 con 0x564535a8c6b0 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.647+0000 7fa859ffb640 10 client.17676 handle_command_reply: tid=0 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 client.17676 shutdown 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 msgr2=0x564535a7c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a7c0f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa8600bea40 tx=0x7fa8600036f0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 shutdown_connections 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535a8c6b0 0x564535a8eaa0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a7c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a7cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 --2- 172.21.15.79:0/2623873843 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a7c0f0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa859ffb640 0 client.17676 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 >> 172.21.15.79:0/2623873843 conn(0x564535a693e0 msgr2=0x564535a6b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 shutdown_connections 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 1 -- 172.21.15.79:0/2623873843 wait complete. 2024-04-22T06:56:07.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.648+0000 7fa87b017740 20 client.17676 trim_cache size 0 max 16384 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 Processor -- start 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 -- start start 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a7e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a6f5a0 0x564535a88260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 0x564535a79a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701aeec0 con 0x564535a786a0 2024-04-22T06:56:07.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701ac790 con 0x564535a6ebf0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa87b017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701aa600 con 0x564535a6f5a0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 0x564535a79a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 0x564535a79a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40818/0 (socket says 172.21.15.79:40818) 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 -- 172.21.15.79:0/2207450822 learned_addr learned my addr 172.21.15.79:0/2207450822 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 -- 172.21.15.79:0/2207450822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a6f5a0 msgr2=0x564535a88260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8758ef640 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a6f5a0 0x564535a88260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8760f0640 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a7e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a6f5a0 0x564535a88260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 -- 172.21.15.79:0/2207450822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 msgr2=0x564535a7e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a7e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8768f1640 1 -- 172.21.15.79:0/2207450822 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564535a8ee10 con 0x564535a786a0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.665+0000 7fa8760f0640 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a7e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa8768f1640 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 0x564535a79a90 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fa8680107e0 tx=0x7fa8680096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3d5e419cfa78a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa85bfff640 1 -- 172.21.15.79:0/2207450822 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868010d60 con 0x564535a786a0 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa85bfff640 1 -- 172.21.15.79:0/2207450822 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868024070 con 0x564535a786a0 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa85bfff640 1 -- 172.21.15.79:0/2207450822 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868029c90 con 0x564535a786a0 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 -- 172.21.15.79:0/2207450822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 msgr2=0x564535a79a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 0x564535a79a90 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fa8680107e0 tx=0x7fa8680096c0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 -- 172.21.15.79:0/2207450822 shutdown_connections 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a786a0 0x564535a79a90 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a6f5a0 0x564535a88260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 --2- 172.21.15.79:0/2207450822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6ebf0 0x564535a7e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 -- 172.21.15.79:0/2207450822 >> 172.21.15.79:0/2207450822 conn(0x564535a693e0 msgr2=0x564535a68680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 -- 172.21.15.79:0/2207450822 shutdown_connections 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.666+0000 7fa87b017740 1 -- 172.21.15.79:0/2207450822 wait complete. 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 -- start start 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 Processor -- start 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a84770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.670 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a84cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a818f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701aac50 con 0x564535a6ebf0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701ac0c0 con 0x564535a6f5a0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701ab320 con 0x564535a786a0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa87b017740 10 client.0 _ll_register_callbacks cb 0x7fa8600f65e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a84770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a84770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40822/0 (socket says 172.21.15.79:40822) 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 -- 172.21.15.79:0/59426645 learned_addr learned my addr 172.21.15.79:0/59426645 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8758ef640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a84cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8768f1640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a818f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 -- 172.21.15.79:0/59426645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 msgr2=0x564535a818f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a818f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 -- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 msgr2=0x564535a84cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:07.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a84cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 -- 172.21.15.79:0/59426645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645356a6310 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8768f1640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a818f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa8760f0640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a84770 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa8600c7630 tx=0x7fa860009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab32e85e89d758e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.667+0000 7fa859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.79:3300/0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa859ffb640 1 -- 172.21.15.79:0/59426645 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600be410 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa859ffb640 1 -- 172.21.15.79:0/59426645 <== mon.0 v2:172.21.15.79:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7fa8600f6880 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa868024070 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa859ffb640 1 -- 172.21.15.79:0/59426645 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa8600bad50 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 10 client.17682 fetch_fsmap learned FSMap version 91 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x5645356a6310 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa859ffb640 1 -- 172.21.15.79:0/59426645 <== mon.0 v2:172.21.15.79:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868024070 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa859ffb640 1 -- 172.21.15.79:0/59426645 <== mon.0 v2:172.21.15.79:3300/0 5 ==== fsmap(e 91) ==== 8338+0+0 (secure 0 0 0) 0x7fa8600ba800 con 0x564535a6ebf0 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 10 client.17682 fetch_fsmap finished waiting for FSMap version 91 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 10 client.17682 resolve_mds: validated gid 24467 aka daemon mds.b 2024-04-22T06:56:07.672 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 20 client.17682 populate_metadata read hostname 'smithi079' 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535af3730 0x564535af5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 4 client.17682 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 --> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x5645359db890 con 0x564535af3730 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.668+0000 7fa8758ef640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535af3730 0x564535af5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa8758ef640 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535af3730 0x564535af5b20 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285b8cc7838174dd server_cookie=43980e60d950d553 in_seq=0 out_seq=0 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa859ffb640 10 client.17682 ms_handle_connect on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa859ffb640 1 -- 172.21.15.79:0/59426645 <== mds.0 v2:172.21.15.190:6838/4154999918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x5645359db890 con 0x564535af3730 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa859ffb640 10 client.17682 handle_command_reply: tid=0 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 client.17682 shutdown 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 msgr2=0x564535a84770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a84770 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa8600c7630 tx=0x7fa860009560 comp rx=0 tx=0).stop 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 shutdown_connections 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:6838/4154999918,v1:172.21.15.190:6839/4154999918] conn(0x564535af3730 0x564535af5b20 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564535a6f5a0 0x564535a84cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564535a786a0 0x564535a818f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa859ffb640 0 client.17682 ms_handle_reset on v2:172.21.15.190:6838/4154999918 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 --2- 172.21.15.79:0/59426645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x564535a6ebf0 0x564535a84770 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 >> 172.21.15.79:0/59426645 conn(0x564535a693e0 msgr2=0x564535a7c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 shutdown_connections 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 1 -- 172.21.15.79:0/59426645 wait complete. 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.669+0000 7fa87b017740 20 client.17682 trim_cache size 0 max 16384 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 91, 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stdout: "leader": 24467, 2024-04-22T06:56:07.674 INFO:teuthology.orchestra.run.smithi079.stdout: "set_version": 38, 2024-04-22T06:56:07.675 INFO:teuthology.orchestra.run.smithi079.stdout: "sets": {} 2024-04-22T06:56:07.675 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:56:07.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa85c0a3d20 msgr2=0x7fa85c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa85c0a3d20 0x7fa85c0a61e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa868009fd0 tx=0x7fa868005bd0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 msgr2=0x7fa870199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870199520 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa860005940 tx=0x7fa86002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 shutdown_connections 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fa85c0a3d20 0x7fa85c0a61e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa870106160 0x7fa870199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87019a040 0x7fa8701a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 --2- 172.21.15.79:0/1038525051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa870101090 0x7fa870199520 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 >> 172.21.15.79:0/1038525051 conn(0x7fa8700771d0 msgr2=0x7fa8700fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 shutdown_connections 2024-04-22T06:56:07.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.670+0000 7fa878354640 1 -- 172.21.15.79:0/1038525051 wait complete. 2024-04-22T06:56:07.679 INFO:tasks.quiescer:done joining Quiescers 2024-04-22T06:56:07.679 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-22T06:56:07.688 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:56:07.696 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-22T06:56:07.728 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-22T06:56:07.728 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:56:07.728 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:56:07.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-22T06:56:07.809 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-22T06:56:07.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 -- 172.21.15.79:0/3825936171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc101aa0 msgr2=0x7fc0fc105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 --2- 172.21.15.79:0/3825936171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc101aa0 0x7fc0fc105a90 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009900 tx=0x7fc0f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 -- 172.21.15.79:0/3825936171 shutdown_connections 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 --2- 172.21.15.79:0/3825936171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc106170 0x7fc0fc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 --2- 172.21.15.79:0/3825936171 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc101aa0 0x7fc0fc105a90 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 --2- 172.21.15.79:0/3825936171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 -- 172.21.15.79:0/3825936171 >> 172.21.15.79:0/3825936171 conn(0x7fc0fc077fd0 msgr2=0x7fc0fc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 -- 172.21.15.79:0/3825936171 shutdown_connections 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.977+0000 7fc102f59640 1 -- 172.21.15.79:0/3825936171 wait complete. 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 Processor -- start 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 -- start start 2024-04-22T06:56:07.978 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0fc106170 0x7fc0fc198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc19cf80 0x7fc0fc19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc1110b0 con 0x7fc0fc1010f0 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc110f30 con 0x7fc0fc106170 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc102f59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc111230 con 0x7fc0fc19cf80 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc100cf5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc100cf5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40842/0 (socket says 172.21.15.79:40842) 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc100cf5640 1 -- 172.21.15.79:0/255170190 learned_addr learned my addr 172.21.15.79:0/255170190 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc0fbfff640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0fc106170 0x7fc0fc198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.978+0000 7fc1014f6640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc19cf80 0x7fc0fc19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc100cf5640 1 -- 172.21.15.79:0/255170190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc19cf80 msgr2=0x7fc0fc19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc100cf5640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc19cf80 0x7fc0fc19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc100cf5640 1 -- 172.21.15.79:0/255170190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0fc106170 msgr2=0x7fc0fc198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:07.979 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc100cf5640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0fc106170 0x7fc0fc198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc100cf5640 1 -- 172.21.15.79:0/255170190 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc19da70 con 0x7fc0fc1010f0 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc1014f6640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc19cf80 0x7fc0fc19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc100cf5640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1986c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00d950 tx=0x7fc0ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec017030 con 0x7fc0fc1010f0 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0ec004020 con 0x7fc0fc1010f0 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0fc19dd60 con 0x7fc0fc1010f0 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0047a0 con 0x7fc0fc1010f0 2024-04-22T06:56:07.980 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.979+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0fc1a5520 con 0x7fc0fc1010f0 2024-04-22T06:56:07.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.980+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0fc04fce0 con 0x7fc0fc1010f0 2024-04-22T06:56:07.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.981+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc0ec009a30 con 0x7fc0fc1010f0 2024-04-22T06:56:07.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.982+0000 7fc0f9ffb640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc0e40a3dc0 0x7fc0e40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:07.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.982+0000 7fc0fbfff640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc0e40a3dc0 0x7fc0e40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:07.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.982+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7fc0ec0d0ee0 con 0x7fc0fc1010f0 2024-04-22T06:56:07.981 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.982+0000 7fc0fbfff640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc0e40a3dc0 0x7fc0e40a6280 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0006010 tx=0x7fc0f0004040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:07.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:07.986+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0ec09d000 con 0x7fc0fc1010f0 2024-04-22T06:56:08.183 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.185+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fc0fc101aa0 con 0x7fc0e40a3dc0 2024-04-22T06:56:08.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.194+0000 7fc0f9ffb640 1 -- 172.21.15.79:0/255170190 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fc0fc101aa0 con 0x7fc0e40a3dc0 2024-04-22T06:56:08.192 INFO:teuthology.orchestra.run.smithi079.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T06:44:32", "first": "2024-04-22T06:45:00", "last": "2024-04-22T06:56:00", "last_pruned": "2024-04-22T06:56:00", "created_count": 12, "pruned_count": 6, "active": true} 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc0e40a3dc0 msgr2=0x7fc0e40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc0e40a3dc0 0x7fc0e40a6280 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0006010 tx=0x7fc0f0004040 comp rx=0 tx=0).stop 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 msgr2=0x7fc0fc1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1986c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00d950 tx=0x7fc0ec00de10 comp rx=0 tx=0).stop 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 shutdown_connections 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc0e40a3dc0 0x7fc0e40a6280 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0fc106170 0x7fc0fc198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0fc19cf80 0x7fc0fc19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 --2- 172.21.15.79:0/255170190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0fc1010f0 0x7fc0fc1986c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 >> 172.21.15.79:0/255170190 conn(0x7fc0fc077fd0 msgr2=0x7fc0fc069970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:08.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 shutdown_connections 2024-04-22T06:56:08.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.196+0000 7fc102f59640 1 -- 172.21.15.79:0/255170190 wait complete. 2024-04-22T06:56:08.205 DEBUG:teuthology.orchestra.run.smithi079:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T06:56:08.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 -- 172.21.15.79:0/3582167204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c38102b80 msgr2=0x7f0c38106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 --2- 172.21.15.79:0/3582167204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c38102b80 0x7f0c38106bc0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009900 tx=0x7f0c2802bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 -- 172.21.15.79:0/3582167204 shutdown_connections 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 --2- 172.21.15.79:0/3582167204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c381072a0 0x7f0c3810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 --2- 172.21.15.79:0/3582167204 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c38102b80 0x7f0c38106bc0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 --2- 172.21.15.79:0/3582167204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c381021d0 0x7f0c381025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 -- 172.21.15.79:0/3582167204 >> 172.21.15.79:0/3582167204 conn(0x7f0c380793e0 msgr2=0x7f0c380797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 -- 172.21.15.79:0/3582167204 shutdown_connections 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.308+0000 7f0c405f4640 1 -- 172.21.15.79:0/3582167204 wait complete. 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 Processor -- start 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 -- start start 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 0x7f0c381986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c381072a0 0x7f0c38198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c3819cf60 0x7f0c3819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f0c381121e0 con 0x7f0c381021d0 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c38112060 con 0x7f0c3819cf60 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c405f4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0c38112360 con 0x7f0c381072a0 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 0x7f0c381986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 0x7f0c381986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40866/0 (socket says 172.21.15.79:40866) 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 -- 172.21.15.79:0/2486828885 learned_addr learned my addr 172.21.15.79:0/2486828885 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3eb91640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c3819cf60 0x7f0c3819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3db8f640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c381072a0 0x7f0c38198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 -- 172.21.15.79:0/2486828885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c381072a0 msgr2=0x7f0c38198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c381072a0 0x7f0c38198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 -- 172.21.15.79:0/2486828885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c3819cf60 msgr2=0x7f0c3819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c3819cf60 0x7f0c3819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.309+0000 7f0c3e390640 1 -- 172.21.15.79:0/2486828885 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3819da50 con 0x7f0c381021d0 2024-04-22T06:56:08.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.310+0000 7f0c3e390640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 0x7f0c381986a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400e9f0 tx=0x7f0c3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:08.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.310+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400cd80 con 0x7f0c381021d0 2024-04-22T06:56:08.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.310+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3819dd40 con 0x7f0c381021d0 2024-04-22T06:56:08.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.310+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c340054d0 con 0x7f0c381021d0 2024-04-22T06:56:08.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.310+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34010690 con 0x7f0c381021d0 2024-04-22T06:56:08.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.310+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c381a54b0 con 0x7f0c381021d0 2024-04-22T06:56:08.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.311+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c3804fce0 con 0x7f0c381021d0 2024-04-22T06:56:08.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.312+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c34004e90 con 0x7f0c381021d0 2024-04-22T06:56:08.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.312+0000 7f0c2f7fe640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c240a3dc0 0x7f0c240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.313+0000 7f0c3db8f640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c240a3dc0 0x7f0c240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.313+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7f0c340d1180 con 0x7f0c381021d0 2024-04-22T06:56:08.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.313+0000 7f0c3db8f640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c240a3dc0 0x7f0c240a6280 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009230 tx=0x7f0c28005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:08.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.317+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c3409d2a0 con 0x7f0c381021d0 2024-04-22T06:56:08.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:08 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/497862852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:08.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:08 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/497862852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:08.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.517+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 --> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f0c38102b80 con 0x7f0c240a3dc0 2024-04-22T06:56:08.523 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.525+0000 7f0c2f7fe640 1 -- 172.21.15.79:0/2486828885 <== mgr.24827 v2:172.21.15.79:6800/1352275504 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f0c38102b80 con 0x7f0c240a3dc0 2024-04-22T06:56:08.523 INFO:teuthology.orchestra.run.smithi079.stdout:/ 1m 6m3h 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c240a3dc0 msgr2=0x7f0c240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c240a3dc0 0x7f0c240a6280 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009230 tx=0x7f0c28005fd0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 msgr2=0x7f0c381986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 0x7f0c381986a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400e9f0 tx=0x7f0c3400edf0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 shutdown_connections 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f0c240a3dc0 0x7f0c240a6280 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c3819cf60 0x7f0c3819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c381072a0 0x7f0c38198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 --2- 172.21.15.79:0/2486828885 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c381021d0 0x7f0c381986a0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 >> 172.21.15.79:0/2486828885 conn(0x7f0c380793e0 msgr2=0x7f0c380ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:08.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 shutdown_connections 2024-04-22T06:56:08.526 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.527+0000 7f0c405f4640 1 -- 172.21.15.79:0/2486828885 wait complete. 2024-04-22T06:56:08.536 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-22T06:56:08.547 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-22T06:56:08.547 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls' 2024-04-22T06:56:08.548 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-22T06:56:08.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:56:08.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:08 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/497862852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 -- 172.21.15.79:0/3389569313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3758107270 msgr2=0x7f375810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 --2- 172.21.15.79:0/3389569313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3758107270 0x7f375810ee40 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ca0 tx=0x7f375002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 -- 172.21.15.79:0/3389569313 shutdown_connections 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 --2- 172.21.15.79:0/3389569313 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3758107270 0x7f375810ee40 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 --2- 172.21.15.79:0/3389569313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3758102b50 0x7f3758106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 --2- 172.21.15.79:0/3389569313 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37581021a0 0x7f3758102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 -- 172.21.15.79:0/3389569313 >> 172.21.15.79:0/3389569313 conn(0x7f37580793a0 msgr2=0x7f37580797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:08.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 -- 172.21.15.79:0/3389569313 shutdown_connections 2024-04-22T06:56:08.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 -- 172.21.15.79:0/3389569313 wait complete. 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.696+0000 7f375d17e640 1 Processor -- start 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 -- start start 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 0x7f37581986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3758107270 0x7f3758198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f375819cf50 0x7f375819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f37581121e0 con 0x7f37581021a0 2024-04-22T06:56:08.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3758112060 con 0x7f375819cf50 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375d17e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3758112360 con 0x7f3758107270 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 0x7f37581986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 0x7f37581986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40882/0 (socket says 172.21.15.79:40882) 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 -- 172.21.15.79:0/2734364983 learned_addr learned my addr 172.21.15.79:0/2734364983 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375659c640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3758107270 0x7f3758198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375759e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f375819cf50 0x7f375819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 -- 172.21.15.79:0/2734364983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3758107270 msgr2=0x7f3758198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3758107270 0x7f3758198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 -- 172.21.15.79:0/2734364983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f375819cf50 msgr2=0x7f375819d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f375819cf50 0x7f375819d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 -- 172.21.15.79:0/2734364983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375819da40 con 0x7f37581021a0 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f375759e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f375819cf50 0x7f375819d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:08.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.697+0000 7f3756d9d640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 0x7f37581986e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f374800c890 tx=0x7f374800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:08.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.698+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748004220 con 0x7f37581021a0 2024-04-22T06:56:08.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.698+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375819dd30 con 0x7f37581021a0 2024-04-22T06:56:08.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.698+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37480043c0 con 0x7f37581021a0 2024-04-22T06:56:08.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.698+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37581a5570 con 0x7f37581021a0 2024-04-22T06:56:08.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.698+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748004a00 con 0x7f37581021a0 2024-04-22T06:56:08.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.699+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3758069230 con 0x7f37581021a0 2024-04-22T06:56:08.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.700+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f37480079b0 con 0x7f37581021a0 2024-04-22T06:56:08.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.700+0000 7f373ffff640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37400a3d20 0x7f37400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:08.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.701+0000 7f375659c640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37400a3d20 0x7f37400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:08.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.701+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7f37480d0ee0 con 0x7f37581021a0 2024-04-22T06:56:08.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.701+0000 7f375659c640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37400a3d20 0x7f37400a61e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3758101830 tx=0x7f374c00b390 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:08.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.704+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3748014030 con 0x7f37581021a0 2024-04-22T06:56:08.884 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.886+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37581999e0 con 0x7f37581021a0 2024-04-22T06:56:08.886 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.888+0000 7f373ffff640 1 -- 172.21.15.79:0/2734364983 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+25289 (secure 0 0 0) 0x7f374809d000 con 0x7f37581021a0 2024-04-22T06:56:08.886 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:08.886 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":116,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:56:04.189603+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6834/1769517674":"2024-04-23T06:55:44.932995+0000","172.21.15.193:6835/2674386425":"2024-04-23T06:55:44.714527+0000","172.21.15.193:6834/2674386425":"2024-04-23T06:55:44.714527+0000","172.21.15.190:6837/1832563615":"2024-04-23T06:53:55.123865+0000","172.21.15.190:6836/1832563615":"2024-04-23T06:53:55.123865+0000","172.21.15.193:6838/3285898346":"2024-04-23T06:53:54.951144+0000","172.21.15.193:6834/99277862":"2024-04-23T06:52:48.004463+0000","172.21.15.190:6834/1167513219":"2024-04-23T06:52:47.992797+0000","172.21.15.193:6835/99277862":"2024-04-23T06:52:48.004463+0000","172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.190:6836/2879949884":"2024-04-23T06:45:31.980863+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.190:6836/1635214680":"2024-04-23T06:48:37.138355+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000","172.21.15.190:6834/2543246858":"2024-04-23T06:46:40.618172+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6837/2879949884":"2024-04-23T06:45:31.980863+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.190:6834/833753836":"2024-04-23T06:49:52.347745+0000","172.21.15.190:6837/1169559599":"2024-04-23T06:51:19.325817+0000","172.21.15.190:6836/1169559599":"2024-04-23T06:51:19.325817+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.193:6834/3775212644":"2024-04-23T06:46:40.297133+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6835/1167513219":"2024-04-23T06:52:47.992797+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.193:6838/746968688":"2024-04-23T06:45:31.955779+0000","172.21.15.193:6835/3775212644":"2024-04-23T06:46:40.297133+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6835/2543246858":"2024-04-23T06:46:40.618172+0000","172.21.15.190:6835/1769517674":"2024-04-23T06:55:44.932995+0000","172.21.15.193:6835/3147972524":"2024-04-23T06:49:52.390484+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.193:6838/1588799940":"2024-04-23T06:48:37.120397+0000","172.21.15.193:6839/3285898346":"2024-04-23T06:53:54.951144+0000","172.21.15.193:6839/1588799940":"2024-04-23T06:48:37.120397+0000","172.21.15.193:6839/1413649941":"2024-04-23T06:51:19.325685+0000","172.21.15.193:6838/1413649941":"2024-04-23T06:51:19.325685+0000","172.21.15.193:6839/746968688":"2024-04-23T06:45:31.955779+0000","172.21.15.190:6837/1635214680":"2024-04-23T06:48:37.138355+0000","172.21.15.190:6835/833753836":"2024-04-23T06:49:52.347745+0000","172.21.15.193:6834/3147972524":"2024-04-23T06:49:52.390484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:56:08.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37400a3d20 msgr2=0x7f37400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37400a3d20 0x7f37400a61e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3758101830 tx=0x7f374c00b390 comp rx=0 tx=0).stop 2024-04-22T06:56:08.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 msgr2=0x7f37581986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:08.888 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 0x7f37581986e0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f374800c890 tx=0x7f374800cd50 comp rx=0 tx=0).stop 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 shutdown_connections 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f37400a3d20 0x7f37400a61e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f375819cf50 0x7f375819d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3758107270 0x7f3758198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 --2- 172.21.15.79:0/2734364983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37581021a0 0x7f37581986e0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 >> 172.21.15.79:0/2734364983 conn(0x7f37580793a0 msgr2=0x7f37580ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 shutdown_connections 2024-04-22T06:56:08.889 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:08.890+0000 7f375d17e640 1 -- 172.21.15.79:0/2734364983 wait complete. 2024-04-22T06:56:08.900 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:56:08.900 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:56:09.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 -- 172.21.15.79:0/1019584593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc10c510 msgr2=0x7f64cc10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 --2- 172.21.15.79:0/1019584593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc10c510 0x7f64cc10e9b0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009ca0 tx=0x7f64c402c230 comp rx=0 tx=0).stop 2024-04-22T06:56:09.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 -- 172.21.15.79:0/1019584593 shutdown_connections 2024-04-22T06:56:09.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 --2- 172.21.15.79:0/1019584593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc103af0 0x7f64cc10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 --2- 172.21.15.79:0/1019584593 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc10c510 0x7f64cc10e9b0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 --2- 172.21.15.79:0/1019584593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64cc1031d0 0x7f64cc1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 -- 172.21.15.79:0/1019584593 >> 172.21.15.79:0/1019584593 conn(0x7f64cc077f50 msgr2=0x7f64cc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 -- 172.21.15.79:0/1019584593 shutdown_connections 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 -- 172.21.15.79:0/1019584593 wait complete. 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 Processor -- start 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.005+0000 7f64d387b640 1 -- start start 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 0x7f64cc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc103af0 0x7f64cc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64cc10c510 0x7f64cc19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc1121a0 con 0x7f64cc1031d0 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc112020 con 0x7f64cc103af0 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc112320 con 0x7f64cc10c510 2024-04-22T06:56:09.006 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 0x7f64cc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 0x7f64cc198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40900/0 (socket says 172.21.15.79:40900) 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 -- 172.21.15.79:0/991878903 learned_addr learned my addr 172.21.15.79:0/991878903 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d0e16640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc103af0 0x7f64cc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1e18640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64cc10c510 0x7f64cc19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 -- 172.21.15.79:0/991878903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64cc10c510 msgr2=0x7f64cc19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64cc10c510 0x7f64cc19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 -- 172.21.15.79:0/991878903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc103af0 msgr2=0x7f64cc198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc103af0 0x7f64cc198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 -- 172.21.15.79:0/991878903 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc19d6f0 con 0x7f64cc1031d0 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d1617640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 0x7f64cc198700 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f64c800e9f0 tx=0x7f64c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c800cd80 con 0x7f64cc1031d0 2024-04-22T06:56:09.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64cc19d9e0 con 0x7f64cc1031d0 2024-04-22T06:56:09.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64c80054d0 con 0x7f64cc1031d0 2024-04-22T06:56:09.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8010690 con 0x7f64cc1031d0 2024-04-22T06:56:09.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.006+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64cc1a5270 con 0x7f64cc1031d0 2024-04-22T06:56:09.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.007+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64cc068d70 con 0x7f64cc1031d0 2024-04-22T06:56:09.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.009+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f64c8004e90 con 0x7f64cc1031d0 2024-04-22T06:56:09.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.009+0000 7f64c27fc640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64b80a3d20 0x7f64b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.009+0000 7f64d0e16640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64b80a3d20 0x7f64b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.009+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7f64c80d12c0 con 0x7f64cc1031d0 2024-04-22T06:56:09.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.010+0000 7f64d0e16640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64b80a3d20 0x7f64b80a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f64cc10e340 tx=0x7f64bc005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.014+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64c809d3e0 con 0x7f64cc1031d0 2024-04-22T06:56:09.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.192+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f64cc1076f0 con 0x7f64cc1031d0 2024-04-22T06:56:09.190 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.192+0000 7f64c27fc640 1 -- 172.21.15.79:0/991878903 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f64c809cce0 con 0x7f64cc1031d0 2024-04-22T06:56:09.191 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.194+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64b80a3d20 msgr2=0x7f64b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.194+0000 7f64d387b640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64b80a3d20 0x7f64b80a61e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f64cc10e340 tx=0x7f64bc005f90 comp rx=0 tx=0).stop 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.194+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 msgr2=0x7f64cc198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.194+0000 7f64d387b640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 0x7f64cc198700 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f64c800e9f0 tx=0x7f64c800edf0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 shutdown_connections 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f64b80a3d20 0x7f64b80a61e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64cc103af0 0x7f64cc198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64cc10c510 0x7f64cc19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 --2- 172.21.15.79:0/991878903 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f64cc1031d0 0x7f64cc198700 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 >> 172.21.15.79:0/991878903 conn(0x7f64cc077f50 msgr2=0x7f64cc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 shutdown_connections 2024-04-22T06:56:09.193 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.195+0000 7f64d387b640 1 -- 172.21.15.79:0/991878903 wait complete. 2024-04-22T06:56:09.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:56:09.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 -- 172.21.15.79:0/2365051142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc107270 msgr2=0x7f07dc10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 --2- 172.21.15.79:0/2365051142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc107270 0x7f07dc10ee40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009ca0 tx=0x7f07d402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 -- 172.21.15.79:0/2365051142 shutdown_connections 2024-04-22T06:56:09.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 --2- 172.21.15.79:0/2365051142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06b1f0 0x7f07dc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 --2- 172.21.15.79:0/2365051142 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc107270 0x7f07dc10ee40 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 --2- 172.21.15.79:0/2365051142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07dc06a840 0x7f07dc06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 -- 172.21.15.79:0/2365051142 >> 172.21.15.79:0/2365051142 conn(0x7f07dc077eb0 msgr2=0x7f07dc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 -- 172.21.15.79:0/2365051142 shutdown_connections 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.308+0000 7f07e18f2640 1 -- 172.21.15.79:0/2365051142 wait complete. 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 Processor -- start 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 -- start start 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06a840 0x7f07dc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07dc107270 0x7f07dc198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 0x7f07dc19d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc1121d0 con 0x7f07dc19cf70 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc112050 con 0x7f07dc06a840 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07e18f2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc112350 con 0x7f07dc107270 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 0x7f07dc19d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 0x7f07dc19d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40930/0 (socket says 172.21.15.79:40930) 2024-04-22T06:56:09.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 -- 172.21.15.79:0/1770732232 learned_addr learned my addr 172.21.15.79:0/1770732232 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07d3fff640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07dc107270 0x7f07dc198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07daffd640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06a840 0x7f07dc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 -- 172.21.15.79:0/1770732232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07dc107270 msgr2=0x7f07dc198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07dc107270 0x7f07dc198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 -- 172.21.15.79:0/1770732232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06a840 msgr2=0x7f07dc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06a840 0x7f07dc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.309+0000 7f07db7fe640 1 -- 172.21.15.79:0/1770732232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc19da60 con 0x7f07dc19cf70 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.310+0000 7f07db7fe640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 0x7f07dc19d420 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f07d4005e30 tx=0x7f07d4034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.310+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4039040 con 0x7f07dc19cf70 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.310+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07d4044070 con 0x7f07dc19cf70 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.310+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc19dcf0 con 0x7f07dc19cf70 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.310+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4003e60 con 0x7f07dc19cf70 2024-04-22T06:56:09.310 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.310+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc1a54b0 con 0x7f07dc19cf70 2024-04-22T06:56:09.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.311+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07dc0691c0 con 0x7f07dc19cf70 2024-04-22T06:56:09.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.312+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f07d40324c0 con 0x7f07dc19cf70 2024-04-22T06:56:09.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.312+0000 7f07e08f0640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f07c80a3d20 0x7f07c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.312+0000 7f07daffd640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f07c80a3d20 0x7f07c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.312+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7f07d40f3000 con 0x7f07dc19cf70 2024-04-22T06:56:09.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.313+0000 7f07daffd640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f07c80a3d20 0x7f07c80a61e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f07cc005fd0 tx=0x7f07cc005850 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.317+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07d40bf230 con 0x7f07dc19cf70 2024-04-22T06:56:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:09 smithi190 ceph-mon[31421]: pgmap v382: 97 pgs: 97 active+clean; 23 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:56:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:09 smithi190 ceph-mon[31421]: from='client.17688 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:09 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2734364983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:09.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:09 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/991878903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:09.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:09 smithi079 ceph-mon[27946]: pgmap v382: 97 pgs: 97 active+clean; 23 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:56:09.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:09 smithi079 ceph-mon[27946]: from='client.17688 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:09.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:09 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2734364983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:09.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:09 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/991878903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:09.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.494+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f07dc06b1f0 con 0x7f07dc19cf70 2024-04-22T06:56:09.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.494+0000 7f07e08f0640 1 -- 172.21.15.79:0/1770732232 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) ==== 104+0+11 (secure 0 0 0) 0x7f07d40beb30 con 0x7f07dc19cf70 2024-04-22T06:56:09.492 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f07c80a3d20 msgr2=0x7f07c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f07c80a3d20 0x7f07c80a61e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f07cc005fd0 tx=0x7f07cc005850 comp rx=0 tx=0).stop 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 msgr2=0x7f07dc19d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 0x7f07dc19d420 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f07d4005e30 tx=0x7f07d4034760 comp rx=0 tx=0).stop 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 shutdown_connections 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07dc19cf70 0x7f07dc19d420 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07dc107270 0x7f07dc198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f07c80a3d20 0x7f07c80a61e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 --2- 172.21.15.79:0/1770732232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06a840 0x7f07dc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 >> 172.21.15.79:0/1770732232 conn(0x7f07dc077eb0 msgr2=0x7f07dc0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 shutdown_connections 2024-04-22T06:56:09.495 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.496+0000 7f07e18f2640 1 -- 172.21.15.79:0/1770732232 wait complete. 2024-04-22T06:56:09.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:56:09.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4212541240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a241031e0 msgr2=0x7f1a241035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4212541240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a241031e0 0x7f1a241035c0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009900 tx=0x7f1a1402bd60 comp rx=0 tx=0).stop 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4212541240 shutdown_connections 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4212541240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a2410c520 0x7f1a2410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4212541240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 0x7f1a2410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4212541240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a241031e0 0x7f1a241035c0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4212541240 >> 172.21.15.79:0/4212541240 conn(0x7f1a240771d0 msgr2=0x7f1a240775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4212541240 shutdown_connections 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.608+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4212541240 wait complete. 2024-04-22T06:56:09.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 Processor -- start 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 -- start start 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a241031e0 0x7f1a24100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 0x7f1a24101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 0x7f1a241018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f1a241121e0 con 0x7f1a2410c520 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1a24112060 con 0x7f1a241031e0 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a29fd0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a24112360 con 0x7f1a24103b00 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a23fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 0x7f1a241018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a23fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 0x7f1a241018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40952/0 (socket says 172.21.15.79:40952) 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a23fff640 1 -- 172.21.15.79:0/4053683567 learned_addr learned my addr 172.21.15.79:0/4053683567 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a237fe640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a241031e0 0x7f1a24100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.609+0000 7f1a22ffd640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 0x7f1a24101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a23fff640 1 -- 172.21.15.79:0/4053683567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 msgr2=0x7f1a24101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a23fff640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 0x7f1a24101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a23fff640 1 -- 172.21.15.79:0/4053683567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a241031e0 msgr2=0x7f1a24100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a23fff640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a241031e0 0x7f1a24100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a23fff640 1 -- 172.21.15.79:0/4053683567 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a241a6ba0 con 0x7f1a2410c520 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a22ffd640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 0x7f1a24101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a23fff640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 0x7f1a241018b0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00cce0 tx=0x7f1a1c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c00f450 con 0x7f1a2410c520 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a241a6e90 con 0x7f1a2410c520 2024-04-22T06:56:09.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a1c00faa0 con 0x7f1a2410c520 2024-04-22T06:56:09.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c018b70 con 0x7f1a2410c520 2024-04-22T06:56:09.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.610+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a241a7360 con 0x7f1a2410c520 2024-04-22T06:56:09.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.611+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a240691c0 con 0x7f1a2410c520 2024-04-22T06:56:09.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.612+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a1c018d10 con 0x7f1a2410c520 2024-04-22T06:56:09.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.612+0000 7f1a20ff9640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a0c0a3d50 0x7f1a0c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.613+0000 7f1a237fe640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a0c0a3d50 0x7f1a0c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.613+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7f1a1c0d1170 con 0x7f1a2410c520 2024-04-22T06:56:09.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.613+0000 7f1a237fe640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a0c0a3d50 0x7f1a0c0a6210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a14006010 tx=0x7f1a14000bc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.617+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a1c014030 con 0x7f1a2410c520 2024-04-22T06:56:09.693 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:09 smithi193 ceph-mon[31815]: pgmap v382: 97 pgs: 97 active+clean; 23 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:56:09.693 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:09 smithi193 ceph-mon[31815]: from='client.17688 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:09.694 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:09 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2734364983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:09.694 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:09 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/991878903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:09.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.796+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1a24107110 con 0x7f1a2410c520 2024-04-22T06:56:09.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.797+0000 7f1a20ff9640 1 -- 172.21.15.79:0/4053683567 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) ==== 100+0+11 (secure 0 0 0) 0x7f1a1c09d290 con 0x7f1a2410c520 2024-04-22T06:56:09.795 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.798+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a0c0a3d50 msgr2=0x7f1a0c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.798+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a0c0a3d50 0x7f1a0c0a6210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a14006010 tx=0x7f1a14000bc0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.798+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 msgr2=0x7f1a241018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.798+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 0x7f1a241018b0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00cce0 tx=0x7f1a1c00a620 comp rx=0 tx=0).stop 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 shutdown_connections 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a2410c520 0x7f1a241018b0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a24103b00 0x7f1a24101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f1a0c0a3d50 0x7f1a0c0a6210 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 --2- 172.21.15.79:0/4053683567 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a241031e0 0x7f1a24100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 >> 172.21.15.79:0/4053683567 conn(0x7f1a240771d0 msgr2=0x7f1a240fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 shutdown_connections 2024-04-22T06:56:09.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.799+0000 7f1a29fd0640 1 -- 172.21.15.79:0/4053683567 wait complete. 2024-04-22T06:56:09.808 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-22T06:56:09.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 -- 172.21.15.79:0/2205575043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c10c570 msgr2=0x7f887c10e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 --2- 172.21.15.79:0/2205575043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c10c570 0x7f887c10e9e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8874009ca0 tx=0x7f887402c180 comp rx=0 tx=0).stop 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 -- 172.21.15.79:0/2205575043 shutdown_connections 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 --2- 172.21.15.79:0/2205575043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c10c570 0x7f887c10e9e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 --2- 172.21.15.79:0/2205575043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 0x7f887c10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 --2- 172.21.15.79:0/2205575043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c103230 0x7f887c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 -- 172.21.15.79:0/2205575043 >> 172.21.15.79:0/2205575043 conn(0x7f887c077fb0 msgr2=0x7f887c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 -- 172.21.15.79:0/2205575043 shutdown_connections 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 -- 172.21.15.79:0/2205575043 wait complete. 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 Processor -- start 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.908+0000 7f88826eb640 1 -- start start 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 0x7f887c100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 0x7f887c1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c10c570 0x7f887c101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f887c1121b0 con 0x7f887c103230 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f887c112030 con 0x7f887c10c570 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f887c112330 con 0x7f887c103b50 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 0x7f887c100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 0x7f887c100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40968/0 (socket says 172.21.15.79:40968) 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 -- 172.21.15.79:0/1459603239 learned_addr learned my addr 172.21.15.79:0/1459603239 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887b7fe640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 0x7f887c1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 -- 172.21.15.79:0/1459603239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 msgr2=0x7f887c1013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 0x7f887c1013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 -- 172.21.15.79:0/1459603239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c10c570 msgr2=0x7f887c101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f8880c88640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c10c570 0x7f887c101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c10c570 0x7f887c101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:09.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 -- 172.21.15.79:0/1459603239 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c1a6af0 con 0x7f887c103230 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887b7fe640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 0x7f887c1013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f8880c88640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c10c570 0x7f887c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f887bfff640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 0x7f887c100e80 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f886400c920 tx=0x7f886400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864007c70 con 0x7f887c103230 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887c1a6de0 con 0x7f887c103230 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8864007e10 con 0x7f887c103230 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864018820 con 0x7f887c103230 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.909+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887c1a72f0 con 0x7f887c103230 2024-04-22T06:56:09.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.910+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887c0691c0 con 0x7f887c103230 2024-04-22T06:56:09.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.911+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f886400f450 con 0x7f887c103230 2024-04-22T06:56:09.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.912+0000 7f88797fa640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88680a3d50 0x7f88680a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:09.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.912+0000 7f887b7fe640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88680a3d50 0x7f88680a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:09.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.912+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9933+0+0 (secure 0 0 0) 0x7f88640d11e0 con 0x7f887c103230 2024-04-22T06:56:09.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.912+0000 7f887b7fe640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88680a3d50 0x7f88680a6210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f886c005e00 tx=0x7f886c005c70 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:09.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:09.916+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8864014030 con 0x7f887c103230 2024-04-22T06:56:10.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.140+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f887c102360 con 0x7f887c103230 2024-04-22T06:56:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.148+0000 7f88797fa640 1 -- 172.21.15.79:0/1459603239 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746060 (secure 0 0 0) 0x7f886400de20 con 0x7f887c103230 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 26, 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "active_gid": 24827, 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "active_name": "x", 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "active_addrs": { 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:10.156 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6800", 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 1352275504 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v1", 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:6801", 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 1352275504 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "active_addr": "172.21.15.79:6801/1352275504", 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "active_change": "2024-04-22T06:44:24.944156+0000", 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "active_mgr_features": 4540701547738038271, 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "available": true, 2024-04-22T06:56:10.157 INFO:teuthology.orchestra.run.smithi079.stdout: "standbys": [ 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "gid": 24799, 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "z", 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_features": 4540701547738038271, 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "available_modules": [ 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "alerts", 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.159 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.160 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.161 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.162 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.163 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_destination": { 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_destination", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.164 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_from_name": { 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_from_name", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "Ceph", 2024-04-22T06:56:10.165 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Email From: name", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_host": { 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_host", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.166 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP server", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_password": { 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_password", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.167 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Password to authenticate with", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_port": { 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_port", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "465", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.168 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP port", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_sender": { 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_sender", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.169 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_ssl": { 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_ssl", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.170 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_user": { 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_user", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.171 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "User to authenticate as", 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "balancer", 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "active": { 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "active", 2024-04-22T06:56:10.172 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "begin_time": { 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "begin_time", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.173 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0000", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "begin_weekday": { 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "begin_weekday", 2024-04-22T06:56:10.174 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "6", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_max_iterations": { 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:56:10.175 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "25", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "250", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_metrics": { 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_metrics", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.176 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_step": { 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_step", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.177 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0.5", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0.001", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "0.999", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "end_time": { 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "end_time", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.178 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "2359", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "end_weekday": { 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "end_weekday", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0", 2024-04-22T06:56:10.179 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "6", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.180 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.181 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.182 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.183 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.184 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "min_score": { 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "min_score", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.185 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": { 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mode", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "upmap", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.186 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "crush-compat", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "none", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "read", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap-read" 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Balancer mode", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "pool_ids": { 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pool_ids", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.187 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.188 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap_max_deviation": { 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "upmap_max_deviation", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.189 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap_max_optimizations": { 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.190 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cephadm", 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.191 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_down_multiplier": { 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_down_multiplier", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3.0", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.192 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_refresh_rate": { 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_refresh_rate", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "20", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_starting_port": { 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_starting_port", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "4721", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.194 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_ptrace": { 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_ptrace", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:56:10.195 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "autotune_interval": { 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "autotune_interval", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.196 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0.7", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.197 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm_log_destination": { 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cephadm_log_destination", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "file", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "file,syslog", 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: "syslog" 2024-04-22T06:56:10.198 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "cgroups_split": { 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cgroups_split", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.199 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "config_checks_enabled": { 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_checks_enabled", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.200 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "config_dashboard": { 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_dashboard", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:56:10.201 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_alertmanager": { 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_alertmanager", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.202 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_base": { 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_base", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:56:10.203 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_elasticsearch": { 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "elasticsearch container image", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.204 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_grafana": { 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_grafana", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.205 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_haproxy": { 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_haproxy", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "HAproxy container image", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.206 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_agent": { 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.207 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_collector": { 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.208 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_query": { 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger query container image", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.209 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_keepalived": { 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_keepalived", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Keepalived container image", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.210 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_loki": { 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_loki", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Loki container image", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.211 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_node_exporter": { 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_node_exporter", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.212 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_nvmeof": { 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_nvmeof", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Nvme-of container image", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.213 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_prometheus": { 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_prometheus", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_promtail": { 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_promtail", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Promtail container image", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.215 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_samba": { 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_samba", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_snmp_gateway": { 2024-04-22T06:56:10.216 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "container_init": { 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_init", 2024-04-22T06:56:10.217 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "daemon_cache_timeout": { 2024-04-22T06:56:10.218 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:56:10.219 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "900", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "default_registry": { 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default_registry", 2024-04-22T06:56:10.220 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "device_cache_timeout": { 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_cache_timeout", 2024-04-22T06:56:10.221 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1800", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "device_enhanced_scan": { 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_enhanced_scan", 2024-04-22T06:56:10.222 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "facts_cache_timeout": { 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "facts_cache_timeout", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.223 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "grafana_dashboards_path": { 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.224 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "host_check_interval": { 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "host_check_interval", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.225 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "hw_monitoring": { 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "hw_monitoring", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.226 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "inventory_list_all": { 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "inventory_list_all", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.227 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.228 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.229 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "log_refresh_metadata": { 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_refresh_metadata", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.230 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.231 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.232 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.233 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.234 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "*", 2024-04-22T06:56:10.235 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "max_count_per_host": { 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_count_per_host", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.236 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "max_osd_draining_count": { 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_osd_draining_count", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.237 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "migration_current": { 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "migration_current", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.238 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "internal - do not modify", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": { 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mode", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "root", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.239 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm-package", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "root" 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "oob_default_addr": { 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "oob_default_addr", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "169.254.1.1", 2024-04-22T06:56:10.240 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "prometheus_alerts_path": { 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:56:10.241 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_insecure": { 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_insecure", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.242 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_password": { 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_password", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.243 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_url": { 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_url", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.244 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_username": { 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_username", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.245 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "secure_monitoring_stack": { 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.246 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "service_discovery_port": { 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "service_discovery_port", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8765", 2024-04-22T06:56:10.247 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "ssh_config_file": { 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssh_config_file", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.248 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "use_agent": { 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "use_agent", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.249 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "use_repo_digest": { 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "use_repo_digest", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.250 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_failed_host_check": { 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.251 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_stray_daemons": { 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.252 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_stray_hosts": { 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:56:10.253 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crash", 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.254 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.255 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.256 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.257 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.258 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "retain_interval": { 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "retain_interval", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.259 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "31536000", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_recent_interval": { 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_recent_interval", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.260 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1209600", 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "dashboard", 2024-04-22T06:56:10.261 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.262 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.263 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.265 INFO:teuthology.orchestra.run.smithi079.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.266 INFO:teuthology.orchestra.run.smithi079.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.271 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.272 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.273 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.274 INFO:teuthology.orchestra.run.smithi079.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.275 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.276 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_URL": { 2024-04-22T06:56:10.277 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:56:10.278 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:56:10.279 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:56:10.280 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:56:10.281 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:56:10.282 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:56:10.283 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "[]", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:56:10.284 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "{}", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:56:10.285 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:56:10.286 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.287 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.288 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.289 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.290 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.291 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.292 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.293 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.294 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.295 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.296 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.297 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.298 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "45", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.299 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.300 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.301 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.302 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.303 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.304 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.305 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.306 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "cross_origin_url": { 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cross_origin_url", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.307 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "crt_file": { 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crt_file", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.308 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "debug": { 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "debug", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.309 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "jwt_token_ttl": { 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "jwt_token_ttl", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "28800", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.310 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "key_file": { 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "key_file", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.311 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.312 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.313 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.315 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.316 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "motd": { 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "motd", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.317 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "The message of the day", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.318 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "::", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.319 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8080", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.320 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl": { 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.321 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl_server_port": { 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl_server_port", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8443", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.322 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_behaviour": { 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_behaviour", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "redirect", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.323 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect" 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_error_status_code": { 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.324 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "500", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "400", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "599", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "url_prefix": { 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "url_prefix", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.325 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "devicehealth", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.326 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_monitoring": { 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enable_monitoring", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "monitor device health metrics", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.327 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.328 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.329 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.330 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.331 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "mark_out_threshold": { 2024-04-22T06:56:10.332 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mark_out_threshold", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "2419200", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.333 INFO:teuthology.orchestra.run.smithi079.stdout: "pool_name": { 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pool_name", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "device_health_metrics", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.334 INFO:teuthology.orchestra.run.smithi079.stdout: "retention_period": { 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "retention_period", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15552000", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.335 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "scrape_frequency": { 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "scrape_frequency", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "86400", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "self_heal": { 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "self_heal", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_threshold": { 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_threshold", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "7257600", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.339 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "diskprediction_local", 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.340 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.341 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.342 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.343 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.344 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "predict_interval": { 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "predict_interval", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "86400", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.345 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "predictor_model": { 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "predictor_model", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "prophetstor", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.346 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.347 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "influx", 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": false, 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "batch_size": { 2024-04-22T06:56:10.348 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "batch_size", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5000", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "database": { 2024-04-22T06:56:10.349 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "database", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "ceph", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.350 INFO:teuthology.orchestra.run.smithi079.stdout: "hostname": { 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "hostname", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.351 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "30", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "5", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.352 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.353 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.354 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.355 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.356 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "password": { 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "password", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.357 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "port": { 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "port", 2024-04-22T06:56:10.358 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8086", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB server port", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl": { 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.359 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "false", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "threads": { 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "threads", 2024-04-22T06:56:10.360 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "32", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "username": { 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "username", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.361 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "verify_ssl": { 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "verify_ssl", 2024-04-22T06:56:10.362 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "true", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.363 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "insights", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.364 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.365 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.366 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.367 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.368 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "iostat", 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.369 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.370 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.371 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.373 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "k8sevents", 2024-04-22T06:56:10.374 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_event_retention_days": { 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "7", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.375 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "config_check_secs": { 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_check_secs", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "10", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.376 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.377 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.378 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.379 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.380 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.381 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "localpool", 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "failure_domain": { 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "failure_domain", 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.382 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "host", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.383 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.384 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.386 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.387 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "min_size": { 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "min_size", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.388 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "num_rep": { 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "num_rep", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.389 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_num": { 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pg_num", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "128", 2024-04-22T06:56:10.390 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "prefix": { 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prefix", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.391 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "subtree": { 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "subtree", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "rack", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.392 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mds_autoscaler", 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.394 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.395 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.396 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.397 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mirroring", 2024-04-22T06:56:10.398 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.399 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.400 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.401 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.402 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.403 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "nfs", 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.404 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.405 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.406 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.407 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.408 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "orchestrator", 2024-04-22T06:56:10.409 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "fail_fs": { 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "fail_fs", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:56:10.410 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.411 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.412 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.413 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.414 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.415 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator": { 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "orchestrator", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "rook", 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: "test_orchestrator" 2024-04-22T06:56:10.416 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Orchestrator backend", 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "osd_perf_query", 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.417 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.418 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.419 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.420 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.421 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.422 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "osd_support", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.423 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.424 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.425 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.426 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.427 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pg_autoscaler", 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.428 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.429 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.430 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.431 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.432 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.433 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "threshold": { 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "threshold", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.434 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3.0", 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1.0", 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "scaling threshold", 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "progress", 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.435 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_pg_recovery_event": { 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": { 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enabled", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.437 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.438 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.439 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.440 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.441 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "max_completed_events": { 2024-04-22T06:56:10.442 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_completed_events", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "50", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.443 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.444 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prometheus", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "cache": { 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cache", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.445 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "exclude_perf_counters": { 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "exclude_perf_counters", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.446 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.447 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.448 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.449 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.450 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.451 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_stats_pools": { 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_stats_pools", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.452 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "300", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.453 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "scrape_interval": { 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "scrape_interval", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15.0", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.454 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "::", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.455 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "9283", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:56:10.456 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "stale_cache_strategy": { 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "stale_cache_strategy", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "log", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.457 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_behaviour": { 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_behaviour", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "default", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "default", 2024-04-22T06:56:10.458 INFO:teuthology.orchestra.run.smithi079.stdout: "error" 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_error_status_code": { 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "500", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "400", 2024-04-22T06:56:10.459 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "599", 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_support", 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.460 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.461 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.462 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.463 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.464 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "max_concurrent_snap_create": { 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.465 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.466 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "trash_purge_schedule": { 2024-04-22T06:56:10.467 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "trash_purge_schedule", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.468 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "restful", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_auth": { 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enable_auth", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.469 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "key_file": { 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "key_file", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.470 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.471 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.472 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.473 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.474 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.475 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.476 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.477 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rgw", 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.478 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.479 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.480 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.481 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.482 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rook", 2024-04-22T06:56:10.483 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.484 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.485 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.486 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.487 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.488 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "storage_class": { 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "storage_class", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "local", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.489 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "selftest", 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.490 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.491 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.492 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.493 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.494 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "roption1": { 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "roption1", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.495 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "roption2": { 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "roption2", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "xyz", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.496 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption1": { 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption1", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.497 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption2": { 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption2", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.498 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption3": { 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption3", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.499 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption4": { 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption4", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.500 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption5": { 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption5", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.501 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption6": { 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption6", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.502 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption7": { 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption7", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "42", 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.503 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "testkey": { 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testkey", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.504 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "testlkey": { 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testlkey", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.505 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "testnewline": { 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testnewline", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.506 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snap_schedule", 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_m_granularity": { 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_m_granularity", 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.507 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "dump_on_update": { 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "dump_on_update", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.508 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.509 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.510 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.511 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.512 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.513 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.514 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "stats", 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.515 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.516 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.517 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.518 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.519 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "status", 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.520 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.521 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.522 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.523 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.524 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.525 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "telegraf", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "address": { 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "address", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.526 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.527 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.528 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.529 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.530 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.531 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.532 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "telemetry", 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_basic": { 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_basic", 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.533 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_crash": { 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_crash", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.534 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_device": { 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_device", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.535 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_ident": { 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_ident", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.536 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_perf": { 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_perf", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.537 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "contact": { 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "contact", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.538 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "description": { 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "description", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.539 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "device_url": { 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_url", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.540 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": { 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enabled", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.541 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "24", 2024-04-22T06:56:10.542 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "8", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "last_opt_revision": { 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "last_opt_revision", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1", 2024-04-22T06:56:10.543 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "leaderboard": { 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "leaderboard", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.544 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "leaderboard_description": { 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "leaderboard_description", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.545 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.546 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.547 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.548 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.549 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.550 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "organization": { 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "organization", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.551 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "proxy": { 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "proxy", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.552 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "url": { 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "url", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.553 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "test_orchestrator", 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.554 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.555 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.558 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "volumes", 2024-04-22T06:56:10.559 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.560 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.561 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.562 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.563 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.564 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "max_concurrent_clones": { 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_concurrent_clones", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "4", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.565 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "periodic_async_work": { 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "periodic_async_work", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Periodically check for async work", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.566 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "snapshot_clone_delay": { 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.567 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.568 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix", 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "discovery_interval": { 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "discovery_interval", 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "100", 2024-04-22T06:56:10.569 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "identifier": { 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "identifier", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.570 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.571 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.573 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.574 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.575 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.576 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_host": { 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_host", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.577 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_port": { 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_port", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10051", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.578 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_sender": { 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_sender", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.579 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "gid": 24824, 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "y", 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr_features": 4540701547738038271, 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: "available_modules": [ 2024-04-22T06:56:10.580 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "alerts", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:56:10.581 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.582 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.583 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.584 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.585 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.586 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_destination": { 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_destination", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:56:10.587 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_from_name": { 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_from_name", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "Ceph", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Email From: name", 2024-04-22T06:56:10.588 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_host": { 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_host", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP server", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.589 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_password": { 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_password", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Password to authenticate with", 2024-04-22T06:56:10.590 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_port": { 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_port", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "465", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP port", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.591 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_sender": { 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_sender", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.592 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_ssl": { 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_ssl", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.593 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_user": { 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_user", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "User to authenticate as", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.594 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "balancer", 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "active": { 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "active", 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.595 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "begin_time": { 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "begin_time", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.596 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0000", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "begin_weekday": { 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "begin_weekday", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.597 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "6", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_max_iterations": { 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.598 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "25", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "250", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_metrics": { 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_metrics", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:56:10.599 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_step": { 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_step", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0.5", 2024-04-22T06:56:10.600 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0.001", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "0.999", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "end_time": { 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "end_time", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "2359", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.601 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "end_weekday": { 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "end_weekday", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0", 2024-04-22T06:56:10.602 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "6", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.603 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.604 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.605 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.606 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.607 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "min_score": { 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "min_score", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.608 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": { 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mode", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "upmap", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "crush-compat", 2024-04-22T06:56:10.609 INFO:teuthology.orchestra.run.smithi079.stdout: "none", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "read", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap-read" 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Balancer mode", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "pool_ids": { 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pool_ids", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.610 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.611 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap_max_deviation": { 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "upmap_max_deviation", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.612 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap_max_optimizations": { 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.613 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cephadm", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.614 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_down_multiplier": { 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_down_multiplier", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3.0", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.615 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_refresh_rate": { 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_refresh_rate", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "20", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.616 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_starting_port": { 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_starting_port", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "4721", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_ptrace": { 2024-04-22T06:56:10.617 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_ptrace", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.618 INFO:teuthology.orchestra.run.smithi079.stdout: "autotune_interval": { 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "autotune_interval", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:56:10.619 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0.7", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm_log_destination": { 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cephadm_log_destination", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.620 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "file", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "file,syslog", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "syslog" 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.621 INFO:teuthology.orchestra.run.smithi079.stdout: "cgroups_split": { 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cgroups_split", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.622 INFO:teuthology.orchestra.run.smithi079.stdout: "config_checks_enabled": { 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_checks_enabled", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "config_dashboard": { 2024-04-22T06:56:10.623 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_dashboard", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_alertmanager": { 2024-04-22T06:56:10.624 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_alertmanager", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_base": { 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_base", 2024-04-22T06:56:10.625 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_elasticsearch": { 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:56:10.626 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "elasticsearch container image", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_grafana": { 2024-04-22T06:56:10.627 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_grafana", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_haproxy": { 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_haproxy", 2024-04-22T06:56:10.628 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "HAproxy container image", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_agent": { 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.629 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_collector": { 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:56:10.630 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_query": { 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.631 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger query container image", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_keepalived": { 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_keepalived", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.632 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Keepalived container image", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_loki": { 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_loki", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.633 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Loki container image", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_node_exporter": { 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_node_exporter", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.634 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_nvmeof": { 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_nvmeof", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.635 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Nvme-of container image", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_prometheus": { 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_prometheus", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.636 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_promtail": { 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_promtail", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.637 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Promtail container image", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_samba": { 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_samba", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.638 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_snmp_gateway": { 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.639 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "container_init": { 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_init", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.640 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "daemon_cache_timeout": { 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.641 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "900", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "default_registry": { 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default_registry", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.643 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "device_cache_timeout": { 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_cache_timeout", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.644 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1800", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "device_enhanced_scan": { 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_enhanced_scan", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.645 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "facts_cache_timeout": { 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "facts_cache_timeout", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.646 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "grafana_dashboards_path": { 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "host_check_interval": { 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "host_check_interval", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.648 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "hw_monitoring": { 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "hw_monitoring", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.649 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "inventory_list_all": { 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "inventory_list_all", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.650 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.651 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.652 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "log_refresh_metadata": { 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_refresh_metadata", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.653 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.654 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.655 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.656 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.657 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.658 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "*", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "max_count_per_host": { 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_count_per_host", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.659 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "max_osd_draining_count": { 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_osd_draining_count", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.660 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "migration_current": { 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "migration_current", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.661 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "internal - do not modify", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": { 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mode", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "root", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.662 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm-package", 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "root" 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "oob_default_addr": { 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "oob_default_addr", 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.663 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "169.254.1.1", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "prometheus_alerts_path": { 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.664 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_insecure": { 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_insecure", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.665 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_password": { 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_password", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.666 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_url": { 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_url", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.667 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_username": { 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_username", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.668 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "secure_monitoring_stack": { 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.669 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "service_discovery_port": { 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "service_discovery_port", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.670 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8765", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "ssh_config_file": { 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssh_config_file", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.671 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "use_agent": { 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "use_agent", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.672 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "use_repo_digest": { 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "use_repo_digest", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.673 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_failed_host_check": { 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.674 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_stray_daemons": { 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.675 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_stray_hosts": { 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.676 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crash", 2024-04-22T06:56:10.677 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.678 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.679 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.680 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.681 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.682 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "retain_interval": { 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "retain_interval", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "31536000", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.683 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_recent_interval": { 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_recent_interval", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1209600", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.684 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "dashboard", 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.685 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.686 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.687 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.688 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.689 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.690 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.691 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.692 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.693 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.694 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.695 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.696 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.697 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.698 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.699 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.700 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.701 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.702 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_URL": { 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.703 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.704 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.705 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.706 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.707 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.708 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "[]", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.709 INFO:teuthology.orchestra.run.smithi079.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "{}", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.710 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.711 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.712 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.713 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.714 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.715 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.716 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.717 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.718 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.719 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.721 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.722 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.723 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "45", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.724 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.725 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.726 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.727 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.728 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.729 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.730 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.731 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "cross_origin_url": { 2024-04-22T06:56:10.732 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cross_origin_url", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "crt_file": { 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crt_file", 2024-04-22T06:56:10.733 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "debug": { 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "debug", 2024-04-22T06:56:10.734 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "jwt_token_ttl": { 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "jwt_token_ttl", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.735 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "28800", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "key_file": { 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "key_file", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.736 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.737 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.738 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.739 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.740 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.741 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "motd": { 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "motd", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.742 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "The message of the day", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.743 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "::", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.744 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8080", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.745 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl": { 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.746 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl_server_port": { 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl_server_port", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8443", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.747 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_behaviour": { 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_behaviour", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "redirect", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.748 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect" 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_error_status_code": { 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "500", 2024-04-22T06:56:10.749 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "400", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "599", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "url_prefix": { 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "url_prefix", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.750 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "devicehealth", 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.751 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_monitoring": { 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enable_monitoring", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "monitor device health metrics", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.752 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.753 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.754 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.755 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.756 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "mark_out_threshold": { 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mark_out_threshold", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.757 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "2419200", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "pool_name": { 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pool_name", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.758 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "device_health_metrics", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "retention_period": { 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "retention_period", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.759 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15552000", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "scrape_frequency": { 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "scrape_frequency", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.760 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "86400", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "self_heal": { 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "self_heal", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.761 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.762 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_threshold": { 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_threshold", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "7257600", 2024-04-22T06:56:10.763 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "diskprediction_local", 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.764 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.765 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.766 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.767 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.768 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "predict_interval": { 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "predict_interval", 2024-04-22T06:56:10.769 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "86400", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "predictor_model": { 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "predictor_model", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.770 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "prophetstor", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.771 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "influx", 2024-04-22T06:56:10.772 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": false, 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "batch_size": { 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "batch_size", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5000", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.773 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "database": { 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "database", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "ceph", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.774 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "hostname": { 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "hostname", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.775 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "30", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "5", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.776 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.777 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.778 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.779 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.780 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "password": { 2024-04-22T06:56:10.781 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "password", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "port": { 2024-04-22T06:56:10.782 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "port", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8086", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB server port", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl": { 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl", 2024-04-22T06:56:10.783 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "false", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "threads": { 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "threads", 2024-04-22T06:56:10.784 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "32", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "username": { 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "username", 2024-04-22T06:56:10.785 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "verify_ssl": { 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "verify_ssl", 2024-04-22T06:56:10.786 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "true", 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:10.787 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "insights", 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.788 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.789 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.790 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.791 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.792 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "iostat", 2024-04-22T06:56:10.793 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.794 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.795 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.796 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.797 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.798 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "k8sevents", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_event_retention_days": { 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "7", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.799 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "config_check_secs": { 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_check_secs", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "10", 2024-04-22T06:56:10.800 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.801 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.802 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.803 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.804 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.805 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "localpool", 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "failure_domain": { 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "failure_domain", 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "host", 2024-04-22T06:56:10.806 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.807 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.808 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.809 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.810 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.811 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "min_size": { 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "min_size", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.812 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "num_rep": { 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "num_rep", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3", 2024-04-22T06:56:10.813 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_num": { 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pg_num", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "128", 2024-04-22T06:56:10.814 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "prefix": { 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prefix", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.815 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "subtree": { 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "subtree", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "rack", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.816 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mds_autoscaler", 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.817 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.818 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.819 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.820 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.821 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.822 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mirroring", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.823 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.824 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.825 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.826 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.827 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "nfs", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.828 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.829 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.830 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.831 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.832 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "orchestrator", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "fail_fs": { 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "fail_fs", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.833 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.834 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.835 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.836 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.837 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.838 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator": { 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "orchestrator", 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.839 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm", 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "rook", 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "test_orchestrator" 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Orchestrator backend", 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.840 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "osd_perf_query", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.841 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.842 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.843 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.844 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.845 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "osd_support", 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.846 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.847 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.848 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.849 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.850 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pg_autoscaler", 2024-04-22T06:56:10.851 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.852 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.853 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.854 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.855 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.856 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.857 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "threshold": { 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "threshold", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3.0", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1.0", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "scaling threshold", 2024-04-22T06:56:10.858 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "progress", 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_pg_recovery_event": { 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.859 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": { 2024-04-22T06:56:10.860 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enabled", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.861 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.862 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.863 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "max_completed_events": { 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_completed_events", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "50", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prometheus", 2024-04-22T06:56:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "cache": { 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cache", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.869 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "exclude_perf_counters": { 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "exclude_perf_counters", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:56:10.870 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.871 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.872 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.873 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.874 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_stats_pools": { 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_stats_pools", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "300", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "scrape_interval": { 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "scrape_interval", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15.0", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "::", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "9283", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "stale_cache_strategy": { 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "stale_cache_strategy", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "log", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_behaviour": { 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_behaviour", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "default", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "default", 2024-04-22T06:56:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: "error" 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_error_status_code": { 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "500", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "400", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "599", 2024-04-22T06:56:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_support", 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.887 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.888 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "max_concurrent_snap_create": { 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:56:10.889 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:56:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "trash_purge_schedule": { 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "trash_purge_schedule", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.891 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.892 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "restful", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_auth": { 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enable_auth", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.893 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "key_file": { 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "key_file", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.894 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.895 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.896 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.897 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.898 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.899 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:10.900 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.901 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rgw", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.902 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.903 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.904 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.905 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.906 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rook", 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.907 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.908 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.909 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.910 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.911 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "storage_class": { 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "storage_class", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.912 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "local", 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.913 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "selftest", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.914 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.915 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.917 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.917 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.917 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.917 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.917 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.917 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.918 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.919 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.920 INFO:teuthology.orchestra.run.smithi079.stdout: "roption1": { 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "roption1", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.921 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "roption2": { 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "roption2", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "xyz", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.922 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption1": { 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption1", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.923 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption2": { 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption2", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.924 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption3": { 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption3", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption4": { 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption4", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.925 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption5": { 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption5", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.926 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption6": { 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption6", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.927 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption7": { 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption7", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.928 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "42", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "testkey": { 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testkey", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.929 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "testlkey": { 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testlkey", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.930 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "testnewline": { 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testnewline", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.931 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.932 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snap_schedule", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_m_granularity": { 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_m_granularity", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.933 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "dump_on_update": { 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "dump_on_update", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.934 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.935 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.936 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.937 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.938 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.939 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "stats", 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.940 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.941 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.942 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.943 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.944 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.945 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "status", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.946 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.947 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.948 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.949 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.950 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "telegraf", 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.951 INFO:teuthology.orchestra.run.smithi079.stdout: "address": { 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "address", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.952 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.953 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.954 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.955 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.956 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.957 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "telemetry", 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_basic": { 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_basic", 2024-04-22T06:56:10.958 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_crash": { 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_crash", 2024-04-22T06:56:10.959 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_device": { 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_device", 2024-04-22T06:56:10.960 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_ident": { 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_ident", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.961 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_perf": { 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_perf", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.962 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "contact": { 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "contact", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.963 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "description": { 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "description", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.964 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "device_url": { 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_url", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.965 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": { 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enabled", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.966 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.967 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "24", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "8", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "last_opt_revision": { 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "last_opt_revision", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.968 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "leaderboard": { 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "leaderboard", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.969 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "leaderboard_description": { 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "leaderboard_description", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.970 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.971 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.972 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.973 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.974 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.975 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "organization": { 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "organization", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.976 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "proxy": { 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "proxy", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.977 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "url": { 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "url", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.978 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "test_orchestrator", 2024-04-22T06:56:10.979 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.980 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.981 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.982 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.983 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.984 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "volumes", 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.988 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:10.989 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "max_concurrent_clones": { 2024-04-22T06:56:10.990 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_concurrent_clones", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "4", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "periodic_async_work": { 2024-04-22T06:56:10.991 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "periodic_async_work", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Periodically check for async work", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.992 INFO:teuthology.orchestra.run.smithi079.stdout: "snapshot_clone_delay": { 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:56:10.993 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.994 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix", 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "discovery_interval": { 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "discovery_interval", 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "100", 2024-04-22T06:56:10.995 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "identifier": { 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "identifier", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.996 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:10.997 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:10.998 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:10.999 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.000 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.001 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.002 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_host": { 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_host", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.003 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_port": { 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_port", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.004 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10051", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_sender": { 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_sender", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.005 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.006 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "modules": [ 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm", 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "dashboard", 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "iostat", 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "nfs", 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "restful", 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "snap_schedule" 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "available_modules": [ 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "alerts", 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.007 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.008 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.009 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.010 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.011 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.012 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.013 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_destination": { 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_destination", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.014 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_from_name": { 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_from_name", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "Ceph", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Email From: name", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.015 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_host": { 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_host", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP server", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.016 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_password": { 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_password", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Password to authenticate with", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.017 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_port": { 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_port", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "465", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP port", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.018 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_sender": { 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_sender", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.019 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_ssl": { 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_ssl", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.020 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "smtp_user": { 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "smtp_user", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "User to authenticate as", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.021 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "balancer", 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "active": { 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "active", 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.022 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "begin_time": { 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "begin_time", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.023 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0000", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "begin_weekday": { 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "begin_weekday", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.024 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "6", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_max_iterations": { 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.025 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "25", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "250", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_metrics": { 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_metrics", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.026 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "crush_compat_step": { 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crush_compat_step", 2024-04-22T06:56:11.027 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0.5", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0.001", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "0.999", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "end_time": { 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "end_time", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.028 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "2359", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "end_weekday": { 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "end_weekday", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:11.029 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "0", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "6", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.030 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.031 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.032 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.033 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.034 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "min_score": { 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "min_score", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:11.035 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": { 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mode", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "upmap", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.036 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "crush-compat", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "none", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "read", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap-read" 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Balancer mode", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "pool_ids": { 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pool_ids", 2024-04-22T06:56:11.037 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:11.038 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap_max_deviation": { 2024-04-22T06:56:11.039 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "upmap_max_deviation", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "upmap_max_optimizations": { 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:56:11.040 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cephadm", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_down_multiplier": { 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_down_multiplier", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3.0", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_refresh_rate": { 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_refresh_rate", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "20", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "agent_starting_port": { 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "agent_starting_port", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "4721", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_ptrace": { 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_ptrace", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.045 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "autotune_interval": { 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "autotune_interval", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.046 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0.7", 2024-04-22T06:56:11.047 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm_log_destination": { 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cephadm_log_destination", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.048 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "file", 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "file,syslog", 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "syslog" 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "cgroups_split": { 2024-04-22T06:56:11.049 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cgroups_split", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "config_checks_enabled": { 2024-04-22T06:56:11.050 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_checks_enabled", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "config_dashboard": { 2024-04-22T06:56:11.051 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_dashboard", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.052 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_alertmanager": { 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_alertmanager", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.053 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_base": { 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_base", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.054 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_elasticsearch": { 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "elasticsearch container image", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.055 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_grafana": { 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_grafana", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.056 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_haproxy": { 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_haproxy", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "HAproxy container image", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.057 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_agent": { 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_collector": { 2024-04-22T06:56:11.058 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_jaeger_query": { 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:56:11.059 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Jaeger query container image", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_keepalived": { 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_keepalived", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.060 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Keepalived container image", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_loki": { 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_loki", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.061 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Loki container image", 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_node_exporter": { 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_node_exporter", 2024-04-22T06:56:11.062 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_nvmeof": { 2024-04-22T06:56:11.063 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_nvmeof", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Nvme-of container image", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_prometheus": { 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_prometheus", 2024-04-22T06:56:11.064 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_promtail": { 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_promtail", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.065 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Promtail container image", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_samba": { 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_samba", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.066 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "container_image_snmp_gateway": { 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.067 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "container_init": { 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "container_init", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.068 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "daemon_cache_timeout": { 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.069 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.070 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "900", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "default_registry": { 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default_registry", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "docker.io", 2024-04-22T06:56:11.071 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "device_cache_timeout": { 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_cache_timeout", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1800", 2024-04-22T06:56:11.072 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "device_enhanced_scan": { 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_enhanced_scan", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.073 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "facts_cache_timeout": { 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "facts_cache_timeout", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.074 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "grafana_dashboards_path": { 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.075 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "host_check_interval": { 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "host_check_interval", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.076 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "hw_monitoring": { 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "hw_monitoring", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.077 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "inventory_list_all": { 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "inventory_list_all", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.078 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.079 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.080 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "log_refresh_metadata": { 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_refresh_metadata", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.081 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.082 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.083 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.084 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.085 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "*", 2024-04-22T06:56:11.086 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "max_count_per_host": { 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_count_per_host", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.087 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "max_osd_draining_count": { 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_osd_draining_count", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.088 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "migration_current": { 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "migration_current", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.089 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "internal - do not modify", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "mode": { 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mode", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "root", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm-package", 2024-04-22T06:56:11.090 INFO:teuthology.orchestra.run.smithi079.stdout: "root" 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "oob_default_addr": { 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "oob_default_addr", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "169.254.1.1", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.091 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "prometheus_alerts_path": { 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.092 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_insecure": { 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_insecure", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.093 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_password": { 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_password", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.094 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_url": { 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_url", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.095 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "registry_username": { 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "registry_username", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.096 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "secure_monitoring_stack": { 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.097 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "service_discovery_port": { 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "service_discovery_port", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8765", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.098 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "ssh_config_file": { 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssh_config_file", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.099 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "use_agent": { 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "use_agent", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.100 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "use_repo_digest": { 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "use_repo_digest", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:56:11.101 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_failed_host_check": { 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:56:11.102 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_stray_daemons": { 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:56:11.103 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_on_stray_hosts": { 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.104 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crash", 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.105 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.107 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.108 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.109 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "retain_interval": { 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "retain_interval", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.110 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "31536000", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_recent_interval": { 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_recent_interval", 2024-04-22T06:56:11.111 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1209600", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.112 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "dashboard", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.113 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.114 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.115 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.116 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.117 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.118 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.119 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.120 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.121 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.122 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.124 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.125 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.126 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.127 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.128 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_URL": { 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.129 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.130 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.131 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.132 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.133 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.134 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "[]", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.135 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "{}", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.136 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.137 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.138 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.139 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.140 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.141 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.142 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.143 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.144 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.145 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.146 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.147 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.148 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.149 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "45", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.150 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.151 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "admin", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.152 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.153 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.154 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.155 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.156 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:56:11.157 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.158 INFO:teuthology.orchestra.run.smithi079.stdout: "cross_origin_url": { 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cross_origin_url", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.159 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "crt_file": { 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "crt_file", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "debug": { 2024-04-22T06:56:11.160 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "debug", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "jwt_token_ttl": { 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "jwt_token_ttl", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.161 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "28800", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "key_file": { 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "key_file", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.162 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.163 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.164 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.165 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.166 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.167 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "motd": { 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "motd", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.168 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "The message of the day", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.169 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "::", 2024-04-22T06:56:11.170 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8080", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.171 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl": { 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.172 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl_server_port": { 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl_server_port", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8443", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.173 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_behaviour": { 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_behaviour", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "redirect", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.174 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect" 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_error_status_code": { 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.175 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "500", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "400", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "599", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "url_prefix": { 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "url_prefix", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.176 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "devicehealth", 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.177 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_monitoring": { 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enable_monitoring", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "monitor device health metrics", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.178 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.179 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.180 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.181 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.182 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.183 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "mark_out_threshold": { 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mark_out_threshold", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "2419200", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.184 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "pool_name": { 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pool_name", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "device_health_metrics", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.185 INFO:teuthology.orchestra.run.smithi079.stdout: "retention_period": { 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "retention_period", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15552000", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:56:11.186 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "scrape_frequency": { 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "scrape_frequency", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "86400", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:56:11.187 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "self_heal": { 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "self_heal", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.188 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:56:11.189 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "warn_threshold": { 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "warn_threshold", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "7257600", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:56:11.190 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "diskprediction_local", 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.191 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.192 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.193 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.194 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.195 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "predict_interval": { 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "predict_interval", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "86400", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.196 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "predictor_model": { 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "predictor_model", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "prophetstor", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.197 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "600", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.198 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "influx", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": false, 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "batch_size": { 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "batch_size", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5000", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.199 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "database": { 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "database", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "ceph", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.200 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "hostname": { 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "hostname", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.201 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "30", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "5", 2024-04-22T06:56:11.202 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.203 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.204 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.205 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.206 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.207 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "password": { 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "password", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.208 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "port": { 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "port", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "8086", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.209 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "InfluxDB server port", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "ssl": { 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ssl", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "false", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.210 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "threads": { 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "threads", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:11.211 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "32", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "username": { 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "username", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.212 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "verify_ssl": { 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "verify_ssl", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "true", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:11.213 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "insights", 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.214 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.215 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.216 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.217 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.218 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "iostat", 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.219 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.220 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.221 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.222 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.223 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "k8sevents", 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.224 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_event_retention_days": { 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "7", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "config_check_secs": { 2024-04-22T06:56:11.225 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "config_check_secs", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "10", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.226 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.227 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.228 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.229 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.230 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "localpool", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "failure_domain": { 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "failure_domain", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "host", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.231 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.232 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.233 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.234 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "min_size": { 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "min_size", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "num_rep": { 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "num_rep", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_num": { 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pg_num", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "128", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "prefix": { 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prefix", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.240 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "subtree": { 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "subtree", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "rack", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.241 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mds_autoscaler", 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.242 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.243 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.244 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.245 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.246 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mirroring", 2024-04-22T06:56:11.247 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.250 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.251 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.252 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "nfs", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.253 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.254 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.255 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.256 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.257 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "orchestrator", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "fail_fs": { 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "fail_fs", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.258 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.259 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.260 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.261 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.262 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.263 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator": { 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "orchestrator", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "cephadm", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "rook", 2024-04-22T06:56:11.264 INFO:teuthology.orchestra.run.smithi079.stdout: "test_orchestrator" 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Orchestrator backend", 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "osd_perf_query", 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.265 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.266 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.267 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.268 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.269 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.270 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "osd_support", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.271 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.272 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.274 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.275 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "pg_autoscaler", 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.276 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.277 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.278 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.279 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.280 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.281 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.282 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "threshold": { 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "threshold", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "3.0", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1.0", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "scaling threshold", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.283 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "progress", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_pg_recovery_event": { 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.284 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": { 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enabled", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.285 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.286 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.287 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.288 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.289 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "max_completed_events": { 2024-04-22T06:56:11.290 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_completed_events", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "50", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "sleep_interval": { 2024-04-22T06:56:11.291 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "sleep_interval", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "5", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.292 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "prometheus", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "cache": { 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "cache", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.293 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "exclude_perf_counters": { 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "exclude_perf_counters", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.294 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.295 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.296 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.297 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.298 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.299 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_stats_pools": { 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_stats_pools", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.300 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "300", 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.301 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "scrape_interval": { 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "scrape_interval", 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15.0", 2024-04-22T06:56:11.302 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "::", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.303 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "9283", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.304 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "stale_cache_strategy": { 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "stale_cache_strategy", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "log", 2024-04-22T06:56:11.305 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_behaviour": { 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_behaviour", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "default", 2024-04-22T06:56:11.306 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "default", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "error" 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "standby_error_status_code": { 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.307 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "500", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "400", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "599", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_support", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.308 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.309 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.310 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.311 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.312 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.313 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.314 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.314 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.314 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.314 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.314 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.314 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "max_concurrent_snap_create": { 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.315 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.316 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "trash_purge_schedule": { 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "trash_purge_schedule", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.317 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "restful", 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "enable_auth": { 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enable_auth", 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.318 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "key_file": { 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "key_file", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.319 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.320 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.321 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.322 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.323 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.324 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "server_addr": { 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_addr", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.325 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "server_port": { 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "server_port", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.326 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rgw", 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.327 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.328 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.329 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.330 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.331 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.332 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rook", 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.333 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.334 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.335 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.336 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.337 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.338 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.339 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "storage_class": { 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "storage_class", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.340 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "local", 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.341 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "selftest", 2024-04-22T06:56:11.342 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.342 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.342 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.342 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.342 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.342 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.343 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.344 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.345 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.346 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.346 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.346 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.346 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.346 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.347 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.347 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.347 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.347 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.347 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.347 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.348 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.349 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "roption1": { 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "roption1", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.350 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "roption2": { 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "roption2", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "xyz", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.351 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption1": { 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption1", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.352 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption2": { 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption2", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.353 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption3": { 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption3", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "float", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.354 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption4": { 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption4", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption5": { 2024-04-22T06:56:11.355 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption5", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.356 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption6": { 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption6", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.357 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "rwoption7": { 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rwoption7", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "1", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "42", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.358 INFO:teuthology.orchestra.run.smithi079.stdout: "testkey": { 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testkey", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "testlkey": { 2024-04-22T06:56:11.359 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testlkey", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "testnewline": { 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "testnewline", 2024-04-22T06:56:11.360 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snap_schedule", 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.361 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "allow_m_granularity": { 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "allow_m_granularity", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.362 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "dump_on_update": { 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "dump_on_update", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.363 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.364 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.365 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.366 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.367 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.368 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "stats", 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.369 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.370 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.371 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.372 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.373 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.374 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.375 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.376 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "status", 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.377 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.378 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.379 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.380 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.380 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.380 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.380 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.380 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.381 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.381 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.381 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.381 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.381 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.381 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.382 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.383 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.384 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "telegraf", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "address": { 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "address", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.385 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "15", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.386 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.387 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.388 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.389 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.390 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.391 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "telemetry", 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.392 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_basic": { 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_basic", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_crash": { 2024-04-22T06:56:11.393 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_crash", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_device": { 2024-04-22T06:56:11.394 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_device", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_ident": { 2024-04-22T06:56:11.395 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_ident", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.396 INFO:teuthology.orchestra.run.smithi079.stdout: "channel_perf": { 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "channel_perf", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.397 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "contact": { 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "contact", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.398 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "description": { 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "description", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.399 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "device_url": { 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "device_url", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.400 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": { 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "enabled", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.401 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "24", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "8", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.402 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "last_opt_revision": { 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "last_opt_revision", 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "1", 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.403 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "leaderboard": { 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "leaderboard", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.404 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "leaderboard_description": { 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "leaderboard_description", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.405 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.406 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.407 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.408 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.409 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.410 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.411 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "organization": { 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "organization", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.412 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "proxy": { 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "proxy", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.413 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "url": { 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "url", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.414 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.415 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "test_orchestrator", 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.416 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.417 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.418 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.419 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.420 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.421 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.422 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "volumes", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.423 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.424 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.427 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "max_concurrent_clones": { 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "max_concurrent_clones", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "4", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.428 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "periodic_async_work": { 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "periodic_async_work", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.429 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Periodically check for async work", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "snapshot_clone_delay": { 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "0", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.430 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "True", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.431 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix", 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "can_run": true, 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "error_string": "", 2024-04-22T06:56:11.432 INFO:teuthology.orchestra.run.smithi079.stdout: "module_options": { 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "discovery_interval": { 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "discovery_interval", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "uint", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "100", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.433 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "identifier": { 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "identifier", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.434 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "interval": { 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "interval", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "secs", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "60", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.435 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "log_level": { 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_level", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.436 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster": { 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.437 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_cluster_level": { 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "info", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.438 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [ 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "critical", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "debug", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "error", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "info", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "warning" 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.439 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "log_to_file": { 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "log_to_file", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "bool", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 1, 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "False", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.440 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_host": { 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_host", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.441 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_port": { 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_port", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "int", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "10051", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.442 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "zabbix_sender": { 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "zabbix_sender", 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "str", 2024-04-22T06:56:11.443 INFO:teuthology.orchestra.run.smithi079.stdout: "level": "advanced", 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "flags": 0, 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "min": "", 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "max": "", 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "enum_allowed": [], 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "desc": "", 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "long_desc": "", 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "see_also": [] 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.444 INFO:teuthology.orchestra.run.smithi079.stdout: "services": { 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "dashboard": "https://172.21.15.79:8443/" 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "always_on_modules": { 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "octopus": [ 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "balancer", 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "crash", 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "devicehealth", 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator", 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_autoscaler", 2024-04-22T06:56:11.445 INFO:teuthology.orchestra.run.smithi079.stdout: "progress", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_support", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "status", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "telemetry", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "volumes" 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "pacific": [ 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "balancer", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "crash", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "devicehealth", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_autoscaler", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "progress", 2024-04-22T06:56:11.446 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_support", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "status", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "telemetry", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "volumes" 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "quincy": [ 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "balancer", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "crash", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "devicehealth", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_autoscaler", 2024-04-22T06:56:11.447 INFO:teuthology.orchestra.run.smithi079.stdout: "progress", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_support", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "status", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "telemetry", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "volumes" 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "reef": [ 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "balancer", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "crash", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "devicehealth", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_autoscaler", 2024-04-22T06:56:11.448 INFO:teuthology.orchestra.run.smithi079.stdout: "progress", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_support", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "status", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "telemetry", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "volumes" 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "squid": [ 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "balancer", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "crash", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "devicehealth", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "orchestrator", 2024-04-22T06:56:11.449 INFO:teuthology.orchestra.run.smithi079.stdout: "pg_autoscaler", 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "progress", 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "rbd_support", 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "status", 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "telemetry", 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "volumes" 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "last_failure_osd_epoch": 88, 2024-04-22T06:56:11.450 INFO:teuthology.orchestra.run.smithi079.stdout: "active_clients": [ 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "devicehealth", 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:0", 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 565277869 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.451 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "libcephsqlite", 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:0", 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 3428829738 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.452 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "rbd_support", 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:0", 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 3466473406 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.453 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snap_schedule", 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:0", 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 28248081 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "snap_schedule", 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:11.454 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:0", 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 4207854255 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "volumes", 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: "addrvec": [ 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: "type": "v2", 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: "addr": "172.21.15.79:0", 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: "nonce": 1377606748 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-04-22T06:56:11.455 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:56:11.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:10 smithi193 ceph-mon[31815]: from='client.17694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:11.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:10 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1770732232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:11.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:10 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4053683567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:11.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:10 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1459603239' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:56:11.457 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:10 smithi190 ceph-mon[31421]: from='client.17694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:11.457 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:10 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1770732232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:11.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:10 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4053683567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:11.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:10 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1459603239' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:56:11.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:11 smithi190 ceph-mon[31421]: pgmap v383: 97 pgs: 97 active+clean; 20 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:56:11.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88680a3d50 msgr2=0x7f88680a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:11.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88680a3d50 0x7f88680a6210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f886c005e00 tx=0x7f886c005c70 comp rx=0 tx=0).stop 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 msgr2=0x7f887c100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 0x7f887c100e80 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f886400c920 tx=0x7f886400cde0 comp rx=0 tx=0).stop 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 shutdown_connections 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f88680a3d50 0x7f88680a6210 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f887c10c570 0x7f887c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f887c103b50 0x7f887c1013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 --2- 172.21.15.79:0/1459603239 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f887c103230 0x7f887c100e80 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 >> 172.21.15.79:0/1459603239 conn(0x7f887c077fb0 msgr2=0x7f887c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 shutdown_connections 2024-04-22T06:56:11.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:10.153+0000 7f88826eb640 1 -- 172.21.15.79:0/1459603239 wait complete. 2024-04-22T06:56:11.466 DEBUG:tasks.check_counter:Getting stats from a 2024-04-22T06:56:11.466 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-22T06:56:11.466 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:10 smithi079 ceph-mon[27946]: from='client.17694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:11.466 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1770732232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:11.466 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4053683567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:11.466 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:10 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1459603239' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:56:11.489 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:11 smithi079 ceph-mon[27946]: pgmap v383: 97 pgs: 97 active+clean; 20 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:56:11.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:11 smithi193 ceph-mon[31815]: pgmap v383: 97 pgs: 97 active+clean; 20 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:56:11.795 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:12.490 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:12 smithi079 ceph-mon[27946]: pgmap v384: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-22T06:56:12.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:12 smithi193 ceph-mon[31815]: pgmap v384: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-22T06:56:12.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:12 smithi190 ceph-mon[31421]: pgmap v384: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-22T06:56:14.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:14 smithi193 ceph-mon[31815]: pgmap v385: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 25 KiB/s wr, 103 op/s 2024-04-22T06:56:14.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:14 smithi190 ceph-mon[31421]: pgmap v385: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 25 KiB/s wr, 103 op/s 2024-04-22T06:56:14.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:14 smithi079 ceph-mon[27946]: pgmap v385: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 25 KiB/s wr, 103 op/s 2024-04-22T06:56:15.221 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:14 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:56:15.221 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:14 smithi193 systemd[1]: Stopped Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:56:15.222 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:14 smithi193 systemd[1]: Starting Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:56:15.222 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 podman[69793]: 2024-04-22 06:56:15.082623566 +0000 UTC m=+0.018556891 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:56:15.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:15 smithi193 ceph-mon[31815]: osdmap e117: 12 total, 12 up, 12 in 2024-04-22T06:56:15.500 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 podman[69793]: 2024-04-22 06:56:15.224401377 +0000 UTC m=+0.160334703 container create 0bbcacc172bdf61c59f511c804c5de427223b56c8b5791dbc626887ae2a141c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:56:15.500 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 podman[69793]: 2024-04-22 06:56:15.427272544 +0000 UTC m=+0.363205874 container init 0bbcacc172bdf61c59f511c804c5de427223b56c8b5791dbc626887ae2a141c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD) 2024-04-22T06:56:15.500 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 podman[69793]: 2024-04-22 06:56:15.429952605 +0000 UTC m=+0.365885937 container start 0bbcacc172bdf61c59f511c804c5de427223b56c8b5791dbc626887ae2a141c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-22T06:56:15.500 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 ceph-mds[69812]: -- 172.21.15.193:0/801156564 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556d6565a40 con 0x5556d6538800 2024-04-22T06:56:15.500 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[69808]: starting mds.c at 2024-04-22T06:56:15.500 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 bash[69793]: 0bbcacc172bdf61c59f511c804c5de427223b56c8b5791dbc626887ae2a141c6 2024-04-22T06:56:15.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:15 smithi190 ceph-mon[31421]: osdmap e117: 12 total, 12 up, 12 in 2024-04-22T06:56:15.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:15 smithi079 ceph-mon[27946]: osdmap e117: 12 total, 12 up, 12 in 2024-04-22T06:56:15.939 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:15 smithi193 systemd[1]: Started Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:56:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:16 smithi193 ceph-mon[31815]: pgmap v387: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 25 KiB/s wr, 136 op/s 2024-04-22T06:56:16.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:16 smithi193 ceph-mon[31815]: daemon mds.c restarted 2024-04-22T06:56:16.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:16 smithi190 ceph-mon[31421]: pgmap v387: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 25 KiB/s wr, 136 op/s 2024-04-22T06:56:16.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:16 smithi190 ceph-mon[31421]: daemon mds.c restarted 2024-04-22T06:56:16.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:16 smithi079 ceph-mon[27946]: pgmap v387: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 25 KiB/s wr, 136 op/s 2024-04-22T06:56:16.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:16 smithi079 ceph-mon[27946]: daemon mds.c restarted 2024-04-22T06:56:17.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:17 smithi193 ceph-mon[31815]: osdmap e118: 12 total, 12 up, 12 in 2024-04-22T06:56:17.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:17 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6836/2488235874,v1:172.21.15.193:6837/2488235874] up:boot 2024-04-22T06:56:17.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:17 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:56:17.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:17 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:17.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:17 smithi193 ceph-mon[31815]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:56:17.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:17 smithi190 ceph-mon[31421]: osdmap e118: 12 total, 12 up, 12 in 2024-04-22T06:56:17.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:17 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6836/2488235874,v1:172.21.15.193:6837/2488235874] up:boot 2024-04-22T06:56:17.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:17 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:56:17.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:17 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:17.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:17 smithi190 ceph-mon[31421]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:56:17.736 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:17 smithi079 ceph-mon[27946]: osdmap e118: 12 total, 12 up, 12 in 2024-04-22T06:56:17.737 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:17 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6836/2488235874,v1:172.21.15.193:6837/2488235874] up:boot 2024-04-22T06:56:17.737 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:17 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:56:17.737 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:17 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:17.737 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:17 smithi079 ceph-mon[27946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:56:18.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:18 smithi079 ceph-mon[27946]: pgmap v389: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 38 KiB/s wr, 317 op/s 2024-04-22T06:56:18.685 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:56:18.685 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:18.685 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:18.685 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:18.685 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.001039726, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.000097717, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:18.686 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 241, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 228, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 1597008, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 139392, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.051943863, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.016052211, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.038954093, 2024-04-22T06:56:18.687 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.006725512, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 228, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.021322664, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000093520 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 1597008, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 139392 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:18.688 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 173, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 186560, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.015279604, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.010594542, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.002211049, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 170, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.026675769, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000156916 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:18.689 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 186560 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 0, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:18.690 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:18.691 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:18.692 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 148955, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 30, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:18.693 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 2, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 239, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 1560574, 2024-04-22T06:56:18.694 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 239, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 239, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 1560574, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:18.695 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:56:18.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:18 smithi193 ceph-mon[31815]: pgmap v389: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 38 KiB/s wr, 317 op/s 2024-04-22T06:56:18.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:18 smithi190 ceph-mon[31421]: pgmap v389: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 38 KiB/s wr, 317 op/s 2024-04-22T06:56:19.400 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-22T06:56:19.400 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-22T06:56:19.400 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-22T06:56:19.400 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-22T06:56:19.400 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-22T06:56:19.400 DEBUG:tasks.check_counter:Getting stats from d 2024-04-22T06:56:19.401 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:20.813 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:20 smithi079 ceph-mon[27946]: pgmap v390: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8.0 KiB/s wr, 287 op/s 2024-04-22T06:56:20.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:20 smithi193 ceph-mon[31815]: pgmap v390: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8.0 KiB/s wr, 287 op/s 2024-04-22T06:56:20.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:20 smithi190 ceph-mon[31421]: pgmap v390: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8.0 KiB/s wr, 287 op/s 2024-04-22T06:56:21.370 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 4, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 4, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 2992, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 448, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.001464820, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.000028245, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.000613363, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:21.371 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 3, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000658472, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000219490 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:21.372 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 243, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 232, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 1595648, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 142784, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.049452484, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.016368183, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.035488734, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.006663717, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 231, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.027814507, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000120409 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.373 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 1595648, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 142784 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 178, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 187440, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:21.374 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.014658080, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.010176373, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.002067031, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 175, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.022262766, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000127215 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 187440 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:21.376 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:21.377 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:21.378 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 149057, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 32, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:21.379 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 2, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:21.380 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 245, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 1561354, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 245, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 245, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 1561354, 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-04-22T06:56:21.381 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:21.382 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:21.382 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:56:22.059 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-22T06:56:22.059 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-22T06:56:22.059 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-22T06:56:22.059 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-22T06:56:22.059 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-22T06:56:22.060 DEBUG:tasks.check_counter:Getting stats from g 2024-04-22T06:56:22.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-22T06:56:22.296 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:22.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:22 smithi190 ceph-mon[31421]: pgmap v391: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.4 KiB/s wr, 212 op/s 2024-04-22T06:56:22.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:22 smithi079 ceph-mon[27946]: pgmap v391: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.4 KiB/s wr, 212 op/s 2024-04-22T06:56:22.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:22 smithi193 ceph-mon[31815]: pgmap v391: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.4 KiB/s wr, 212 op/s 2024-04-22T06:56:24.549 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:24 smithi079 ceph-mon[27946]: pgmap v392: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.6 KiB/s wr, 155 op/s 2024-04-22T06:56:24.549 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.001107481, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.000108163, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:24.550 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:24.551 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 6, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 176, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 3312, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 187184, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.015050692, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.010427697, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.002086231, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 174, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.017205652, 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000098883 2024-04-22T06:56:24.552 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 187184 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 239, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 228, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 1594560, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 140224, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:24.553 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.057030202, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.015908512, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.036011073, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.013660496, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 227, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.015341882, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000067585 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.554 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 1594560, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 140224 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 0, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:24.555 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:24.556 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:24.557 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 148995, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 31, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 0, 2024-04-22T06:56:24.558 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 2, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-04-22T06:56:24.559 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 241, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 1560834, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 241, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 241, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 1560834, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:24.560 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:56:24.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:24 smithi193 ceph-mon[31815]: pgmap v392: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.6 KiB/s wr, 155 op/s 2024-04-22T06:56:24.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:24 smithi190 ceph-mon[31421]: pgmap v392: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.6 KiB/s wr, 155 op/s 2024-04-22T06:56:25.253 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-22T06:56:25.253 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-22T06:56:25.253 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-22T06:56:25.253 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-22T06:56:25.253 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-22T06:56:25.253 DEBUG:tasks.check_counter:Getting stats from j 2024-04-22T06:56:25.253 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-22T06:56:25.500 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:26.660 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:26 smithi079 ceph-mon[27946]: pgmap v393: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.9 KiB/s wr, 138 op/s 2024-04-22T06:56:26.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:26 smithi193 ceph-mon[31815]: pgmap v393: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.9 KiB/s wr, 138 op/s 2024-04-22T06:56:26.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:26 smithi190 ceph-mon[31421]: pgmap v393: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 5.9 KiB/s wr, 138 op/s 2024-04-22T06:56:27.216 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 249, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 236, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 1599184, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 144512, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.049148351, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.015537466, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.036332998, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.006184934, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 236, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.017995378, 2024-04-22T06:56:27.217 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000076251 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 1599184, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 144512 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 179, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 187616, 2024-04-22T06:56:27.218 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.015096158, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.010970237, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.001602115, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 176, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.022448563, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000127548 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:27.219 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 187616 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_total_time": 0.000264929, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_recv_time": 0.000194216, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:27.220 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "cct": { 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "total_workers": 0, 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: "unhealthy_workers": 0 2024-04-22T06:56:27.221 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "mempool": { 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:27.222 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:27.223 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_items": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_bytes": 148955, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_anon_items": 30, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_items": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:27.224 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_items": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "pgmap_items": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "mds_co_items": 2, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_1_items": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: "unittest_2_items": 0 2024-04-22T06:56:27.225 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 0, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "max": 104857600, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "get_started": 0, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "get": 247, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "get_sum": 1561614, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "get_or_fail_success": 247, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "take": 0, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "take_sum": 0, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "put": 247, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "put_sum": 1561614, 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "wait": { 2024-04-22T06:56:27.226 INFO:teuthology.orchestra.run.smithi079.stdout: "avgcount": 0, 2024-04-22T06:56:27.227 INFO:teuthology.orchestra.run.smithi079.stdout: "sum": 0.000000000, 2024-04-22T06:56:27.227 INFO:teuthology.orchestra.run.smithi079.stdout: "avgtime": 0.000000000 2024-04-22T06:56:27.227 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:27.227 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-04-22T06:56:27.227 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-04-22T06:56:27.941 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-22T06:56:27.941 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-22T06:56:27.941 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-22T06:56:27.941 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-22T06:56:27.941 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-22T06:56:27.941 DEBUG:tasks.check_counter:Getting stats from b 2024-04-22T06:56:27.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-22T06:56:28.215 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:28.590 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:28 smithi190 ceph-mon[31421]: pgmap v394: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 22 op/s 2024-04-22T06:56:28.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:28 smithi193 ceph-mon[31815]: pgmap v394: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 22 op/s 2024-04-22T06:56:28.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:28 smithi079 ceph-mon[27946]: pgmap v394: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.1 KiB/s wr, 22 op/s 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 7387, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 6619, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 2135186, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 309997480, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 147, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 9, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 2.352203768, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 1.109919043, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.221662421, 2024-04-22T06:56:29.965 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.965622132, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 6589, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.997023721, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000151316 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1794, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000347471, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000193 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 409568, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 114896 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.966 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 3568, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 3455, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 2877147, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 159812343, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 74, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 2, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 1.016785285, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.576295090, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.142849166, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.294083284, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3438, 2024-04-22T06:56:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.804547363, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000234016 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 889, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000213979, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000240 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 2000064, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 547696 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 200070, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 202523, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 73391786, 2024-04-22T06:56:29.968 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 192357598, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 37, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 12.659892109, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 5.482594880, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 15.967610532, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.700405906, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 202466, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 23.189991071, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000114537 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 328759, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.077905208, 2024-04-22T06:56:29.969 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000236 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 71801296, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 79158944 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4248, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 154.417729787, 2024-04-22T06:56:29.970 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.036350689 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2414, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.082568517, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000034204 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 316, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 316, 2024-04-22T06:56:29.971 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 316, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 16.908547165, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.053508060 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 162, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 128, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 45, 2024-04-22T06:56:29.972 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 83, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 63, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 3, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 4, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 2, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 472, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 463, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-04-22T06:56:29.973 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 167, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 4, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 234, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 6, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 167, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:29.974 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 67, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 7, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 60, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 100, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 177, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 3213, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 193960, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 193798, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 13, 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 49 2024-04-22T06:56:29.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 22, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 4, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 22, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 18, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:29.976 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 11, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 246, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 7, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-04-22T06:56:29.977 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 194646, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 194593, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 194593, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 53, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 8, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 297, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 296, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 296, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 1, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 1, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 1, 2024-04-22T06:56:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 673746270, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 674051812, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 8735, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 5636.099110326, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.645231724 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 100, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 125, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 25, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 22, 2024-04-22T06:56:29.979 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 22, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 0, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 128, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 146, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 18, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 4, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 62, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 58, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 79340, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 231800 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 413, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 19, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 316, 2024-04-22T06:56:29.980 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 36, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 35, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 22, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.024874837, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001130674 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 143, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 4.312776954, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.030159279 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.981 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 70, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 7.238260761, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.103403725 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.982 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000180882, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000180882 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.983 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.984 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 12, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.003040283, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000253356 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 10, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.014824462, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001482446 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.985 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 12, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 5.078814762, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.423234563 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000854729, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000427364 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-04-22T06:56:29.986 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 5, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.012067375, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002413475 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.001926636, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000963318 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.987 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.033178133, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.008294533 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 6, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.143767006, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.023961167 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.988 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000863770, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000863770 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 8, 2024-04-22T06:56:29.990 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.018334729, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002291841 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 18, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.024781846, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001376769 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:29.991 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 2, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 6, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 4, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 2, 2024-04-22T06:56:29.992 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 58, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 29, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 706, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:29.994 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:29.995 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 344325, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 149, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 10456, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 258, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:29.996 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 376830, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 2735, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 12971, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 670542699, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 37, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 12971, 2024-04-22T06:56:29.997 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 12971, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 4030.183078893, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.310707199 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 12971, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 13382 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 12971, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 182, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 12789, 2024-04-22T06:56:29.998 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 140, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 44, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 8, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 8736, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 173, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 3408, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:29.999 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 174, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 88, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-04-22T06:56:30.000 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 118, 2024-04-22T06:56:30.001 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 34, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 12, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 12, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 594, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 0, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 17 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-04-22T06:56:30.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 276 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 519, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 7, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 2578, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 18, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:30.003 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 211021, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 51144689, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 211021, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 211021, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 51144689, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.004 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 12971, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 670545381, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 12971, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 670545381, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:30.005 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 12971, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 12971, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 12971, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 12971, 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:30.006 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 18, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 2682, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 18, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1, 2024-04-22T06:56:30.007 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 2682, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x55ca330c6500": { 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 194646, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 669857508, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 194646, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:30.008 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 8735, 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 669857508, 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:30.009 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-22T06:56:30.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:30 smithi190 ceph-mon[31421]: pgmap v395: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1023 B/s wr, 17 op/s 2024-04-22T06:56:30.763 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-22T06:56:30.763 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-22T06:56:30.763 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-22T06:56:30.764 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2024-04-22T06:56:30.764 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=6 2024-04-22T06:56:30.764 DEBUG:tasks.check_counter:Getting stats from e 2024-04-22T06:56:30.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-22T06:56:30.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:30 smithi193 ceph-mon[31815]: pgmap v395: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1023 B/s wr, 17 op/s 2024-04-22T06:56:30.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:30 smithi079 ceph-mon[27946]: pgmap v395: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1023 B/s wr, 17 op/s 2024-04-22T06:56:31.011 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:32.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:32 smithi190 ceph-mon[31421]: pgmap v396: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 71, 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 68, 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 181672, 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 24328, 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:32.760 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.008715923, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.002269555, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.004461946, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.001824857, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 67, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.003627970, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000054148 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 170848, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 10336 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 8, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 18, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 4500, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 105493, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.002612095, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.000677099, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.001652984, 2024-04-22T06:56:32.762 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000077606, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 17, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002868949, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000168761 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 104528 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:32.763 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 91, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 93, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 31480, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 28996, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.010014501, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.003129655, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.003263321, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.002811506, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 91, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.005277076, 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000057989 2024-04-22T06:56:32.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-04-22T06:56:32.765 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 223, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.006550325, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000029373 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 0, 2024-04-22T06:56:32.767 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 1, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-04-22T06:56:32.768 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 0, 2024-04-22T06:56:32.769 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:32.770 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:32.771 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-04-22T06:56:32.772 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 18, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-04-22T06:56:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 4198635, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 4198467, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 18 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-04-22T06:56:32.774 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 3, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 3, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 1, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 1, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 0, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 42208, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 215416 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.775 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.776 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.777 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.778 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-04-22T06:56:32.780 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.781 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.783 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.784 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.785 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:32.786 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 2, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 5, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 3, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 4, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 0, 2024-04-22T06:56:32.787 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 3803, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:32.788 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:32.789 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 188496, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 47, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:32.790 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 10456, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 258, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 8512, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 13, 2024-04-22T06:56:32.791 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 138, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 21, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 0, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 138, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 138, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.083997755, 2024-04-22T06:56:32.792 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000608679 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 138, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 139 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 138, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 138, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 0, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 88, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 0, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 48, 2024-04-22T06:56:32.793 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 1, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:32.794 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-04-22T06:56:32.795 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 118, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 3, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 5, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 5, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-04-22T06:56:32.796 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 0, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 2, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 0 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 0, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:32.797 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 166, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 198366, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:32.798 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 166, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 166, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 198366, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:32.799 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 138, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 4336, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 138, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 4336, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:32.800 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 138, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 138, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 138, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 138, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-04-22T06:56:32.802 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x56527dde6780": { 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:32.803 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:32.804 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-22T06:56:32.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:32 smithi193 ceph-mon[31815]: pgmap v396: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:56:32.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:32 smithi079 ceph-mon[27946]: pgmap v396: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:56:33.593 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-22T06:56:33.593 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-22T06:56:33.593 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-22T06:56:33.593 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:33.593 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:33.593 DEBUG:tasks.check_counter:Getting stats from h 2024-04-22T06:56:33.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-22T06:56:33.833 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:34.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:34 smithi193 ceph-mon[31815]: pgmap v397: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:56:34.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:34 smithi190 ceph-mon[31421]: pgmap v397: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:56:34.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:34 smithi079 ceph-mon[27946]: pgmap v397: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:56:35.564 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 200, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 186, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 270442, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 76697, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 8, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.024259458, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.008345027, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.009797323, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.004792561, 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 185, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.013500270, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000072974 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 8, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000001349, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000168 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 222672, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 32128 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:35.566 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 46, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 95, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 38977, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 184653, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 8, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.009174807, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.003783686, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.004344625, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000547463, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 95, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.008641641, 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000090964 2024-04-22T06:56:35.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 74, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000015111, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000204 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 145296 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 240, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 248, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 70586, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 77645, 2024-04-22T06:56:35.568 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 8, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.019399360, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.003703057, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.007416582, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.006941838, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 245, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.009580979, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000039106 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 12, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000002565, 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000213 2024-04-22T06:56:35.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 2720 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 575, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.016358806, 2024-04-22T06:56:35.570 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000028450 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000085649, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000021412 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 0, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 0, 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2024-04-22T06:56:35.571 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 11, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 10, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 10, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 7, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 3, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:35.572 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 10, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 1, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2024-04-22T06:56:35.573 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 22, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 10738142177, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 6, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:35.574 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:35.575 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 1, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2024-04-22T06:56:35.576 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "evlrg": 0, 2024-04-22T06:56:35.577 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 1, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 20, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 2, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "segmjr": 2, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 4194304, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 4199219, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 4198883, 2024-04-22T06:56:35.578 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.002456482, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002456482 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 19 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 13, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 14, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 1, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 11, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 12, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 1, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 10, 2024-04-22T06:56:35.579 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 10, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 50504, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 239992 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 1, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2024-04-22T06:56:35.580 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.581 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.582 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.583 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2024-04-22T06:56:35.584 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.585 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.585 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.585 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2024-04-22T06:56:35.585 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.586 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.587 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.592 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.593 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 2, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 6, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 4, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 5, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 0, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 0, 2024-04-22T06:56:35.595 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 4584, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:35.596 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:35.597 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 253737, 2024-04-22T06:56:35.598 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 83, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 10456, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 258, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:35.599 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 52984, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 175, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 363, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 679, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 0, 2024-04-22T06:56:35.600 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 363, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 363, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.258993243, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000713480 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 363, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 407 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 363, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 354, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 9, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2024-04-22T06:56:35.601 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 217, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 113, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 1, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 1, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 1, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 5, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 22, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:35.602 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2024-04-22T06:56:35.603 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 118, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 10, 2024-04-22T06:56:35.604 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 10, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 10, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 1, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 46, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 0 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 0, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2024-04-22T06:56:35.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:35.606 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 482, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 329614, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 482, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 482, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 329614, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:35.607 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 363, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 5178, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 363, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 5178, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 363, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 363, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 363, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 363, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.609 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2024-04-22T06:56:35.610 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x55638b52e780": { 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 1, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 168, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 1, 2024-04-22T06:56:35.611 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 1, 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 168, 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:35.612 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-22T06:56:36.386 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-22T06:56:36.386 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-22T06:56:36.386 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-22T06:56:36.387 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:36.387 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:36.387 DEBUG:tasks.check_counter:Getting stats from k 2024-04-22T06:56:36.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-22T06:56:36.628 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:36.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:36 smithi190 ceph-mon[31421]: pgmap v398: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:56:36.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:36 smithi193 ceph-mon[31815]: pgmap v398: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:56:36.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:36 smithi079 ceph-mon[27946]: pgmap v398: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:56:38.373 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 253, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 240, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 1600272, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 147072, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.049591507, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.014557126, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.036624109, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.006762070, 2024-04-22T06:56:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 240, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.013286149, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000055358 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1600272, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 147072 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 182, 2024-04-22T06:56:38.375 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 188144, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.011267751, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.007460964, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.001379854, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 179, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.014913867, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000083317 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:38.376 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 188144 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.000256531, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.000189067, 2024-04-22T06:56:38.377 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:38.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:38.379 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:38.380 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:38.381 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 148945, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 29, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:38.382 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 2, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 251, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 1562134, 2024-04-22T06:56:38.383 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 251, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 251, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1562134, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-22T06:56:38.384 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-22T06:56:38.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:38 smithi190 ceph-mon[31421]: pgmap v399: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:56:38.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:38 smithi193 ceph-mon[31815]: pgmap v399: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:56:38.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:38 smithi079 ceph-mon[27946]: pgmap v399: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:56:39.115 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-22T06:56:39.116 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-22T06:56:39.116 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-22T06:56:39.116 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-22T06:56:39.116 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-22T06:56:39.116 DEBUG:tasks.check_counter:Getting stats from c 2024-04-22T06:56:39.116 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-22T06:56:39.367 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:40.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:40 smithi193 ceph-mon[31815]: pgmap v400: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:40.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:40 smithi190 ceph-mon[31421]: pgmap v400: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:40.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:40 smithi079 ceph-mon[27946]: pgmap v400: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:41.171 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-22T06:56:41.171 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:41.171 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 50, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 49, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 474613, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 16638, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.007073406, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.001653267, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.004718320, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000833938, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 48, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.004620125, 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000096252 2024-04-22T06:56:41.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 160112, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 7088 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 32, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 37, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 14881, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 105831, 2024-04-22T06:56:41.173 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.005755058, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.001410515, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002707658, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.001112361, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 36, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.004218618, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000117183 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 97184 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 28, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 30, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 6945, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 8910, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.003713984, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.000780026, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.001669233, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000960667, 2024-04-22T06:56:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 28, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.002990884, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000106817 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-22T06:56:41.176 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 132, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.003818443, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000028927 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-04-22T06:56:41.178 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 71, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 71, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 71, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 2, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-04-22T06:56:41.179 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:41.180 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 4, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 234, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-04-22T06:56:41.182 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:41.183 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 53, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-04-22T06:56:41.184 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 1, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 1, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 673746270, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 674051812, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 674044588, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 53 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.185 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 30, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 48, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 18, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 11, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 11, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 0, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 71, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 71, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 0, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 42292, 2024-04-22T06:56:41.186 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215416 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.187 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-04-22T06:56:41.189 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.190 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-04-22T06:56:41.191 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.191 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.191 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.191 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-04-22T06:56:41.191 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.192 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.193 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-04-22T06:56:41.198 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 2, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 1939, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:41.199 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:41.200 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:41.201 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 205783, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 85, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 10456, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 258, 2024-04-22T06:56:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 167505, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 1411, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-04-22T06:56:41.203 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 85, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 4, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 85, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 85, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.068998920, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000811752 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 85, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 86 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 85, 2024-04-22T06:56:41.204 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 85, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 50, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 32, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-04-22T06:56:41.205 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-04-22T06:56:41.206 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 0, 2024-04-22T06:56:41.207 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 7, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 7, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-04-22T06:56:41.208 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:41.209 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 106, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 482901, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 106, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 106, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 482901, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 85, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 305552, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 85, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 305552, 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:41.211 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 85, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 85, 2024-04-22T06:56:41.212 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 85, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 85, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:41.213 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x5556d74ac6e0": { 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:41.214 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:41.215 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-22T06:56:41.985 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-22T06:56:41.985 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-22T06:56:41.985 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-22T06:56:41.985 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:41.985 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:41.986 DEBUG:tasks.check_counter:Getting stats from f 2024-04-22T06:56:41.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-22T06:56:42.221 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:42.608 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:42 smithi193 ceph-mon[31815]: pgmap v401: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:42.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:42 smithi190 ceph-mon[31421]: pgmap v401: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:42.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:42 smithi079 ceph-mon[27946]: pgmap v401: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:44.237 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 90, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 85, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 193472, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 30512, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.012583974, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.003291185, 2024-04-22T06:56:44.238 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.006194415, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.002711033, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 85, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.006444681, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000075819 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 174656, 2024-04-22T06:56:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 12704 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 3, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 17, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 716, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 108362, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.002121432, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.000787850, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.001046699, 2024-04-22T06:56:44.240 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000066659, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 15, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.001601452, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000106763 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:56:44.241 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 107664 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 116, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 118, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 34992, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 36895, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.009176903, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.002605089, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.003026742, 2024-04-22T06:56:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.002892880, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 116, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.005974595, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000051505 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:44.243 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 286, 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.008929298, 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000031221 2024-04-22T06:56:44.244 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-04-22T06:56:44.245 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 0, 2024-04-22T06:56:44.246 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 1, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:44.247 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-04-22T06:56:44.248 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:44.249 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-04-22T06:56:44.250 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:44.251 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-04-22T06:56:44.252 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 20, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 4199219, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 4199051, 2024-04-22T06:56:44.253 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 20 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 3, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 3, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 0, 2024-04-22T06:56:44.254 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 1, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 1, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 40836, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215416 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:44.255 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.256 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.257 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.258 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-04-22T06:56:44.259 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.260 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.261 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.262 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.263 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.264 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.265 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.266 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-04-22T06:56:44.267 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.267 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.268 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 6, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 4, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 5, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 4771, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-22T06:56:44.269 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:44.270 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:44.271 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 180359, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 41, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:44.272 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 10456, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 258, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 8512, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 13, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.273 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 174, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 0, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 174, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 174, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.100998435, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000580450 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-04-22T06:56:44.274 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 174, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 175 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 174, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 174, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 112, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 60, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-04-22T06:56:44.275 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-04-22T06:56:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-04-22T06:56:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 118, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 3, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 4, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 4, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 205, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 206228, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 205, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 205, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 206228, 2024-04-22T06:56:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 174, 2024-04-22T06:56:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 4920, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 174, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 4920, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 174, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 174, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 174, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 174, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x5573ae240780": { 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:44.286 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-22T06:56:44.792 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:44 smithi079 ceph-mon[27946]: pgmap v402: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:44.877 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:44 smithi193 ceph-mon[31815]: pgmap v402: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:44.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:44 smithi190 ceph-mon[31421]: pgmap v402: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:45.230 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-22T06:56:45.230 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-22T06:56:45.230 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-22T06:56:45.230 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:45.230 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:45.234 DEBUG:tasks.check_counter:Getting stats from i 2024-04-22T06:56:45.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-22T06:56:45.469 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:46.596 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:46 smithi193 ceph-mon[31815]: pgmap v403: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:56:46.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:46 smithi190 ceph-mon[31421]: pgmap v403: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:56:46.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:46 smithi079 ceph-mon[27946]: pgmap v403: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:56:47.143 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-22T06:56:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:47.143 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 198, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 184, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 258610, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 76569, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.023323210, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.006625057, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.009725743, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.005859855, 2024-04-22T06:56:47.144 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 183, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.010349712, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000056555 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 8, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000001234, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000154 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 223488, 2024-04-22T06:56:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 34048 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 25, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 65, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 17320, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 158441, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.005958165, 2024-04-22T06:56:47.146 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.002284310, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002574886, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000562725, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 63, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.005018857, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000079664 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 7, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000001565, 2024-04-22T06:56:47.147 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000223 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 464, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 150128 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 268, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 291, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 102437, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 111172, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-04-22T06:56:47.148 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.030014517, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.009935710, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.010141083, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.007745801, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 290, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.014659299, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000050549 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 86, 2024-04-22T06:56:47.149 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000039248, 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000456 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3360, 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 1632 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-22T06:56:47.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 563, 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.019238270, 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000034170 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-22T06:56:47.151 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4, 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000099025, 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000024756 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-04-22T06:56:47.152 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 11, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 10, 2024-04-22T06:56:47.153 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 10, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 7, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 3, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 10, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 1, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-04-22T06:56:47.154 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 22, 2024-04-22T06:56:47.155 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 10738142177, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:47.156 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:47.157 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 1, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:47.158 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:47.159 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 1, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 18, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-04-22T06:56:47.160 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 4198635, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 4198299, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.002329248, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002329248 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.161 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 17 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 13, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 14, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 1, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 11, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 12, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 1, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 10, 2024-04-22T06:56:47.162 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 10, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 50760, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 239992 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 2, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-04-22T06:56:47.163 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:47.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-04-22T06:56:47.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.168 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.170 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:47.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 5, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 3, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 4, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 3722, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.176 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:47.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:47.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:47.179 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 229479, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 73, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 10456, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 258, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:47.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 52984, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 175, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-22T06:56:47.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 363, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 679, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 0, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 363, 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-04-22T06:56:47.182 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 363, 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.229996444, 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000633599 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 363, 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 407 2024-04-22T06:56:47.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 363, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 354, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 9, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 217, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-04-22T06:56:47.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 113, 2024-04-22T06:56:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 1, 2024-04-22T06:56:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 1, 2024-04-22T06:56:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-04-22T06:56:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-04-22T06:56:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 1, 2024-04-22T06:56:47.185 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-04-22T06:56:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 5, 2024-04-22T06:56:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-04-22T06:56:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 22, 2024-04-22T06:56:47.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-04-22T06:56:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-04-22T06:56:47.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-04-22T06:56:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-04-22T06:56:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-04-22T06:56:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-04-22T06:56:47.188 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-04-22T06:56:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-04-22T06:56:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-04-22T06:56:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-04-22T06:56:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-04-22T06:56:47.189 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-04-22T06:56:47.190 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 118, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 10, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 8, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 8, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 1, 2024-04-22T06:56:47.191 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 46, 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.192 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:47.193 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 487, 2024-04-22T06:56:47.194 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 327328, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 487, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 487, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 327328, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.195 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 363, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 4594, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 363, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 4594, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:47.197 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 363, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 363, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 363, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 363, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.198 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-22T06:56:47.199 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55e68111a780": { 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 1, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 168, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 1, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:47.200 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 1, 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 168, 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:47.201 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-22T06:56:47.925 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-22T06:56:47.926 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-22T06:56:47.926 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-22T06:56:47.926 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:47.926 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:47.926 DEBUG:tasks.check_counter:Getting stats from l 2024-04-22T06:56:47.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 146994a8-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-22T06:56:48.170 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:48.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:48 smithi193 ceph-mon[31815]: pgmap v404: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:48.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:48 smithi190 ceph-mon[31421]: pgmap v404: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:48.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:48 smithi079 ceph-mon[27946]: pgmap v404: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:49.868 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.000975164, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.000209726, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:49.869 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 185, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 188672, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.012027394, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.008189715, 2024-04-22T06:56:49.870 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.001391500, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 182, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.020165324, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000110798 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:49.871 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 188672 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 257, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 244, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1601984, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 149328, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.052469410, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.015248930, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.039649109, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.006394081, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 244, 2024-04-22T06:56:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.014013510, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000057432 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1601984, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 149328 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 0, 2024-04-22T06:56:49.873 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:49.874 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:49.875 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 149007, 2024-04-22T06:56:49.876 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 30, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:49.877 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 2, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-22T06:56:49.878 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 255, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 1563291, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 255, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 255, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1563291, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-22T06:56:49.879 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-22T06:56:50.659 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-22T06:56:50.660 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-22T06:56:50.660 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-22T06:56:50.660 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-22T06:56:50.660 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-22T06:56:50.660 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-22T06:56:50.673 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T06:56:50.673 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:56:50.673 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:56:50.710 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:56:50.710 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:56:50.768 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:56:50.833 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:50 smithi079 ceph-mon[27946]: pgmap v405: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:50.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 -- 172.21.15.79:0/1730079737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e98069620 msgr2=0x7f7e98069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:50.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1730079737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e98069620 0x7f7e98069a60 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009900 tx=0x7f7e8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:50.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 -- 172.21.15.79:0/1730079737 shutdown_connections 2024-04-22T06:56:50.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1730079737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e9810e810 0x7f7e98110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1730079737 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e98069620 0x7f7e98069a60 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1730079737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e98102e80 0x7f7e980690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 -- 172.21.15.79:0/1730079737 >> 172.21.15.79:0/1730079737 conn(0x7f7e98077e30 msgr2=0x7f7e98078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 -- 172.21.15.79:0/1730079737 shutdown_connections 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 -- 172.21.15.79:0/1730079737 wait complete. 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.920+0000 7f7e9d567640 1 Processor -- start 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 -- start start 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e98069620 0x7f7e981a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e98102e80 0x7f7e981a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 0x7f7e9819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f7e98114390 con 0x7f7e9810e810 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e98114210 con 0x7f7e98069620 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e9d567640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7e98114510 con 0x7f7e98102e80 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 0x7f7e9819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:50.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 0x7f7e9819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59856/0 (socket says 172.21.15.79:59856) 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 -- 172.21.15.79:0/1214082928 learned_addr learned my addr 172.21.15.79:0/1214082928 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e967fc640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e98102e80 0x7f7e981a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e96ffd640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e98069620 0x7f7e981a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 -- 172.21.15.79:0/1214082928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e98102e80 msgr2=0x7f7e981a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e98102e80 0x7f7e981a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 -- 172.21.15.79:0/1214082928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e98069620 msgr2=0x7f7e981a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e98069620 0x7f7e981a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 -- 172.21.15.79:0/1214082928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9819afb0 con 0x7f7e9810e810 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.921+0000 7f7e977fe640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 0x7f7e9819a6f0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000d6a0 tx=0x7f7e9000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.922+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90017070 con 0x7f7e9810e810 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.922+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e90009820 con 0x7f7e9810e810 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.922+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90004c40 con 0x7f7e9810e810 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.922+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9819b2a0 con 0x7f7e9810e810 2024-04-22T06:56:50.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.922+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e981a75b0 con 0x7f7e9810e810 2024-04-22T06:56:50.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.923+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e98101430 con 0x7f7e9810e810 2024-04-22T06:56:50.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.924+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7e900099d0 con 0x7f7e9810e810 2024-04-22T06:56:50.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.925+0000 7f7e7ffff640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7e800a3d20 0x7f7e800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:50.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.925+0000 7f7e96ffd640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7e800a3d20 0x7f7e800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:50.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.925+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7f7e900d0590 con 0x7f7e9810e810 2024-04-22T06:56:50.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.925+0000 7f7e96ffd640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7e800a3d20 0x7f7e800a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7e88005c40 tx=0x7f7e8800a590 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:50.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:50.929+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e9009c0b0 con 0x7f7e9810e810 2024-04-22T06:56:50.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:50 smithi193 ceph-mon[31815]: pgmap v405: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:50.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:50 smithi190 ceph-mon[31421]: pgmap v405: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:51.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.108+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e981a7780 con 0x7f7e9810e810 2024-04-22T06:56:51.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.110+0000 7f7e7ffff640 1 -- 172.21.15.79:0/1214082928 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+25155 (secure 0 0 0) 0x7f7e900a03b0 con 0x7f7e9810e810 2024-04-22T06:56:51.108 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:51.109 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":118,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:56:15.456432+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6834/1769517674":"2024-04-23T06:55:44.932995+0000","172.21.15.193:6835/2674386425":"2024-04-23T06:55:44.714527+0000","172.21.15.193:6834/2674386425":"2024-04-23T06:55:44.714527+0000","172.21.15.190:6837/1832563615":"2024-04-23T06:53:55.123865+0000","172.21.15.190:6836/1832563615":"2024-04-23T06:53:55.123865+0000","172.21.15.193:6838/3285898346":"2024-04-23T06:53:54.951144+0000","172.21.15.193:6834/99277862":"2024-04-23T06:52:48.004463+0000","172.21.15.190:6834/1167513219":"2024-04-23T06:52:47.992797+0000","172.21.15.193:6835/99277862":"2024-04-23T06:52:48.004463+0000","172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.190:6836/2879949884":"2024-04-23T06:45:31.980863+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.190:6836/1635214680":"2024-04-23T06:48:37.138355+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000","172.21.15.190:6834/2543246858":"2024-04-23T06:46:40.618172+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6837/2879949884":"2024-04-23T06:45:31.980863+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.190:6834/833753836":"2024-04-23T06:49:52.347745+0000","172.21.15.190:6837/1169559599":"2024-04-23T06:51:19.325817+0000","172.21.15.190:6836/1169559599":"2024-04-23T06:51:19.325817+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.193:6834/3775212644":"2024-04-23T06:46:40.297133+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6835/1167513219":"2024-04-23T06:52:47.992797+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.193:6838/746968688":"2024-04-23T06:45:31.955779+0000","172.21.15.193:6835/3775212644":"2024-04-23T06:46:40.297133+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6835/2543246858":"2024-04-23T06:46:40.618172+0000","172.21.15.190:6835/1769517674":"2024-04-23T06:55:44.932995+0000","172.21.15.193:6835/3147972524":"2024-04-23T06:49:52.390484+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.193:6838/1588799940":"2024-04-23T06:48:37.120397+0000","172.21.15.193:6839/3285898346":"2024-04-23T06:53:54.951144+0000","172.21.15.193:6839/1588799940":"2024-04-23T06:48:37.120397+0000","172.21.15.193:6839/1413649941":"2024-04-23T06:51:19.325685+0000","172.21.15.193:6838/1413649941":"2024-04-23T06:51:19.325685+0000","172.21.15.193:6839/746968688":"2024-04-23T06:45:31.955779+0000","172.21.15.190:6837/1635214680":"2024-04-23T06:48:37.138355+0000","172.21.15.190:6835/833753836":"2024-04-23T06:49:52.347745+0000","172.21.15.193:6834/3147972524":"2024-04-23T06:49:52.390484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:56:51.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7e800a3d20 msgr2=0x7f7e800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7e800a3d20 0x7f7e800a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7e88005c40 tx=0x7f7e8800a590 comp rx=0 tx=0).stop 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 msgr2=0x7f7e9819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 0x7f7e9819a6f0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000d6a0 tx=0x7f7e9000db60 comp rx=0 tx=0).stop 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 shutdown_connections 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7e9810e810 0x7f7e9819a6f0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e98102e80 0x7f7e981a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f7e800a3d20 0x7f7e800a61e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 --2- 172.21.15.79:0/1214082928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e98069620 0x7f7e981a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 >> 172.21.15.79:0/1214082928 conn(0x7f7e98077e30 msgr2=0x7f7e9810f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 shutdown_connections 2024-04-22T06:56:51.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.112+0000 7f7e9d567640 1 -- 172.21.15.79:0/1214082928 wait complete. 2024-04-22T06:56:51.122 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:42.643333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:21.893989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:22.959860+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:56:51.122 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:56:51.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 -- 172.21.15.79:0/4010646748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d0103b00 msgr2=0x7f18d010bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 --2- 172.21.15.79:0/4010646748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d0103b00 0x7f18d010bfe0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009900 tx=0x7f18c402bd60 comp rx=0 tx=0).stop 2024-04-22T06:56:51.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 -- 172.21.15.79:0/4010646748 shutdown_connections 2024-04-22T06:56:51.266 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 --2- 172.21.15.79:0/4010646748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d010c520 0x7f18d010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 --2- 172.21.15.79:0/4010646748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d0103b00 0x7f18d010bfe0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 --2- 172.21.15.79:0/4010646748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 0x7f18d01035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 -- 172.21.15.79:0/4010646748 >> 172.21.15.79:0/4010646748 conn(0x7f18d00771d0 msgr2=0x7f18d00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 -- 172.21.15.79:0/4010646748 shutdown_connections 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.266+0000 7f18d4a43640 1 -- 172.21.15.79:0/4010646748 wait complete. 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 Processor -- start 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 -- start start 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 0x7f18d019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d0103b00 0x7f18d019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 0x7f18d0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f18d01121b0 con 0x7f18d010c520 2024-04-22T06:56:51.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f18d0112030 con 0x7f18d01031e0 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18d4a43640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18d0112330 con 0x7f18d0103b00 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 0x7f18d0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 0x7f18d0198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59874/0 (socket says 172.21.15.79:59874) 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 -- 172.21.15.79:0/3811469024 learned_addr learned my addr 172.21.15.79:0/3811469024 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18cdd9b640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d0103b00 0x7f18d019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ce59c640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 0x7f18d019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 -- 172.21.15.79:0/3811469024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d0103b00 msgr2=0x7f18d019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d0103b00 0x7f18d019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 -- 172.21.15.79:0/3811469024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 msgr2=0x7f18d019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 0x7f18d019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 -- 172.21.15.79:0/3811469024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d0198d10 con 0x7f18d010c520 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ce59c640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 0x7f18d019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.267+0000 7f18ced9d640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 0x7f18d0198590 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f18c800cad0 tx=0x7f18c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.268+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c800ae20 con 0x7f18d010c520 2024-04-22T06:56:51.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.268+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d0199000 con 0x7f18d010c520 2024-04-22T06:56:51.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.268+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18c8005020 con 0x7f18d010c520 2024-04-22T06:56:51.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.268+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c800f6b0 con 0x7f18d010c520 2024-04-22T06:56:51.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.268+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d01a52a0 con 0x7f18d010c520 2024-04-22T06:56:51.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.269+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d00691c0 con 0x7f18d010c520 2024-04-22T06:56:51.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.270+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f18c80051d0 con 0x7f18d010c520 2024-04-22T06:56:51.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.270+0000 7f18bf7fe640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f18b80a3d20 0x7f18b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.270+0000 7f18ce59c640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f18b80a3d20 0x7f18b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.270+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7f18c80d0db0 con 0x7f18d010c520 2024-04-22T06:56:51.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.271+0000 7f18ce59c640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f18b80a3d20 0x7f18b80a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f18c0005d00 tx=0x7f18c0009550 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:51.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.275+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18c8014030 con 0x7f18d010c520 2024-04-22T06:56:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.456+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18d0107110 con 0x7f18d010c520 2024-04-22T06:56:51.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.457+0000 7f18bf7fe640 1 -- 172.21.15.79:0/3811469024 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) ==== 93+0+10 (secure 0 0 0) 0x7f18c809cf20 con 0x7f18d010c520 2024-04-22T06:56:51.455 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f18b80a3d20 msgr2=0x7f18b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f18b80a3d20 0x7f18b80a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f18c0005d00 tx=0x7f18c0009550 comp rx=0 tx=0).stop 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 msgr2=0x7f18d0198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 0x7f18d0198590 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f18c800cad0 tx=0x7f18c800cf90 comp rx=0 tx=0).stop 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 shutdown_connections 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18d010c520 0x7f18d0198590 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d0103b00 0x7f18d019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f18b80a3d20 0x7f18b80a61e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 --2- 172.21.15.79:0/3811469024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d01031e0 0x7f18d019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.459+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 >> 172.21.15.79:0/3811469024 conn(0x7f18d00771d0 msgr2=0x7f18d00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.460+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 shutdown_connections 2024-04-22T06:56:51.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.460+0000 7f18d4a43640 1 -- 172.21.15.79:0/3811469024 wait complete. 2024-04-22T06:56:51.470 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:56:51.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 -- 172.21.15.79:0/719415383 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c101080 msgr2=0x7ff43c106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 --2- 172.21.15.79:0/719415383 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c101080 0x7ff43c106bb0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009900 tx=0x7ff42c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 -- 172.21.15.79:0/719415383 shutdown_connections 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 --2- 172.21.15.79:0/719415383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c1072e0 0x7ff43c10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 --2- 172.21.15.79:0/719415383 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c101080 0x7ff43c106bb0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 --2- 172.21.15.79:0/719415383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff43c1006d0 0x7ff43c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 -- 172.21.15.79:0/719415383 >> 172.21.15.79:0/719415383 conn(0x7ff43c0773f0 msgr2=0x7ff43c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 -- 172.21.15.79:0/719415383 shutdown_connections 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 -- 172.21.15.79:0/719415383 wait complete. 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.579+0000 7ff441655640 1 Processor -- start 2024-04-22T06:56:51.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 -- start start 2024-04-22T06:56:51.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 0x7ff43c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff43c1072e0 0x7ff43c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c19d010 0x7ff43c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c112190 con 0x7ff43c1006d0 2024-04-22T06:56:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c112010 con 0x7ff43c1072e0 2024-04-22T06:56:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff441655640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c112310 con 0x7ff43c19d010 2024-04-22T06:56:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 0x7ff43c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 0x7ff43c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59894/0 (socket says 172.21.15.79:59894) 2024-04-22T06:56:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 -- 172.21.15.79:0/3190627308 learned_addr learned my addr 172.21.15.79:0/3190627308 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43a7fc640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff43c1072e0 0x7ff43c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43b7fe640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c19d010 0x7ff43c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 -- 172.21.15.79:0/3190627308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c19d010 msgr2=0x7ff43c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c19d010 0x7ff43c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 -- 172.21.15.79:0/3190627308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff43c1072e0 msgr2=0x7ff43c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff43c1072e0 0x7ff43c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 -- 172.21.15.79:0/3190627308 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c19db00 con 0x7ff43c1006d0 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43b7fe640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c19d010 0x7ff43c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.580+0000 7ff43affd640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 0x7ff43c198750 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff43000e9f0 tx=0x7ff43000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.581+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43000cd80 con 0x7ff43c1006d0 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.581+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4300108e0 con 0x7ff43c1006d0 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.581+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43c19ddf0 con 0x7ff43c1006d0 2024-04-22T06:56:51.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.581+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430018980 con 0x7ff43c1006d0 2024-04-22T06:56:51.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.581+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43c1a55b0 con 0x7ff43c1006d0 2024-04-22T06:56:51.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.582+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff43c0684e0 con 0x7ff43c1006d0 2024-04-22T06:56:51.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.583+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff430010430 con 0x7ff43c1006d0 2024-04-22T06:56:51.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.583+0000 7ff423fff640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4240a3d20 0x7ff4240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.584+0000 7ff43a7fc640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4240a3d20 0x7ff4240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.584+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7ff4300d10c0 con 0x7ff43c1006d0 2024-04-22T06:56:51.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.584+0000 7ff43a7fc640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4240a3d20 0x7ff4240a61e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff42c000c00 tx=0x7ff42c005fd0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:51.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.588+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff43009d230 con 0x7ff43c1006d0 2024-04-22T06:56:51.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:51 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1214082928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:51.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:51 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:56:51.740 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:51 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3811469024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:51.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.772+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff43c10a490 con 0x7ff43c1006d0 2024-04-22T06:56:51.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.773+0000 7ff423fff640 1 -- 172.21.15.79:0/3190627308 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) ==== 104+0+11 (secure 0 0 0) 0x7ff43009cb30 con 0x7ff43c1006d0 2024-04-22T06:56:51.771 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-04-22T06:56:51.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4240a3d20 msgr2=0x7ff4240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4240a3d20 0x7ff4240a61e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff42c000c00 tx=0x7ff42c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 msgr2=0x7ff43c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 0x7ff43c198750 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff43000e9f0 tx=0x7ff43000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 shutdown_connections 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7ff4240a3d20 0x7ff4240a61e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff43c1072e0 0x7ff43c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff43c19d010 0x7ff43c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 --2- 172.21.15.79:0/3190627308 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff43c1006d0 0x7ff43c198750 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 >> 172.21.15.79:0/3190627308 conn(0x7ff43c0773f0 msgr2=0x7ff43c105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 shutdown_connections 2024-04-22T06:56:51.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.775+0000 7ff441655640 1 -- 172.21.15.79:0/3190627308 wait complete. 2024-04-22T06:56:51.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:56:51.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:51 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1214082928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:51.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:51 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:56:51.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:51 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3811469024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 -- 172.21.15.79:0/4098157136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940107260 msgr2=0x7f194010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 --2- 172.21.15.79:0/4098157136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940107260 0x7f194010ee30 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f1938009ca0 tx=0x7f193802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 -- 172.21.15.79:0/4098157136 shutdown_connections 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 --2- 172.21.15.79:0/4098157136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940102b40 0x7f1940106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 --2- 172.21.15.79:0/4098157136 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940107260 0x7f194010ee30 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 --2- 172.21.15.79:0/4098157136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 0x7f1940102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 -- 172.21.15.79:0/4098157136 >> 172.21.15.79:0/4098157136 conn(0x7f19400793a0 msgr2=0x7f19400797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 -- 172.21.15.79:0/4098157136 shutdown_connections 2024-04-22T06:56:51.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 -- 172.21.15.79:0/4098157136 wait complete. 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 Processor -- start 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.897+0000 7f194738f640 1 -- start start 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 0x7f1940074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940107260 0x7f1940074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 0x7f19401a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f19401121b0 con 0x7f1940074cd0 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1940112030 con 0x7f1940107260 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1940112330 con 0x7f1940102190 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 0x7f19401a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 0x7f19401a6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59912/0 (socket says 172.21.15.79:59912) 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 -- 172.21.15.79:0/492602165 learned_addr learned my addr 172.21.15.79:0/492602165 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194512b640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 0x7f1940074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194492a640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940107260 0x7f1940074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 -- 172.21.15.79:0/492602165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 msgr2=0x7f1940074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 0x7f1940074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 -- 172.21.15.79:0/492602165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940107260 msgr2=0x7f1940074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940107260 0x7f1940074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 -- 172.21.15.79:0/492602165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19401a7000 con 0x7f1940074cd0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194512b640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 0x7f1940074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194492a640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940107260 0x7f1940074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194592c640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 0x7f19401a6900 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f1938005e30 tx=0x7f1938034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938041020 con 0x7f1940074cd0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1938032970 con 0x7f1940074cd0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19401a7290 con 0x7f1940074cd0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938039040 con 0x7f1940074cd0 2024-04-22T06:56:51.899 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.898+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19401a7780 con 0x7f1940074cd0 2024-04-22T06:56:51.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.900+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19400691c0 con 0x7f1940074cd0 2024-04-22T06:56:51.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.902+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1938044020 con 0x7f1940074cd0 2024-04-22T06:56:51.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.902+0000 7f19367fc640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f192c0a3d50 0x7f192c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:51.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.902+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7f19380f2bd0 con 0x7f1940074cd0 2024-04-22T06:56:51.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.902+0000 7f194512b640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f192c0a3d50 0x7f192c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:51.901 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.903+0000 7f194512b640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f192c0a3d50 0x7f192c0a6210 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f193c009fc0 tx=0x7f193c005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:51.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:51.909+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f193804b020 con 0x7f1940074cd0 2024-04-22T06:56:51.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:51 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1214082928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:51.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:51 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:56:51.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:51 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3811469024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.090+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1940102b40 con 0x7f1940074cd0 2024-04-22T06:56:52.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.090+0000 7f19367fc640 1 -- 172.21.15.79:0/492602165 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) ==== 100+0+11 (secure 0 0 0) 0x7f19380bef70 con 0x7f1940074cd0 2024-04-22T06:56:52.089 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 64 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.092+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f192c0a3d50 msgr2=0x7f192c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.092+0000 7f194738f640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f192c0a3d50 0x7f192c0a6210 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f193c009fc0 tx=0x7f193c005f90 comp rx=0 tx=0).stop 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.092+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 msgr2=0x7f19401a6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.092+0000 7f194738f640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 0x7f19401a6900 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f1938005e30 tx=0x7f1938034650 comp rx=0 tx=0).stop 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.092+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 shutdown_connections 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f192c0a3d50 0x7f192c0a6210 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1940107260 0x7f1940074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1940074cd0 0x7f19401a6900 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 --2- 172.21.15.79:0/492602165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1940102190 0x7f1940074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 >> 172.21.15.79:0/492602165 conn(0x7f19400793a0 msgr2=0x7f19400fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 shutdown_connections 2024-04-22T06:56:52.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.093+0000 7f194738f640 1 -- 172.21.15.79:0/492602165 wait complete. 2024-04-22T06:56:52.102 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T06:56:52.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3828426355 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b006b170 msgr2=0x7fc7b0106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3828426355 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b006b170 0x7fc7b0106ba0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009900 tx=0x7fc7a002bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:52.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3828426355 shutdown_connections 2024-04-22T06:56:52.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3828426355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7b0107280 0x7fc7b010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3828426355 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b006b170 0x7fc7b0106ba0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3828426355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b006a7c0 0x7fc7b006aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3828426355 >> 172.21.15.79:0/3828426355 conn(0x7fc7b0076c50 msgr2=0x7fc7b0077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3828426355 shutdown_connections 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3828426355 wait complete. 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.202+0000 7fc7b55f4640 1 Processor -- start 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 -- start start 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b006a7c0 0x7fc7b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7b0107280 0x7fc7b0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 0x7fc7b019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01121a0 con 0x7fc7b019cfa0 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0112020 con 0x7fc7b006a7c0 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0112320 con 0x7fc7b0107280 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 0x7fc7b019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:52.203 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 0x7fc7b019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59926/0 (socket says 172.21.15.79:59926) 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 -- 172.21.15.79:0/3059926004 learned_addr learned my addr 172.21.15.79:0/3059926004 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7ae7fc640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7b0107280 0x7fc7b0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7aeffd640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b006a7c0 0x7fc7b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 -- 172.21.15.79:0/3059926004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7b0107280 msgr2=0x7fc7b0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7b0107280 0x7fc7b0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 -- 172.21.15.79:0/3059926004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b006a7c0 msgr2=0x7fc7b01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b006a7c0 0x7fc7b01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 -- 172.21.15.79:0/3059926004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b019da90 con 0x7fc7b019cfa0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7af7fe640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 0x7fc7b019d450 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800eb10 tx=0x7fc7a800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8004b20 con 0x7fc7b019cfa0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7a8004cc0 con 0x7fc7b019cfa0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8010670 con 0x7fc7b019cfa0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.203+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b019dd80 con 0x7fc7b019cfa0 2024-04-22T06:56:52.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.204+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b01a5540 con 0x7fc7b019cfa0 2024-04-22T06:56:52.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.205+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b0103680 con 0x7fc7b019cfa0 2024-04-22T06:56:52.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.206+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7a8005000 con 0x7fc7b019cfa0 2024-04-22T06:56:52.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.207+0000 7fc797fff640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7980a3d20 0x7fc7980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:52.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.207+0000 7fc7aeffd640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7980a3d20 0x7fc7980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:52.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.207+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7fc7a80d1470 con 0x7fc7b019cfa0 2024-04-22T06:56:52.206 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.207+0000 7fc7aeffd640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7980a3d20 0x7fc7980a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009bd0 tx=0x7fc7a4005f90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:52.208 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.210+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7a8053ff0 con 0x7fc7b019cfa0 2024-04-22T06:56:52.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.399+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc7b0199a30 con 0x7fc7b019cfa0 2024-04-22T06:56:52.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.400+0000 7fc797fff640 1 -- 172.21.15.79:0/3059926004 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 48 entries v118) ==== 81+0+2915 (secure 0 0 0) 0x7fc7a809d5e0 con 0x7fc7b019cfa0 2024-04-22T06:56:52.398 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6834/1769517674 2024-04-23T06:55:44.932995+0000 2024-04-22T06:56:52.398 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6835/2674386425 2024-04-23T06:55:44.714527+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6834/2674386425 2024-04-23T06:55:44.714527+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6837/1832563615 2024-04-23T06:53:55.123865+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6836/1832563615 2024-04-23T06:53:55.123865+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6838/3285898346 2024-04-23T06:53:54.951144+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6834/99277862 2024-04-23T06:52:48.004463+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6834/1167513219 2024-04-23T06:52:47.992797+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6835/99277862 2024-04-23T06:52:48.004463+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/749551431 2024-04-23T06:44:24.943948+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/482537985 2024-04-23T06:44:24.943948+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:6800/3007546422 2024-04-23T06:44:24.943948+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/4022603925 2024-04-23T06:44:24.943948+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6836/2879949884 2024-04-23T06:45:31.980863+0000 2024-04-22T06:56:52.399 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/196117513 2024-04-23T06:44:24.943948+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6836/1635214680 2024-04-23T06:48:37.138355+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:6801/3007546422 2024-04-23T06:44:24.943948+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/66466683 2024-04-23T06:35:16.561501+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6834/2543246858 2024-04-23T06:46:40.618172+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:6800/3692925838 2024-04-23T06:35:16.561501+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:6801/874344387 2024-04-23T06:34:50.796165+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6837/2879949884 2024-04-23T06:45:31.980863+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/1840338607 2024-04-23T06:35:16.561501+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6834/833753836 2024-04-23T06:49:52.347745+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6837/1169559599 2024-04-23T06:51:19.325817+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6836/1169559599 2024-04-23T06:51:19.325817+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/3938306769 2024-04-23T06:34:50.796165+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/2576810188 2024-04-23T06:35:16.561501+0000 2024-04-22T06:56:52.400 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6834/3775212644 2024-04-23T06:46:40.297133+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:6800/874344387 2024-04-23T06:34:50.796165+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6835/1167513219 2024-04-23T06:52:47.992797+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:6801/3692925838 2024-04-23T06:35:16.561501+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6838/746968688 2024-04-23T06:45:31.955779+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6835/3775212644 2024-04-23T06:46:40.297133+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/1891295292 2024-04-23T06:34:50.796165+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6835/2543246858 2024-04-23T06:46:40.618172+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6835/1769517674 2024-04-23T06:55:44.932995+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6835/3147972524 2024-04-23T06:49:52.390484+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79:0/4061268199 2024-04-23T06:34:50.796165+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6838/1588799940 2024-04-23T06:48:37.120397+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6839/3285898346 2024-04-23T06:53:54.951144+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6839/1588799940 2024-04-23T06:48:37.120397+0000 2024-04-22T06:56:52.401 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6839/1413649941 2024-04-23T06:51:19.325685+0000 2024-04-22T06:56:52.402 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6838/1413649941 2024-04-23T06:51:19.325685+0000 2024-04-22T06:56:52.402 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6839/746968688 2024-04-23T06:45:31.955779+0000 2024-04-22T06:56:52.402 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6837/1635214680 2024-04-23T06:48:37.138355+0000 2024-04-22T06:56:52.402 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.190:6835/833753836 2024-04-23T06:49:52.347745+0000 2024-04-22T06:56:52.402 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.193:6834/3147972524 2024-04-23T06:49:52.390484+0000 2024-04-22T06:56:52.403 INFO:teuthology.orchestra.run.smithi079.stderr:listed 48 entries 2024-04-22T06:56:52.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7980a3d20 msgr2=0x7fc7980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7980a3d20 0x7fc7980a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009bd0 tx=0x7fc7a4005f90 comp rx=0 tx=0).stop 2024-04-22T06:56:52.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 msgr2=0x7fc7b019d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:52.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 0x7fc7b019d450 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800eb10 tx=0x7fc7a800c2d0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.403 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 shutdown_connections 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7b019cfa0 0x7fc7b019d450 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7b0107280 0x7fc7b0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7980a3d20 0x7fc7980a61e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 --2- 172.21.15.79:0/3059926004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7b006a7c0 0x7fc7b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 >> 172.21.15.79:0/3059926004 conn(0x7fc7b0076c50 msgr2=0x7fc7b00ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 shutdown_connections 2024-04-22T06:56:52.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:52.402+0000 7fc7b55f4640 1 -- 172.21.15.79:0/3059926004 wait complete. 2024-04-22T06:56:52.411 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-22T06:56:52.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.412 DEBUG:teuthology.orchestra.run.smithi079:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-22T06:56:52.453 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T06:56:52.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.454 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T06:56:52.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.550 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-04-22T06:56:52.566 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T06:56:52.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.566 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T06:56:52.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.635 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link delete ceph-brx 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: pgmap v406: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3190627308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/492602165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:52 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3059926004' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:56:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: pgmap v406: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3190627308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/492602165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:52 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3059926004' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:56:52.741 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T06:56:52.741 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-04-22T06:56:52.742 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:56:52.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056106,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/56a3268bc80ab15281d773d5053c95ad1ec80a3bea801229dc40c918a90a0b6c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/56a3268bc80ab15281d773d5053c95ad1ec80a3bea801229dc40c918a90a0b6c/diff,workdir=/var/lib/containers/storage/overlay/56a3268bc80ab15281d773d5053c95ad1ec80a3bea801229dc40c918a90a0b6c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9ff140b4e5cb82738ea1e5d5dc8377423bb1d2f8738817d8c8b924210d8f978a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/9ff140b4e5cb82738ea1e5d5dc8377423bb1d2f8738817d8c8b924210d8f978a/diff,workdir=/var/lib/containers/storage/overlay/9ff140b4e5cb82738ea1e5d5dc8377423bb1d2f8738817d8c8b924210d8f978a/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/1c1c6920650e5d1e12d76e3685ec2c300da17c2a4527466077a83d9445bbacab/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/1c1c6920650e5d1e12d76e3685ec2c300da17c2a4527466077a83d9445bbacab/diff,workdir=/var/lib/containers/storage/overlay/1c1c6920650e5d1e12d76e3685ec2c300da17c2a4527466077a83d9445bbacab/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/53ed6a26a7083365e10946e6cca8fd13e9ed1ea162fc6edf92641b3f3da27e60/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/53ed6a26a7083365e10946e6cca8fd13e9ed1ea162fc6edf92641b3f3da27e60/diff,workdir=/var/lib/containers/storage/overlay/53ed6a26a7083365e10946e6cca8fd13e9ed1ea162fc6edf92641b3f3da27e60/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d057ec90e6d71b78cc1e6335d91550f987d0aebea0cfb8b0199e303f6ad1e956/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/d057ec90e6d71b78cc1e6335d91550f987d0aebea0cfb8b0199e303f6ad1e956/diff,workdir=/var/lib/containers/storage/overlay/d057ec90e6d71b78cc1e6335d91550f987d0aebea0cfb8b0199e303f6ad1e956/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e33c9eaa6ac0d93c9f9ba75e428296ff5956981e79c14371a102b69b4bf2e743/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/e33c9eaa6ac0d93c9f9ba75e428296ff5956981e79c14371a102b69b4bf2e743/diff,workdir=/var/lib/containers/storage/overlay/e33c9eaa6ac0d93c9f9ba75e428296ff5956981e79c14371a102b69b4bf2e743/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/930bdc5f11e8d772371627995546178a45ae56150f3a33817503ae49644a6724/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/930bdc5f11e8d772371627995546178a45ae56150f3a33817503ae49644a6724/diff,workdir=/var/lib/containers/storage/overlay/930bdc5f11e8d772371627995546178a45ae56150f3a33817503ae49644a6724/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e121158281814e508feeb00c7fc12980f1e15b9ba5bef465f90c2b7669c8fca2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/e121158281814e508feeb00c7fc12980f1e15b9ba5bef465f90c2b7669c8fca2/diff,workdir=/var/lib/containers/storage/overlay/e121158281814e508feeb00c7fc12980f1e15b9ba5bef465f90c2b7669c8fca2/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a7c44916d3b3ddedfb675b66e527dc56aad1cc8595c2f5ee97596b0ea3d24a4a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/a7c44916d3b3ddedfb675b66e527dc56aad1cc8595c2f5ee97596b0ea3d24a4a/diff,workdir=/var/lib/containers/storage/overlay/a7c44916d3b3ddedfb675b66e527dc56aad1cc8595c2f5ee97596b0ea3d24a4a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/84ed936124baaf5f8134a3d6d59f59e3ed303d60b6cd2f26e68563a277716b2a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3GZQQBAPU75BHNKZ2COUIW2WTN:/var/lib/containers/storage/overlay/l/XYM6LQFUA6BRB4ZLQWADU2CFVV,upperdir=/var/lib/containers/storage/overlay/84ed936124baaf5f8134a3d6d59f59e3ed303d60b6cd2f26e68563a277716b2a/diff,workdir=/var/lib/containers/storage/overlay/84ed936124baaf5f8134a3d6d59f59e3ed303d60b6cd2f26e68563a277716b2a/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-22T06:56:52.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.798 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-04-22T06:56:52.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:56:52.853 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link delete ceph-brx 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: pgmap v406: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3190627308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/492602165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:52.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:52 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3059926004' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:56:52.972 INFO:teuthology.orchestra.run.smithi079.stderr:Cannot find device "ceph-brx" 2024-04-22T06:56:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:56:52.974 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T06:56:52.985 INFO:tasks.cephadm:Teardown begin 2024-04-22T06:56:52.985 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T06:56:52.985 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:56:53.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.086+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/2329405532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc105e30 msgr2=0x7fc7cc106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.086+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/2329405532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc105e30 0x7fc7cc106210 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009cd0 tx=0x7fc7c802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/2329405532 shutdown_connections 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/2329405532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc101b60 0x7fc7cc10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/2329405532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 0x7fc7cc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/2329405532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc105e30 0x7fc7cc106210 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/2329405532 >> 172.21.15.79:0/2329405532 conn(0x7fc7cc077e30 msgr2=0x7fc7cc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/2329405532 shutdown_connections 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/2329405532 wait complete. 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 Processor -- start 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- start start 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 0x7fc7cc1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 0x7fc7cc1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc105e30 0x7fc7cc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc112240 con 0x7fc7cc101b60 2024-04-22T06:56:53.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc1120c0 con 0x7fc7cc105e30 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.087+0000 7fc7d3dc7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc1123c0 con 0x7fc7cc1011a0 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 0x7fc7cc1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 0x7fc7cc1a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41796/0 (socket says 172.21.15.79:41796) 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 -- 172.21.15.79:0/1369070275 learned_addr learned my addr 172.21.15.79:0/1369070275 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d2364640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc105e30 0x7fc7cc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 -- 172.21.15.79:0/1369070275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 msgr2=0x7fc7cc1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1b63640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 0x7fc7cc1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 0x7fc7cc1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 -- 172.21.15.79:0/1369070275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc105e30 msgr2=0x7fc7cc19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc105e30 0x7fc7cc19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 -- 172.21.15.79:0/1369070275 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc19af50 con 0x7fc7cc101b60 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d2364640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc105e30 0x7fc7cc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1b63640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 0x7fc7cc1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:53.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d1362640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 0x7fc7cc1a0b50 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00e9f0 tx=0x7fc7bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:53.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc00cd80 con 0x7fc7cc101b60 2024-04-22T06:56:53.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7bc0054d0 con 0x7fc7cc101b60 2024-04-22T06:56:53.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7cc19b240 con 0x7fc7cc101b60 2024-04-22T06:56:53.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc010690 con 0x7fc7cc101b60 2024-04-22T06:56:53.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.088+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7cc1a7460 con 0x7fc7cc101b60 2024-04-22T06:56:53.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.089+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7cc102570 con 0x7fc7cc101b60 2024-04-22T06:56:53.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.090+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7bc004e90 con 0x7fc7cc101b60 2024-04-22T06:56:53.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.091+0000 7fc7c2ffd640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7b80a3d20 0x7fc7b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.091+0000 7fc7d1b63640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7b80a3d20 0x7fc7b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.091+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7fc7bc0d1290 con 0x7fc7cc101b60 2024-04-22T06:56:53.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.091+0000 7fc7d1b63640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7b80a3d20 0x7fc7b80a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8006010 tx=0x7fc7c8005d80 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:53.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.095+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7bc09d400 con 0x7fc7cc101b60 2024-04-22T06:56:53.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.310+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7cc068de0 con 0x7fc7cc101b60 2024-04-22T06:56:53.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.311+0000 7fc7c2ffd640 1 -- 172.21.15.79:0/1369070275 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+11880 (secure 0 0 0) 0x7fc7bc09cd00 con 0x7fc7cc101b60 2024-04-22T06:56:53.310 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:53.310 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:56:16.456121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":26096,"mds_2":26095},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26095":{"gid":26095,"name":"i","rank":2,"incarnation":86,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6839/3177981466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3177981466},{"type":"v1","addr":"172.21.15.193:6839","nonce":3177981466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26096":{"gid":26096,"name":"h","rank":1,"incarnation":84,"state":"up:active","state_seq":32,"addr":"172.21.15.190:6837/246451352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":246451352},{"type":"v1","addr":"172.21.15.190:6837","nonce":246451352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26234":{"gid":26234,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1936736593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1936736593},{"type":"v1","addr":"172.21.15.190:6835","nonce":1936736593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26236":{"gid":26236,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/755015132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":755015132},{"type":"v1","addr":"172.21.15.193:6835","nonce":755015132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26317":{"gid":26317,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2488235874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2488235874},{"type":"v1","addr":"172.21.15.193:6837","nonce":2488235874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[26095,26096,24467]},"id":1}]} 2024-04-22T06:56:53.310 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 93 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7b80a3d20 msgr2=0x7fc7b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7b80a3d20 0x7fc7b80a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8006010 tx=0x7fc7c8005d80 comp rx=0 tx=0).stop 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 msgr2=0x7fc7cc1a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 0x7fc7cc1a0b50 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00e9f0 tx=0x7fc7bc00edf0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 shutdown_connections 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fc7b80a3d20 0x7fc7b80a61e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7cc105e30 0x7fc7cc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7cc101b60 0x7fc7cc1a0b50 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 --2- 172.21.15.79:0/1369070275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7cc1011a0 0x7fc7cc1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 >> 172.21.15.79:0/1369070275 conn(0x7fc7cc077e30 msgr2=0x7fc7cc10c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 shutdown_connections 2024-04-22T06:56:53.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.313+0000 7fc7d3dc7640 1 -- 172.21.15.79:0/1369070275 wait complete. 2024-04-22T06:56:53.322 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:56:53.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 -- 172.21.15.79:0/1361257838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb43810c5a0 msgr2=0x7fb43810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/1361257838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb43810c5a0 0x7fb43810e9d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb430009ca0 tx=0x7fb43002c180 comp rx=0 tx=0).stop 2024-04-22T06:56:53.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 -- 172.21.15.79:0/1361257838 shutdown_connections 2024-04-22T06:56:53.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/1361257838 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb43810c5a0 0x7fb43810e9d0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/1361257838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb438103af0 0x7fb43810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/1361257838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4381031d0 0x7fb4381035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 -- 172.21.15.79:0/1361257838 >> 172.21.15.79:0/1361257838 conn(0x7fb4380771d0 msgr2=0x7fb4380775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 -- 172.21.15.79:0/1361257838 shutdown_connections 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 -- 172.21.15.79:0/1361257838 wait complete. 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.422+0000 7fb43e0a8640 1 Processor -- start 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 -- start start 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 0x7fb4381986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb438103af0 0x7fb438198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43810c5a0 0x7fb43819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7fb4381121a0 con 0x7fb4381031d0 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb438112020 con 0x7fb438103af0 2024-04-22T06:56:53.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb43e0a8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb438112320 con 0x7fb43810c5a0 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 0x7fb4381986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 0x7fb4381986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41824/0 (socket says 172.21.15.79:41824) 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 -- 172.21.15.79:0/3161217158 learned_addr learned my addr 172.21.15.79:0/3161217158 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb437fff640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43810c5a0 0x7fb43819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb436ffd640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb438103af0 0x7fb438198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 -- 172.21.15.79:0/3161217158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43810c5a0 msgr2=0x7fb43819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43810c5a0 0x7fb43819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 -- 172.21.15.79:0/3161217158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb438103af0 msgr2=0x7fb438198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb438103af0 0x7fb438198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 -- 172.21.15.79:0/3161217158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43819d690 con 0x7fb4381031d0 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb437fff640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43810c5a0 0x7fb43819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.423+0000 7fb4377fe640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 0x7fb4381986a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fb42800e9f0 tx=0x7fb42800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:53.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.424+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42800cd80 con 0x7fb4381031d0 2024-04-22T06:56:53.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.424+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4280108e0 con 0x7fb4381031d0 2024-04-22T06:56:53.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.424+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43819d980 con 0x7fb4381031d0 2024-04-22T06:56:53.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.424+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428018980 con 0x7fb4381031d0 2024-04-22T06:56:53.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.424+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4381a5210 con 0x7fb4381031d0 2024-04-22T06:56:53.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.425+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb438069230 con 0x7fb4381031d0 2024-04-22T06:56:53.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.426+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb428010430 con 0x7fb4381031d0 2024-04-22T06:56:53.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.426+0000 7fb434ff9640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb4200a3d20 0x7fb4200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.427+0000 7fb436ffd640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb4200a3d20 0x7fb4200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.427+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7fb4280d1150 con 0x7fb4381031d0 2024-04-22T06:56:53.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.427+0000 7fb436ffd640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb4200a3d20 0x7fb4200a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb43810e320 tx=0x7fb42c005ed0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:53.429 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.431+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb42809d2c0 con 0x7fb4381031d0 2024-04-22T06:56:53.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.655+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4381999f0 con 0x7fb4381031d0 2024-04-22T06:56:53.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.656+0000 7fb434ff9640 1 -- 172.21.15.79:0/3161217158 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+11880 (secure 0 0 0) 0x7fb42809cbc0 con 0x7fb4381031d0 2024-04-22T06:56:53.654 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:53.654 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6835/2324822974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6834","nonce":2324822974},{"type":"v1","addr":"172.21.15.79:6835","nonce":2324822974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6837/2658402003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6836","nonce":2658402003},{"type":"v1","addr":"172.21.15.79:6837","nonce":2658402003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6839/1490349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6838","nonce":1490349330},{"type":"v1","addr":"172.21.15.79:6839","nonce":1490349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6841/3134754161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6840","nonce":3134754161},{"type":"v1","addr":"172.21.15.79:6841","nonce":3134754161}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1714090730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1714090730},{"type":"v1","addr":"172.21.15.193:6833","nonce":1714090730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1510033395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1510033395},{"type":"v1","addr":"172.21.15.190:6833","nonce":1510033395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:24.011937+0000","modified":"2024-04-22T06:56:16.456121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24467,"mds_1":26096,"mds_2":26095},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24467":{"gid":24467,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.190:6839/4154999918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":4154999918},{"type":"v1","addr":"172.21.15.190:6839","nonce":4154999918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26095":{"gid":26095,"name":"i","rank":2,"incarnation":86,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6839/3177981466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3177981466},{"type":"v1","addr":"172.21.15.193:6839","nonce":3177981466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26096":{"gid":26096,"name":"h","rank":1,"incarnation":84,"state":"up:active","state_seq":32,"addr":"172.21.15.190:6837/246451352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":246451352},{"type":"v1","addr":"172.21.15.190:6837","nonce":246451352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26234":{"gid":26234,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1936736593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1936736593},{"type":"v1","addr":"172.21.15.190:6835","nonce":1936736593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26236":{"gid":26236,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/755015132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":755015132},{"type":"v1","addr":"172.21.15.193:6835","nonce":755015132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26317":{"gid":26317,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2488235874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2488235874},{"type":"v1","addr":"172.21.15.193:6837","nonce":2488235874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24467,"qdb_cluster":[26095,26096,24467]},"id":1}]} 2024-04-22T06:56:53.655 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 93 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb4200a3d20 msgr2=0x7fb4200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb4200a3d20 0x7fb4200a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb43810e320 tx=0x7fb42c005ed0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 msgr2=0x7fb4381986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 0x7fb4381986a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fb42800e9f0 tx=0x7fb42800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 shutdown_connections 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7fb4200a3d20 0x7fb4200a61e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb438103af0 0x7fb438198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb43810c5a0 0x7fb43819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 --2- 172.21.15.79:0/3161217158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4381031d0 0x7fb4381986a0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 >> 172.21.15.79:0/3161217158 conn(0x7fb4380771d0 msgr2=0x7fb4380fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:53.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 shutdown_connections 2024-04-22T06:56:53.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.658+0000 7fb43e0a8640 1 -- 172.21.15.79:0/3161217158 wait complete. 2024-04-22T06:56:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:53 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:56:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:53 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:56:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:53 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:53 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1369070275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:53.667 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:56:53.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 -- 172.21.15.79:0/138807870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c10c650 msgr2=0x7f4e3c10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 --2- 172.21.15.79:0/138807870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c10c650 0x7f4e3c10eaf0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009ca0 tx=0x7f4e3402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 -- 172.21.15.79:0/138807870 shutdown_connections 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 --2- 172.21.15.79:0/138807870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e3c103b90 0x7f4e3c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 --2- 172.21.15.79:0/138807870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c10c650 0x7f4e3c10eaf0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 --2- 172.21.15.79:0/138807870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 -- 172.21.15.79:0/138807870 >> 172.21.15.79:0/138807870 conn(0x7f4e3c0771d0 msgr2=0x7f4e3c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.766+0000 7f4e41c88640 1 -- 172.21.15.79:0/138807870 shutdown_connections 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 -- 172.21.15.79:0/138807870 wait complete. 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 Processor -- start 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 -- start start 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c10c650 0x7f4e3c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e3c198660 0x7f4e3c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.767 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c112340 con 0x7f4e3c1031e0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1121c0 con 0x7f4e3c198660 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e41c88640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1124c0 con 0x7f4e3c10c650 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e3b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.767+0000 7f4e3b7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41842/0 (socket says 172.21.15.79:41842) 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 -- 172.21.15.79:0/3342622036 learned_addr learned my addr 172.21.15.79:0/3342622036 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3bfff640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e3c198660 0x7f4e3c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 -- 172.21.15.79:0/3342622036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c10c650 msgr2=0x7f4e3c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c10c650 0x7f4e3c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 -- 172.21.15.79:0/3342622036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e3c198660 msgr2=0x7f4e3c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e3c198660 0x7f4e3c198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 -- 172.21.15.79:0/3342622036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c199260 con 0x7f4e3c1031e0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e3b7fe640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c19e5a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00d770 tx=0x7f4e2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c017070 con 0x7f4e3c1031e0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2c0054d0 con 0x7f4e3c1031e0 2024-04-22T06:56:53.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c1a5340 con 0x7f4e3c1031e0 2024-04-22T06:56:53.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c004c30 con 0x7f4e3c1031e0 2024-04-22T06:56:53.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.768+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c1a56f0 con 0x7f4e3c1031e0 2024-04-22T06:56:53.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.769+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e3c06ab80 con 0x7f4e3c1031e0 2024-04-22T06:56:53.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.770+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4e2c004e10 con 0x7f4e3c1031e0 2024-04-22T06:56:53.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.771+0000 7f4e38ff9640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4e240a3d20 0x7f4e240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:53.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.771+0000 7f4e3affd640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4e240a3d20 0x7f4e240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:53.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.771+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7f4e2c0d08b0 con 0x7f4e3c1031e0 2024-04-22T06:56:53.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.771+0000 7f4e3affd640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4e240a3d20 0x7f4e240a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e30005d80 tx=0x7f4e30005c90 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:53.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.775+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e2c09ca20 con 0x7f4e3c1031e0 2024-04-22T06:56:53.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:53 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:56:53.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:53 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:56:53.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:53 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:53.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:53 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1369070275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:53.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.951+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e3c1076a0 con 0x7f4e3c1031e0 2024-04-22T06:56:53.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.953+0000 7f4e38ff9640 1 -- 172.21.15.79:0/3342622036 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+25155 (secure 0 0 0) 0x7f4e2c09c320 con 0x7f4e3c1031e0 2024-04-22T06:56:53.951 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-04-22T06:56:53.951 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":118,"fsid":"146994a8-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:23.219533+0000","modified":"2024-04-22T06:56:15.456432+0000","last_up_change":"2024-04-22T06:41:31.159544+0000","last_in_change":"2024-04-22T06:41:10.530054+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:42.643333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:21.893989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:22.959860+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c8d7297-f3dd-4fa6-a1f2-e41ca99ba1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6803","nonce":2633795155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6805","nonce":2633795155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6809","nonce":2633795155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2633795155},{"type":"v1","addr":"172.21.15.79:6807","nonce":2633795155}]},"public_addr":"172.21.15.79:6803/2633795155","cluster_addr":"172.21.15.79:6805/2633795155","heartbeat_back_addr":"172.21.15.79:6809/2633795155","heartbeat_front_addr":"172.21.15.79:6807/2633795155","state":["exists","up"]},{"osd":1,"uuid":"072857b1-5aa7-4510-b9e9-5b6cc192458c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6801","nonce":237015036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6803","nonce":237015036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6807","nonce":237015036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":237015036},{"type":"v1","addr":"172.21.15.190:6805","nonce":237015036}]},"public_addr":"172.21.15.190:6801/237015036","cluster_addr":"172.21.15.190:6803/237015036","heartbeat_back_addr":"172.21.15.190:6807/237015036","heartbeat_front_addr":"172.21.15.190:6805/237015036","state":["exists","up"]},{"osd":2,"uuid":"e26ba0b5-1a7f-41b3-9f1e-18ac6c370b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6801","nonce":4086025232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6803","nonce":4086025232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6807","nonce":4086025232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":4086025232},{"type":"v1","addr":"172.21.15.193:6805","nonce":4086025232}]},"public_addr":"172.21.15.193:6801/4086025232","cluster_addr":"172.21.15.193:6803/4086025232","heartbeat_back_addr":"172.21.15.193:6807/4086025232","heartbeat_front_addr":"172.21.15.193:6805/4086025232","state":["exists","up"]},{"osd":3,"uuid":"0398c3f2-3fd7-40da-bdd0-20c01352af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6811","nonce":433108201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6813","nonce":433108201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6817","nonce":433108201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":433108201},{"type":"v1","addr":"172.21.15.79:6815","nonce":433108201}]},"public_addr":"172.21.15.79:6811/433108201","cluster_addr":"172.21.15.79:6813/433108201","heartbeat_back_addr":"172.21.15.79:6817/433108201","heartbeat_front_addr":"172.21.15.79:6815/433108201","state":["exists","up"]},{"osd":4,"uuid":"4b4fb272-de89-414a-b82c-95b8169e29a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6809","nonce":3437671425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6811","nonce":3437671425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6815","nonce":3437671425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3437671425},{"type":"v1","addr":"172.21.15.190:6813","nonce":3437671425}]},"public_addr":"172.21.15.190:6809/3437671425","cluster_addr":"172.21.15.190:6811/3437671425","heartbeat_back_addr":"172.21.15.190:6815/3437671425","heartbeat_front_addr":"172.21.15.190:6813/3437671425","state":["exists","up"]},{"osd":5,"uuid":"d6cbf2d1-2a90-41c3-a524-243e7ed3e105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6809","nonce":636159812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6811","nonce":636159812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6815","nonce":636159812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":636159812},{"type":"v1","addr":"172.21.15.193:6813","nonce":636159812}]},"public_addr":"172.21.15.193:6809/636159812","cluster_addr":"172.21.15.193:6811/636159812","heartbeat_back_addr":"172.21.15.193:6815/636159812","heartbeat_front_addr":"172.21.15.193:6813/636159812","state":["exists","up"]},{"osd":6,"uuid":"95e2a0de-8900-4bfc-afb5-ed608c8f174d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6819","nonce":1077153926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6821","nonce":1077153926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6825","nonce":1077153926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":1077153926},{"type":"v1","addr":"172.21.15.79:6823","nonce":1077153926}]},"public_addr":"172.21.15.79:6819/1077153926","cluster_addr":"172.21.15.79:6821/1077153926","heartbeat_back_addr":"172.21.15.79:6825/1077153926","heartbeat_front_addr":"172.21.15.79:6823/1077153926","state":["exists","up"]},{"osd":7,"uuid":"f57546c5-ce88-456a-b5c8-0a8ba8dcafae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6817","nonce":2271031515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6819","nonce":2271031515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6823","nonce":2271031515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2271031515},{"type":"v1","addr":"172.21.15.190:6821","nonce":2271031515}]},"public_addr":"172.21.15.190:6817/2271031515","cluster_addr":"172.21.15.190:6819/2271031515","heartbeat_back_addr":"172.21.15.190:6823/2271031515","heartbeat_front_addr":"172.21.15.190:6821/2271031515","state":["exists","up"]},{"osd":8,"uuid":"8f6b199d-ad3e-4ac8-815c-4b30b3a7e335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6817","nonce":3840715130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6819","nonce":3840715130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6823","nonce":3840715130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3840715130},{"type":"v1","addr":"172.21.15.193:6821","nonce":3840715130}]},"public_addr":"172.21.15.193:6817/3840715130","cluster_addr":"172.21.15.193:6819/3840715130","heartbeat_back_addr":"172.21.15.193:6823/3840715130","heartbeat_front_addr":"172.21.15.193:6821/3840715130","state":["exists","up"]},{"osd":9,"uuid":"6a68e58f-14cc-4c45-a45b-5cd0505bff3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6827","nonce":4293144131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6829","nonce":4293144131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6833","nonce":4293144131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":4293144131},{"type":"v1","addr":"172.21.15.79:6831","nonce":4293144131}]},"public_addr":"172.21.15.79:6827/4293144131","cluster_addr":"172.21.15.79:6829/4293144131","heartbeat_back_addr":"172.21.15.79:6833/4293144131","heartbeat_front_addr":"172.21.15.79:6831/4293144131","state":["exists","up"]},{"osd":10,"uuid":"7c7a37b4-3474-4e05-b976-61c5b99dbb98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6825","nonce":2195513042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6827","nonce":2195513042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6831","nonce":2195513042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2195513042},{"type":"v1","addr":"172.21.15.190:6829","nonce":2195513042}]},"public_addr":"172.21.15.190:6825/2195513042","cluster_addr":"172.21.15.190:6827/2195513042","heartbeat_back_addr":"172.21.15.190:6831/2195513042","heartbeat_front_addr":"172.21.15.190:6829/2195513042","state":["exists","up"]},{"osd":11,"uuid":"343a8050-22fd-4408-a1d5-e058b623348d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6825","nonce":3747742483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6827","nonce":3747742483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6831","nonce":3747742483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3747742483},{"type":"v1","addr":"172.21.15.193:6829","nonce":3747742483}]},"public_addr":"172.21.15.193:6825/3747742483","cluster_addr":"172.21.15.193:6827/3747742483","heartbeat_back_addr":"172.21.15.193:6831/3747742483","heartbeat_front_addr":"172.21.15.193:6829/3747742483","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:51.241299+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:15.098670+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:38.934921+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:02.901075+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:28.373905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:53.090225+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:18.869249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:44.876989+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:10.660954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:36.528641+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:02.708838+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:28.772668+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.190:6834/1769517674":"2024-04-23T06:55:44.932995+0000","172.21.15.193:6835/2674386425":"2024-04-23T06:55:44.714527+0000","172.21.15.193:6834/2674386425":"2024-04-23T06:55:44.714527+0000","172.21.15.190:6837/1832563615":"2024-04-23T06:53:55.123865+0000","172.21.15.190:6836/1832563615":"2024-04-23T06:53:55.123865+0000","172.21.15.193:6838/3285898346":"2024-04-23T06:53:54.951144+0000","172.21.15.193:6834/99277862":"2024-04-23T06:52:48.004463+0000","172.21.15.190:6834/1167513219":"2024-04-23T06:52:47.992797+0000","172.21.15.193:6835/99277862":"2024-04-23T06:52:48.004463+0000","172.21.15.79:0/749551431":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/482537985":"2024-04-23T06:44:24.943948+0000","172.21.15.79:6800/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/4022603925":"2024-04-23T06:44:24.943948+0000","172.21.15.190:6836/2879949884":"2024-04-23T06:45:31.980863+0000","172.21.15.79:0/196117513":"2024-04-23T06:44:24.943948+0000","172.21.15.190:6836/1635214680":"2024-04-23T06:48:37.138355+0000","172.21.15.79:6801/3007546422":"2024-04-23T06:44:24.943948+0000","172.21.15.79:0/66466683":"2024-04-23T06:35:16.561501+0000","172.21.15.190:6834/2543246858":"2024-04-23T06:46:40.618172+0000","172.21.15.79:6800/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.79:6801/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6837/2879949884":"2024-04-23T06:45:31.980863+0000","172.21.15.79:0/1840338607":"2024-04-23T06:35:16.561501+0000","172.21.15.190:6834/833753836":"2024-04-23T06:49:52.347745+0000","172.21.15.190:6837/1169559599":"2024-04-23T06:51:19.325817+0000","172.21.15.190:6836/1169559599":"2024-04-23T06:51:19.325817+0000","172.21.15.79:0/3938306769":"2024-04-23T06:34:50.796165+0000","172.21.15.79:0/2576810188":"2024-04-23T06:35:16.561501+0000","172.21.15.193:6834/3775212644":"2024-04-23T06:46:40.297133+0000","172.21.15.79:6800/874344387":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6835/1167513219":"2024-04-23T06:52:47.992797+0000","172.21.15.79:6801/3692925838":"2024-04-23T06:35:16.561501+0000","172.21.15.193:6838/746968688":"2024-04-23T06:45:31.955779+0000","172.21.15.193:6835/3775212644":"2024-04-23T06:46:40.297133+0000","172.21.15.79:0/1891295292":"2024-04-23T06:34:50.796165+0000","172.21.15.190:6835/2543246858":"2024-04-23T06:46:40.618172+0000","172.21.15.190:6835/1769517674":"2024-04-23T06:55:44.932995+0000","172.21.15.193:6835/3147972524":"2024-04-23T06:49:52.390484+0000","172.21.15.79:0/4061268199":"2024-04-23T06:34:50.796165+0000","172.21.15.193:6838/1588799940":"2024-04-23T06:48:37.120397+0000","172.21.15.193:6839/3285898346":"2024-04-23T06:53:54.951144+0000","172.21.15.193:6839/1588799940":"2024-04-23T06:48:37.120397+0000","172.21.15.193:6839/1413649941":"2024-04-23T06:51:19.325685+0000","172.21.15.193:6838/1413649941":"2024-04-23T06:51:19.325685+0000","172.21.15.193:6839/746968688":"2024-04-23T06:45:31.955779+0000","172.21.15.190:6837/1635214680":"2024-04-23T06:48:37.138355+0000","172.21.15.190:6835/833753836":"2024-04-23T06:49:52.347745+0000","172.21.15.193:6834/3147972524":"2024-04-23T06:49:52.390484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:56:53.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:53 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:56:53.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:53 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:56:53.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:53 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' 2024-04-22T06:56:53.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:53 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1369070275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:53.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4e240a3d20 msgr2=0x7f4e240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4e240a3d20 0x7f4e240a61e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e30005d80 tx=0x7f4e30005c90 comp rx=0 tx=0).stop 2024-04-22T06:56:53.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 msgr2=0x7f4e3c19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c19e5a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00d770 tx=0x7f4e2c00dc30 comp rx=0 tx=0).stop 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 shutdown_connections 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f4e240a3d20 0x7f4e240a61e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e3c198660 0x7f4e3c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c10c650 0x7f4e3c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 --2- 172.21.15.79:0/3342622036 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e3c1031e0 0x7f4e3c19e5a0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 >> 172.21.15.79:0/3342622036 conn(0x7f4e3c0771d0 msgr2=0x7f4e3c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 shutdown_connections 2024-04-22T06:56:53.954 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:53.955+0000 7f4e41c88640 1 -- 172.21.15.79:0/3342622036 wait complete. 2024-04-22T06:56:53.965 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 -- 172.21.15.79:0/2868348776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101a40 msgr2=0x7f6154105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 --2- 172.21.15.79:0/2868348776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101a40 0x7f6154105a80 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f6148009900 tx=0x7f614802bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 -- 172.21.15.79:0/2868348776 shutdown_connections 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 --2- 172.21.15.79:0/2868348776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154106160 0x7f615410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 --2- 172.21.15.79:0/2868348776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101a40 0x7f6154105a80 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 --2- 172.21.15.79:0/2868348776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6154101090 0x7f6154101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 -- 172.21.15.79:0/2868348776 >> 172.21.15.79:0/2868348776 conn(0x7f6154077f50 msgr2=0x7f6154078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 -- 172.21.15.79:0/2868348776 shutdown_connections 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.065+0000 7f6159606640 1 -- 172.21.15.79:0/2868348776 wait complete. 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 Processor -- start 2024-04-22T06:56:54.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 -- start start 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 0x7f615410e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154106160 0x7f615410ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f615410f140 0x7f61541a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f6154112200 con 0x7f6154101090 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6154112080 con 0x7f6154106160 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6159606640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6154112380 con 0x7f615410f140 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6152ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 0x7f615410e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6152ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 0x7f615410e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41852/0 (socket says 172.21.15.79:41852) 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f6152ffd640 1 -- 172.21.15.79:0/3271857256 learned_addr learned my addr 172.21.15.79:0/3271857256 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f61537fe640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f615410f140 0x7f61541a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.066+0000 7f61527fc640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154106160 0x7f615410ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6152ffd640 1 -- 172.21.15.79:0/3271857256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f615410f140 msgr2=0x7f61541a6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6152ffd640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f615410f140 0x7f61541a6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6152ffd640 1 -- 172.21.15.79:0/3271857256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154106160 msgr2=0x7f615410ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6152ffd640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154106160 0x7f615410ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6152ffd640 1 -- 172.21.15.79:0/3271857256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61541a7070 con 0x7f6154101090 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6152ffd640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 0x7f615410e6c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f614400ba50 tx=0x7f614400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144019070 con 0x7f6154101090 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614400cbe0 con 0x7f6154101090 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61440075d0 con 0x7f6154101090 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61541a7360 con 0x7f6154101090 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.067+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61541a7830 con 0x7f6154101090 2024-04-22T06:56:54.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.068+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6154068d70 con 0x7f6154101090 2024-04-22T06:56:54.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.069+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f614401c070 con 0x7f6154101090 2024-04-22T06:56:54.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.070+0000 7f613bfff640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f613c0a3d50 0x7f613c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.070+0000 7f61527fc640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f613c0a3d50 0x7f613c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.070+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9845+0+0 (secure 0 0 0) 0x7f61440d02a0 con 0x7f6154101090 2024-04-22T06:56:54.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.070+0000 7f61527fc640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f613c0a3d50 0x7f613c0a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6148006010 tx=0x7f6148004040 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:54.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.074+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f614409c520 con 0x7f6154101090 2024-04-22T06:56:54.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.290+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f61541093f0 con 0x7f6154101090 2024-04-22T06:56:54.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.292+0000 7f613bfff640 1 -- 172.21.15.79:0/3271857256 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v94) ==== 161+0+0 (secure 0 0 0) 0x7f614409be20 con 0x7f6154101090 2024-04-22T06:56:54.291 INFO:teuthology.orchestra.run.smithi079.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.294+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f613c0a3d50 msgr2=0x7f613c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.294+0000 7f6159606640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f613c0a3d50 0x7f613c0a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6148006010 tx=0x7f6148004040 comp rx=0 tx=0).stop 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.294+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 msgr2=0x7f615410e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.294+0000 7f6159606640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 0x7f615410e6c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f614400ba50 tx=0x7f614400bf10 comp rx=0 tx=0).stop 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.294+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 shutdown_connections 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f613c0a3d50 0x7f613c0a6210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6154106160 0x7f615410ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f615410f140 0x7f61541a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.293 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 --2- 172.21.15.79:0/3271857256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6154101090 0x7f615410e6c0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 >> 172.21.15.79:0/3271857256 conn(0x7f6154077f50 msgr2=0x7f61540ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:54.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 shutdown_connections 2024-04-22T06:56:54.294 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.295+0000 7f6159606640 1 -- 172.21.15.79:0/3271857256 wait complete. 2024-04-22T06:56:54.305 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 -- 172.21.15.79:0/1217412431 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc100610 msgr2=0x7f63bc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 --2- 172.21.15.79:0/1217412431 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc100610 0x7f63bc100a90 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00aa10 tx=0x7f63ac02cf70 comp rx=0 tx=0).stop 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 -- 172.21.15.79:0/1217412431 shutdown_connections 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 --2- 172.21.15.79:0/1217412431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100fd0 0x7f63bc10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 --2- 172.21.15.79:0/1217412431 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc100610 0x7f63bc100a90 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 --2- 172.21.15.79:0/1217412431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc106680 0x7f63bc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 -- 172.21.15.79:0/1217412431 >> 172.21.15.79:0/1217412431 conn(0x7f63bc077360 msgr2=0x7f63bc0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:54.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 -- 172.21.15.79:0/1217412431 shutdown_connections 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.408+0000 7f63c4723640 1 -- 172.21.15.79:0/1217412431 wait complete. 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 Processor -- start 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 -- start start 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100610 0x7f63bc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc100fd0 0x7f63bc198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 0x7f63bc19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc10ffb0 con 0x7f63bc106680 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc10fe30 con 0x7f63bc100610 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c4723640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc110130 con 0x7f63bc100fd0 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 0x7f63bc19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 0x7f63bc19cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41862/0 (socket says 172.21.15.79:41862) 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 -- 172.21.15.79:0/4087042976 learned_addr learned my addr 172.21.15.79:0/4087042976 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:54.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c24bf640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100610 0x7f63bc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c1cbe640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc100fd0 0x7f63bc198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 -- 172.21.15.79:0/4087042976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc100fd0 msgr2=0x7f63bc198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc100fd0 0x7f63bc198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 -- 172.21.15.79:0/4087042976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100610 msgr2=0x7f63bc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100610 0x7f63bc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 -- 172.21.15.79:0/4087042976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63bc19d5f0 con 0x7f63bc106680 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c1cbe640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc100fd0 0x7f63bc198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c24bf640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100610 0x7f63bc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.409+0000 7f63c2cc0640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 0x7f63bc19cec0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f63b400eaa0 tx=0x7f63b400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.410+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b400cdb0 con 0x7f63bc106680 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.410+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b4005020 con 0x7f63bc106680 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.410+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63bc19d8e0 con 0x7f63bc106680 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.410+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b40116d0 con 0x7f63bc106680 2024-04-22T06:56:54.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.410+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63bc1a5170 con 0x7f63bc106680 2024-04-22T06:56:54.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.411+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63bc0691c0 con 0x7f63bc106680 2024-04-22T06:56:54.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.412+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f63b40051d0 con 0x7f63bc106680 2024-04-22T06:56:54.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.412+0000 7f63b37fe640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63a80a3da0 0x7f63a80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:54.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.412+0000 7f63c24bf640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63a80a3da0 0x7f63a80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:54.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.413+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 10103+0+0 (secure 0 0 0) 0x7f63b40d1430 con 0x7f63bc106680 2024-04-22T06:56:54.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.413+0000 7f63c24bf640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63a80a3da0 0x7f63a80a6260 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f63b800abe0 tx=0x7f63b800a5c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:54.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.416+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63b4016030 con 0x7f63bc106680 2024-04-22T06:56:54.607 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a[27942]: 2024-04-22T06:56:54.291+0000 7ff4d68a8700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:56:54.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[69527]: ignoring --setuser ceph since I am not root 2024-04-22T06:56:54.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[69527]: ignoring --setgroup ceph since I am not root 2024-04-22T06:56:54.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mds[69531]: -- 172.21.15.193:0/3921444470 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c9c28cba40 con 0x55c9c289ec00 2024-04-22T06:56:54.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[69527]: starting mds.f at 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[69808]: ignoring --setuser ceph since I am not root 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[69808]: ignoring --setgroup ceph since I am not root 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mds[69812]: -- 172.21.15.193:0/3310632550 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565336f49a40 con 0x565336f1d000 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[69808]: starting mds.c at 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[68710]: ignoring --setuser ceph since I am not root 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[68710]: ignoring --setgroup ceph since I am not root 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mds[68714]: -- 172.21.15.193:0/286169285 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fcd5fda40 con 0x557fcd5d0c00 2024-04-22T06:56:54.608 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[68710]: starting mds.i at 2024-04-22T06:56:54.609 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: ignoring --setuser ceph since I am not root 2024-04-22T06:56:54.609 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: ignoring --setgroup ceph since I am not root 2024-04-22T06:56:54.609 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mds[62337]: -- 172.21.15.190:0/1086731225 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558683c5ba40 con 0x558683c2e800 2024-04-22T06:56:54.609 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: starting mds.b at 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[68125]: ignoring --setuser ceph since I am not root 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[68125]: ignoring --setgroup ceph since I am not root 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mds[68129]: -- 172.21.15.190:0/2082999412 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5653bbaada40 con 0x5653bba81000 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[68125]: starting mds.e at 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[67485]: ignoring --setuser ceph since I am not root 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[67485]: ignoring --setgroup ceph since I am not root 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mds[67489]: -- 172.21.15.190:0/2870671264 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b05ffc7a40 con 0x55b05ff9ac00 2024-04-22T06:56:54.610 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[67485]: starting mds.h at 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:54.634+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f63bc02d5c0 con 0x7f63bc106680 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: pgmap v407: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3161217158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3342622036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3271857256' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/3271857256' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:56:54.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: osdmap e119: 12 total, 12 up, 12 in 2024-04-22T06:56:54.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:54 smithi193 ceph-mon[31815]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T06:56:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: pgmap v407: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:56:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3161217158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3342622036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3271857256' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:56:54.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:56:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:56:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/3271857256' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:56:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: osdmap e119: 12 total, 12 up, 12 in 2024-04-22T06:56:54.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:54 smithi190 ceph-mon[31421]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T06:56:54.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: pgmap v407: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-22T06:56:54.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3161217158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:54.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3342622036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:54.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3271857256' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:56:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:56:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:56:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/3271857256' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:56:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: osdmap e119: 12 total, 12 up, 12 in 2024-04-22T06:56:54.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:54 smithi079 ceph-mon[27946]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T06:56:55.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.294+0000 7f63b37fe640 1 -- 172.21.15.79:0/4087042976 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v95) ==== 104+0+0 (secure 0 0 0) 0x7f63b409d520 con 0x7f63bc106680 2024-04-22T06:56:55.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63a80a3da0 msgr2=0x7f63a80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:55.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63a80a3da0 0x7f63a80a6260 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f63b800abe0 tx=0x7f63b800a5c0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 msgr2=0x7f63bc19cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 0x7f63bc19cec0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f63b400eaa0 tx=0x7f63b400ef60 comp rx=0 tx=0).stop 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 shutdown_connections 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63bc106680 0x7f63bc19cec0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63bc100fd0 0x7f63bc198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f63a80a3da0 0x7f63a80a6260 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 --2- 172.21.15.79:0/4087042976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100610 0x7f63bc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 >> 172.21.15.79:0/4087042976 conn(0x7f63bc077360 msgr2=0x7f63bc0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 shutdown_connections 2024-04-22T06:56:55.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.298+0000 7f63c4723640 1 -- 172.21.15.79:0/4087042976 wait complete. 2024-04-22T06:56:55.310 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 -- 172.21.15.79:0/2913033145 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 msgr2=0x7f083010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 --2- 172.21.15.79:0/2913033145 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f083010bfc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f0824009900 tx=0x7f082402bd60 comp rx=0 tx=0).stop 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 -- 172.21.15.79:0/2913033145 shutdown_connections 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 --2- 172.21.15.79:0/2913033145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 0x7f083010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 --2- 172.21.15.79:0/2913033145 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f083010bfc0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 --2- 172.21.15.79:0/2913033145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08301031c0 0x7f08301035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.419+0000 7f0836da4640 1 -- 172.21.15.79:0/2913033145 >> 172.21.15.79:0/2913033145 conn(0x7f0830078000 msgr2=0x7f0830078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:55.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 -- 172.21.15.79:0/2913033145 shutdown_connections 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 -- 172.21.15.79:0/2913033145 wait complete. 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 Processor -- start 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 -- start start 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08301031c0 0x7f08301986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f0830198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 0x7f083019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f08301121f0 con 0x7f0830103ae0 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0830112070 con 0x7f08301031c0 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f0836da4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0830112370 con 0x7f083010c500 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f082ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f0830198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f082ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f0830198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41884/0 (socket says 172.21.15.79:41884) 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.420+0000 7f082ffff640 1 -- 172.21.15.79:0/1676921456 learned_addr learned my addr 172.21.15.79:0/1676921456 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:55.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082ffff640 1 -- 172.21.15.79:0/1676921456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 msgr2=0x7f083019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f0834b40640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08301031c0 0x7f08301986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f0835341640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 0x7f083019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082ffff640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 0x7f083019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082ffff640 1 -- 172.21.15.79:0/1676921456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08301031c0 msgr2=0x7f08301986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082ffff640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08301031c0 0x7f08301986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082ffff640 1 -- 172.21.15.79:0/1676921456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083019d6e0 con 0x7f0830103ae0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f0835341640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 0x7f083019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082ffff640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f0830198c30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0824005940 tx=0x7f082402d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824043020 con 0x7f0830103ae0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082402dda0 con 0x7f0830103ae0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083019d970 con 0x7f0830103ae0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824003bf0 con 0x7f0830103ae0 2024-04-22T06:56:55.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.421+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08301a5260 con 0x7f0830103ae0 2024-04-22T06:56:55.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.422+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0830068d70 con 0x7f0830103ae0 2024-04-22T06:56:55.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.423+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0824046020 con 0x7f0830103ae0 2024-04-22T06:56:55.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.424+0000 7f082dffb640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f08180a3d20 0x7f08180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:55.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.424+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 10103+0+0 (secure 0 0 0) 0x7f08240f3160 con 0x7f0830103ae0 2024-04-22T06:56:55.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.424+0000 7f0834b40640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f08180a3d20 0x7f08180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:55.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.424+0000 7f0834b40640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f08180a3d20 0x7f08180a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f082000ba40 tx=0x7f0820005cc0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:55.426 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.428+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f082404d020 con 0x7f0830103ae0 2024-04-22T06:56:55.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:55.612+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f08301076b0 con 0x7f0830103ae0 2024-04-22T06:56:55.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4087042976' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: Cluster is now healthy 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/4087042976' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6834/2580974748,v1:172.21.15.193:6835/2580974748] up:boot 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6834/1223010830,v1:172.21.15.190:6835/1223010830] up:boot 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6836/2267874224,v1:172.21.15.193:6837/2267874224] up:boot 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.193:6838/4282725338,v1:172.21.15.193:6839/4282725338] up:boot 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6836/237102050,v1:172.21.15.190:6837/237102050] up:boot 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: mds.? [v2:172.21.15.190:6838/1395698646,v1:172.21.15.190:6839/1395698646] up:boot 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: fsmap 12 up:standby 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:56:55.940 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:55.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:56:55.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:56:55.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:55 smithi193 ceph-mon[31815]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4087042976' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: Cluster is now healthy 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/4087042976' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6834/2580974748,v1:172.21.15.193:6835/2580974748] up:boot 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6834/1223010830,v1:172.21.15.190:6835/1223010830] up:boot 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6836/2267874224,v1:172.21.15.193:6837/2267874224] up:boot 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.193:6838/4282725338,v1:172.21.15.193:6839/4282725338] up:boot 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6836/237102050,v1:172.21.15.190:6837/237102050] up:boot 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: mds.? [v2:172.21.15.190:6838/1395698646,v1:172.21.15.190:6839/1395698646] up:boot 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: fsmap 12 up:standby 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:56:55.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:55.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:56:55.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:56:55.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:55 smithi190 ceph-mon[31421]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4087042976' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: Cluster is now healthy 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/4087042976' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6834/2580974748,v1:172.21.15.193:6835/2580974748] up:boot 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6834/1223010830,v1:172.21.15.190:6835/1223010830] up:boot 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6836/2267874224,v1:172.21.15.193:6837/2267874224] up:boot 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.193:6838/4282725338,v1:172.21.15.193:6839/4282725338] up:boot 2024-04-22T06:56:55.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6836/237102050,v1:172.21.15.190:6837/237102050] up:boot 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: mds.? [v2:172.21.15.190:6838/1395698646,v1:172.21.15.190:6839/1395698646] up:boot 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: fsmap 12 up:standby 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:56:55.991 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:55 smithi079 ceph-mon[27946]: from='mgr.24827 172.21.15.79:0/3496305826' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:56:56.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.620+0000 7f082dffb640 1 -- 172.21.15.79:0/1676921456 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) ==== 181+0+0 (secure 0 0 0) 0x7f082403c350 con 0x7f0830103ae0 2024-04-22T06:56:56.619 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'cephfs_metadata' removed 2024-04-22T06:56:56.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.623+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f08180a3d20 msgr2=0x7f08180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:56.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f08180a3d20 0x7f08180a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f082000ba40 tx=0x7f0820005cc0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 msgr2=0x7f0830198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f0830198c30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0824005940 tx=0x7f082402d9a0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 shutdown_connections 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f083010c500 0x7f083019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0830103ae0 0x7f0830198c30 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f08180a3d20 0x7f08180a61e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 --2- 172.21.15.79:0/1676921456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08301031c0 0x7f08301986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 >> 172.21.15.79:0/1676921456 conn(0x7f0830078000 msgr2=0x7f08300fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 shutdown_connections 2024-04-22T06:56:56.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.624+0000 7f0836da4640 1 -- 172.21.15.79:0/1676921456 wait complete. 2024-04-22T06:56:56.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 -- 172.21.15.79:0/3606601104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100450 msgr2=0x7f04b41008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 --2- 172.21.15.79:0/3606601104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100450 0x7f04b41008d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009930 tx=0x7f04a002bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 -- 172.21.15.79:0/3606601104 shutdown_connections 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 --2- 172.21.15.79:0/3606601104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4100e10 0x7f04b410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 --2- 172.21.15.79:0/3606601104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100450 0x7f04b41008d0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 --2- 172.21.15.79:0/3606601104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b4106f30 0x7f04b4107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 -- 172.21.15.79:0/3606601104 >> 172.21.15.79:0/3606601104 conn(0x7f04b40773e0 msgr2=0x7f04b40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 -- 172.21.15.79:0/3606601104 shutdown_connections 2024-04-22T06:56:56.745 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.744+0000 7f04babbd640 1 -- 172.21.15.79:0/3606601104 wait complete. 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 Processor -- start 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 -- start start 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b4100450 0x7f04b419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 0x7f04b419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4106f30 0x7f04b4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41122f0 con 0x7f04b4100e10 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4112170 con 0x7f04b4106f30 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04babbd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4112470 con 0x7f04b4100450 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 0x7f04b419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 0x7f04b419eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41904/0 (socket says 172.21.15.79:41904) 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 -- 172.21.15.79:0/2516936668 learned_addr learned my addr 172.21.15.79:0/2516936668 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b915a640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4106f30 0x7f04b4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b8959640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b4100450 0x7f04b419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:56.746 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 -- 172.21.15.79:0/2516936668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b4100450 msgr2=0x7f04b419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b4100450 0x7f04b419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 -- 172.21.15.79:0/2516936668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4106f30 msgr2=0x7f04b4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4106f30 0x7f04b4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 -- 172.21.15.79:0/2516936668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b4198f30 con 0x7f04b4100e10 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.745+0000 7f04b3fff640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 0x7f04b419eaf0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f04a0005940 tx=0x7f04a002ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.746+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0043020 con 0x7f04b4100e10 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.746+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a002cc80 con 0x7f04b4100e10 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.746+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b41991c0 con 0x7f04b4100e10 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.746+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0003c70 con 0x7f04b4100e10 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.746+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b41a5530 con 0x7f04b4100e10 2024-04-22T06:56:56.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.747+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b4199530 con 0x7f04b4100e10 2024-04-22T06:56:56.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.748+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f04a0046020 con 0x7f04b4100e10 2024-04-22T06:56:56.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.748+0000 7f04b1ffb640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a40a3d20 0x7f04a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:56.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.748+0000 7f04b8959640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a40a3d20 0x7f04a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:56.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.749+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9607+0+0 (secure 0 0 0) 0x7f04a00f3470 con 0x7f04b4100e10 2024-04-22T06:56:56.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.749+0000 7f04b8959640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a40a3d20 0x7f04a40a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f04a80058c0 tx=0x7f04a800a4c0 comp rx=0 tx=0).ready entity=mgr.24827 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:56.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.752+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04a004d020 con 0x7f04b4100e10 2024-04-22T06:56:56.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:56.931+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f04b4199ba0 con 0x7f04b4100e10 2024-04-22T06:56:56.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:56 smithi193 ceph-mon[31815]: pgmap v409: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:56:56.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:56 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1676921456' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:56:56.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:56 smithi190 ceph-mon[31421]: pgmap v409: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:56:56.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:56 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1676921456' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:56:56.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:56 smithi079 ceph-mon[27946]: pgmap v409: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:56:56.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:56 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1676921456' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:56:57.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.627+0000 7f04b1ffb640 1 -- 172.21.15.79:0/2516936668 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) ==== 169+0+0 (secure 0 0 0) 0x7f04a00bf7e0 con 0x7f04b4100e10 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'cephfs_data' removed 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.629+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a40a3d20 msgr2=0x7f04a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.629+0000 7f04babbd640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a40a3d20 0x7f04a40a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f04a80058c0 tx=0x7f04a800a4c0 comp rx=0 tx=0).stop 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.629+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 msgr2=0x7f04b419eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.629+0000 7f04babbd640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 0x7f04b419eaf0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f04a0005940 tx=0x7f04a002ca90 comp rx=0 tx=0).stop 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 shutdown_connections 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:6800/1352275504,v1:172.21.15.79:6801/1352275504] conn(0x7f04a40a3d20 0x7f04a40a61e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04b4106f30 0x7f04b4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04b4100e10 0x7f04b419eaf0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 --2- 172.21.15.79:0/2516936668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04b4100450 0x7f04b419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 >> 172.21.15.79:0/2516936668 conn(0x7f04b40773e0 msgr2=0x7f04b41051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:57.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 shutdown_connections 2024-04-22T06:56:57.637 INFO:teuthology.orchestra.run.smithi079.stderr:2024-04-22T06:56:57.630+0000 7f04babbd640 1 -- 172.21.15.79:0/2516936668 wait complete. 2024-04-22T06:56:57.645 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:56:57.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:56:57.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:56:57.764 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T06:56:57.765 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:56:57.795 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:56:57.811 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:56:57.828 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T06:56:57.828 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T06:56:57.829 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a 2024-04-22T06:56:57.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:57 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/1676921456' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:56:57.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:57 smithi079 ceph-mon[27946]: osdmap e120: 12 total, 12 up, 12 in 2024-04-22T06:56:57.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:57 smithi079 ceph-mon[27946]: from='client.? 172.21.15.79:0/2516936668' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:56:57.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:57 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/1676921456' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:56:57.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:57 smithi193 ceph-mon[31815]: osdmap e120: 12 total, 12 up, 12 in 2024-04-22T06:56:57.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:56:57 smithi193 ceph-mon[31815]: from='client.? 172.21.15.79:0/2516936668' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:56:57.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:57 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/1676921456' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:56:57.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:57 smithi190 ceph-mon[31421]: osdmap e120: 12 total, 12 up, 12 in 2024-04-22T06:56:57.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:57 smithi190 ceph-mon[31421]: from='client.? 172.21.15.79:0/2516936668' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:56:58.240 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:57 smithi079 systemd[1]: Stopping Ceph mon.a for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:56:58.582 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a[27942]: 2024-04-22T06:56:58.312+0000 7ff4dc0b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:56:58.583 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a[27942]: 2024-04-22T06:56:58.312+0000 7ff4dc0b3700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T06:56:58.583 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:58 smithi079 podman[92319]: 2024-04-22 06:56:58.384902335 +0000 UTC m=+0.274199838 container died 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T06:56:58.989 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:58 smithi079 podman[92319]: 2024-04-22 06:56:58.794751601 +0000 UTC m=+0.684049110 container cleanup 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T06:56:58.990 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:58 smithi079 bash[92319]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a 2024-04-22T06:56:59.308 INFO:journalctl@ceph.mon.a.smithi079.stdout:Apr 22 06:56:59 smithi079 podman[92332]: 2024-04-22 06:56:59.038516466 +0000 UTC m=+0.653935842 container remove 5b4c5f2f0f6c0371464a81ad84f6cab38c84d6319b4b4b36d772b4bc99c0857e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:56:59.390 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T06:56:59.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:56:59.438 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T06:56:59.438 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-22T06:56:59.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.b 2024-04-22T06:56:59.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:59 smithi190 systemd[1]: Stopping Ceph mon.b for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:00.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-b[31416]: 2024-04-22T06:56:59.854+0000 7fb30a88a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:00.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:56:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-b[31416]: 2024-04-22T06:56:59.854+0000 7fb30a88a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T06:57:00.203 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 podman[69239]: 2024-04-22 06:57:00.059145057 +0000 UTC m=+0.347391940 container died dedf43993a9b28f79e35f9fdf9114ac3d9099f98c67e507d366e94769271014b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:00.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 podman[69239]: 2024-04-22 06:57:00.476026472 +0000 UTC m=+0.764273361 container cleanup dedf43993a9b28f79e35f9fdf9114ac3d9099f98c67e507d366e94769271014b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T06:57:00.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 bash[69239]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-b 2024-04-22T06:57:00.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T06:57:00.929 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 podman[69251]: 2024-04-22 06:57:00.669658971 +0000 UTC m=+0.606381734 container remove dedf43993a9b28f79e35f9fdf9114ac3d9099f98c67e507d366e94769271014b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T06:57:00.929 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-22T06:57:00.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 systemd[1]: Stopped Ceph mon.b for 146994a8-0072-11ef-bc93-c7b262605968. 2024-04-22T06:57:00.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 22 06:57:00 smithi190 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.b.service: Consumed 13.611s CPU time. 2024-04-22T06:57:00.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:00.960 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-22T06:57:00.960 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-22T06:57:00.960 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.c 2024-04-22T06:57:00.989 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:57:00.614+0000 7f9c4417b700 -1 client.15306: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-04-22T06:57:00.989 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:57:00.615+0000 7f9c4417b700 -1 client.15306: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-04-22T06:57:01.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:57:01 smithi193 systemd[1]: Stopping Ceph mon.c for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:57:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-c[31811]: 2024-04-22T06:57:01.497+0000 7f7739226700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:57:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-c[31811]: 2024-04-22T06:57:01.497+0000 7f7739226700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T06:57:01.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:57:01 smithi193 podman[70771]: 2024-04-22 06:57:01.681688771 +0000 UTC m=+0.530386113 container died 892e02a5d47bb19494d44d356469ec05ed7461b7792fc2f25ff2ac5ecf26bdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T06:57:02.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:57:02 smithi193 podman[70771]: 2024-04-22 06:57:02.182180837 +0000 UTC m=+1.030878186 container cleanup 892e02a5d47bb19494d44d356469ec05ed7461b7792fc2f25ff2ac5ecf26bdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:57:02.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 22 06:57:02 smithi193 bash[70771]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mon-c 2024-04-22T06:57:02.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T06:57:02.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:02.643 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-22T06:57:02.643 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T06:57:02.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x 2024-04-22T06:57:02.964 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:02 smithi079 systemd[1]: Stopping Ceph mgr.x for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:03.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x[28174]: 2024-04-22T06:57:02.965+0000 7f9ce1262700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:03.240 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:02 smithi079 podman[92435]: 2024-04-22 06:57:02.998375768 +0000 UTC m=+0.167473618 container died 56d7b282d42a7c8a4c32bc9ac9452cfac5e9b4b2493c8a84d45de0df073f0fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-22T06:57:03.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:03 smithi079 podman[92435]: 2024-04-22 06:57:03.559024325 +0000 UTC m=+0.728122166 container cleanup 56d7b282d42a7c8a4c32bc9ac9452cfac5e9b4b2493c8a84d45de0df073f0fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:03.990 INFO:journalctl@ceph.mgr.x.smithi079.stdout:Apr 22 06:57:03 smithi079 bash[92435]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-x 2024-04-22T06:57:04.206 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T06:57:04.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:04.239 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T06:57:04.240 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T06:57:04.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.y 2024-04-22T06:57:04.691 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:57:04 smithi190 systemd[1]: Stopping Ceph mgr.y for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:04.952 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:57:04 smithi190 podman[69357]: 2024-04-22 06:57:04.693750587 +0000 UTC m=+0.261008684 container died e4afe626c571633f8a69e222401385de5f602100183bb179a1cacadbe63cec29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:05.372 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:57:05 smithi190 podman[69357]: 2024-04-22 06:57:05.073096972 +0000 UTC m=+0.640355058 container cleanup e4afe626c571633f8a69e222401385de5f602100183bb179a1cacadbe63cec29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:57:05.372 INFO:journalctl@ceph.mgr.y.smithi190.stdout:Apr 22 06:57:05 smithi190 bash[69357]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-y 2024-04-22T06:57:05.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T06:57:05.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:05.581 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T06:57:05.581 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-22T06:57:05.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.z 2024-04-22T06:57:05.939 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:57:05 smithi193 systemd[1]: Stopping Ceph mgr.z for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:06.939 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:57:06 smithi193 podman[70892]: 2024-04-22 06:57:06.553089774 +0000 UTC m=+0.781331125 container died e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T06:57:08.189 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:57:08 smithi193 podman[70892]: 2024-04-22 06:57:08.014445752 +0000 UTC m=+2.242687110 container cleanup e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:57:08.189 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:57:08 smithi193 bash[70892]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z 2024-04-22T06:57:08.459 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:57:08 smithi193 podman[70906]: 2024-04-22 06:57:08.324500171 +0000 UTC m=+1.772161460 container remove e63eb2aa2768dee001568ee19fd209691128c61ceb9d9bbbbeb559e39da48b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T06:57:08.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 22 06:57:08 smithi193 systemd[1]: ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-22T06:57:08.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-22T06:57:08.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:08.532 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-22T06:57:08.532 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T06:57:08.532 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.0 2024-04-22T06:57:08.989 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:08 smithi079 systemd[1]: Stopping Ceph osd.0 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:10.240 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0[40118]: 2024-04-22T06:57:09.772+0000 7fd7ef103700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:10.240 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0[40118]: 2024-04-22T06:57:09.772+0000 7fd7ef103700 -1 osd.0 121 *** Got signal Terminated *** 2024-04-22T06:57:10.240 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0[40118]: 2024-04-22T06:57:09.772+0000 7fd7ef103700 -1 osd.0 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:15.705 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:15 smithi079 podman[92556]: 2024-04-22 06:57:15.396381171 +0000 UTC m=+6.669444665 container died 7149cb2fcc5db21123a44b98e9148847d716fd5e07b62d3369ae51a15c51716f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:57:15.989 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:15 smithi079 podman[92556]: 2024-04-22 06:57:15.814872847 +0000 UTC m=+7.087936339 container cleanup 7149cb2fcc5db21123a44b98e9148847d716fd5e07b62d3369ae51a15c51716f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-04-22T06:57:15.989 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:15 smithi079 bash[92556]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0 2024-04-22T06:57:16.418 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:16 smithi079 podman[92574]: 2024-04-22 06:57:16.091738423 +0000 UTC m=+0.682544147 container remove 7149cb2fcc5db21123a44b98e9148847d716fd5e07b62d3369ae51a15c51716f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:16.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:16 smithi079 podman[92675]: 2024-04-22 06:57:16.323459713 +0000 UTC m=+0.018073028 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:16.676 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:16 smithi079 podman[92675]: 2024-04-22 06:57:16.425888026 +0000 UTC m=+0.120501315 container create f154af7d23297c7e106a7d84c9d9c710035e64d5d62ee8358ecde11a3d629fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:57:16.990 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:16 smithi079 podman[92675]: 2024-04-22 06:57:16.678895336 +0000 UTC m=+0.373508623 container init f154af7d23297c7e106a7d84c9d9c710035e64d5d62ee8358ecde11a3d629fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T06:57:16.990 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:16 smithi079 podman[92675]: 2024-04-22 06:57:16.682500998 +0000 UTC m=+0.377114295 container start f154af7d23297c7e106a7d84c9d9c710035e64d5d62ee8358ecde11a3d629fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:16.990 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:16 smithi079 podman[92675]: 2024-04-22 06:57:16.787338286 +0000 UTC m=+0.481951591 container attach f154af7d23297c7e106a7d84c9d9c710035e64d5d62ee8358ecde11a3d629fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T06:57:17.387 INFO:journalctl@ceph.osd.0.smithi079.stdout:Apr 22 06:57:17 smithi079 podman[92675]: 2024-04-22 06:57:17.033158878 +0000 UTC m=+0.727772177 container died f154af7d23297c7e106a7d84c9d9c710035e64d5d62ee8358ecde11a3d629fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:17.678 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T06:57:17.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:17.718 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T06:57:17.718 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T06:57:17.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.1 2024-04-22T06:57:18.042 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:17 smithi190 systemd[1]: Stopping Ceph osd.1 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:18.452 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1[36018]: 2024-04-22T06:57:18.043+0000 7f0d66933700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:18.453 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1[36018]: 2024-04-22T06:57:18.043+0000 7f0d66933700 -1 osd.1 121 *** Got signal Terminated *** 2024-04-22T06:57:18.453 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1[36018]: 2024-04-22T06:57:18.043+0000 7f0d66933700 -1 osd.1 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:23.790 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:23 smithi190 podman[69493]: 2024-04-22 06:57:23.496077998 +0000 UTC m=+5.575878865 container died 5e3162d4a123e71bf7bf95562987025a4f5b17c3b53021e5dd8d81db518103ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:24.135 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:23 smithi190 podman[69493]: 2024-04-22 06:57:23.894761296 +0000 UTC m=+5.974562164 container cleanup 5e3162d4a123e71bf7bf95562987025a4f5b17c3b53021e5dd8d81db518103ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T06:57:24.135 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:23 smithi190 bash[69493]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1 2024-04-22T06:57:24.436 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:24 smithi190 podman[69512]: 2024-04-22 06:57:24.137697456 +0000 UTC m=+0.645461072 container remove 5e3162d4a123e71bf7bf95562987025a4f5b17c3b53021e5dd8d81db518103ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:57:24.688 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:24 smithi190 podman[69602]: 2024-04-22 06:57:24.346811511 +0000 UTC m=+0.018380692 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:24.689 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:24 smithi190 podman[69602]: 2024-04-22 06:57:24.463801912 +0000 UTC m=+0.135371061 container create a0ef232b6d50e3ac92c99296c32ea81bab5fcf960e9aa7f72d34f6cecd82100e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:24.689 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:24 smithi190 podman[69602]: 2024-04-22 06:57:24.691172145 +0000 UTC m=+0.362741294 container init a0ef232b6d50e3ac92c99296c32ea81bab5fcf960e9aa7f72d34f6cecd82100e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T06:57:24.952 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:24 smithi190 podman[69602]: 2024-04-22 06:57:24.695359572 +0000 UTC m=+0.366928722 container start a0ef232b6d50e3ac92c99296c32ea81bab5fcf960e9aa7f72d34f6cecd82100e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:57:24.952 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:24 smithi190 podman[69602]: 2024-04-22 06:57:24.799790503 +0000 UTC m=+0.471359655 container attach a0ef232b6d50e3ac92c99296c32ea81bab5fcf960e9aa7f72d34f6cecd82100e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-22T06:57:25.264 INFO:journalctl@ceph.osd.1.smithi190.stdout:Apr 22 06:57:25 smithi190 podman[69602]: 2024-04-22 06:57:25.034329389 +0000 UTC m=+0.705898549 container died a0ef232b6d50e3ac92c99296c32ea81bab5fcf960e9aa7f72d34f6cecd82100e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-04-22T06:57:25.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T06:57:25.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:25.752 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T06:57:25.752 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T06:57:25.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.2 2024-04-22T06:57:26.058 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:25 smithi193 systemd[1]: Stopping Ceph osd.2 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:26.439 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2[36978]: 2024-04-22T06:57:26.059+0000 7fe8c86ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:26.439 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2[36978]: 2024-04-22T06:57:26.059+0000 7fe8c86ac700 -1 osd.2 121 *** Got signal Terminated *** 2024-04-22T06:57:26.439 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2[36978]: 2024-04-22T06:57:26.059+0000 7fe8c86ac700 -1 osd.2 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:31.901 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:31 smithi193 podman[71027]: 2024-04-22 06:57:31.598325174 +0000 UTC m=+5.649620582 container died c4eda6159eceee56a946876fa43d99a2a853242d086807e01a6901fcc75344b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T06:57:32.189 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71027]: 2024-04-22 06:57:32.002223783 +0000 UTC m=+6.053519193 container cleanup c4eda6159eceee56a946876fa43d99a2a853242d086807e01a6901fcc75344b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T06:57:32.189 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 bash[71027]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2 2024-04-22T06:57:32.610 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71045]: 2024-04-22 06:57:32.262202035 +0000 UTC m=+0.666558326 container remove c4eda6159eceee56a946876fa43d99a2a853242d086807e01a6901fcc75344b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T06:57:32.939 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71135]: 2024-04-22 06:57:32.515256827 +0000 UTC m=+0.017926283 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:32.939 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71135]: 2024-04-22 06:57:32.645850689 +0000 UTC m=+0.148520180 container create 6cc377b36c05b3985e23beb904e31851d9f91fae2a8941dc5b50ab8f79e213f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-04-22T06:57:32.939 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71135]: 2024-04-22 06:57:32.849652262 +0000 UTC m=+0.352321708 container init 6cc377b36c05b3985e23beb904e31851d9f91fae2a8941dc5b50ab8f79e213f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:57:32.939 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71135]: 2024-04-22 06:57:32.852791845 +0000 UTC m=+0.355461298 container start 6cc377b36c05b3985e23beb904e31851d9f91fae2a8941dc5b50ab8f79e213f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:32.939 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:32 smithi193 podman[71135]: 2024-04-22 06:57:32.94122887 +0000 UTC m=+0.443898325 container attach 6cc377b36c05b3985e23beb904e31851d9f91fae2a8941dc5b50ab8f79e213f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-22T06:57:33.189 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 22 06:57:33 smithi193 podman[71135]: 2024-04-22 06:57:33.149490272 +0000 UTC m=+0.652159726 container died 6cc377b36c05b3985e23beb904e31851d9f91fae2a8941dc5b50ab8f79e213f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-22T06:57:33.775 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T06:57:33.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:33.812 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T06:57:33.812 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T06:57:33.812 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.3 2024-04-22T06:57:34.124 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:33 smithi079 systemd[1]: Stopping Ceph osd.3 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:34.490 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:34 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3[46645]: 2024-04-22T06:57:34.126+0000 7f80f3ba3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:34.490 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:34 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3[46645]: 2024-04-22T06:57:34.126+0000 7f80f3ba3700 -1 osd.3 121 *** Got signal Terminated *** 2024-04-22T06:57:34.490 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:34 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3[46645]: 2024-04-22T06:57:34.126+0000 7f80f3ba3700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:37.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:37 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:37.523+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:38.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:37 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:37.809+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:38.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:38 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:38.542+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:39.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:38.741+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:39.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:38 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:38.809+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:39.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:38 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:38.888+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:39.877 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:39 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:39.778+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:39.877 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:39 smithi079 podman[92782]: 2024-04-22 06:57:39.609682233 +0000 UTC m=+5.604267472 container died f3a3f6bd700a5f3c3bcaebe77cc1e5db0a027f52b12e9d43e68f9d31177a4429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:57:39.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:39 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:39.523+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:40.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:39.773+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:40.239 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:39 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:39.879+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:40.240 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92782]: 2024-04-22 06:57:40.001172993 +0000 UTC m=+5.995758233 container cleanup f3a3f6bd700a5f3c3bcaebe77cc1e5db0a027f52b12e9d43e68f9d31177a4429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:57:40.240 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 bash[92782]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3 2024-04-22T06:57:40.615 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92799]: 2024-04-22 06:57:40.29433629 +0000 UTC m=+0.679011987 container remove f3a3f6bd700a5f3c3bcaebe77cc1e5db0a027f52b12e9d43e68f9d31177a4429 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:57:40.937 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92887]: 2024-04-22 06:57:40.520557164 +0000 UTC m=+0.018404060 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:40.937 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92887]: 2024-04-22 06:57:40.636929094 +0000 UTC m=+0.134775976 container create 809df6c79caecf34f75f32cbb2889027bdd0a9a3e606f3ec4d7077b9cd8403ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T06:57:40.937 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92887]: 2024-04-22 06:57:40.856326822 +0000 UTC m=+0.354173704 container init 809df6c79caecf34f75f32cbb2889027bdd0a9a3e606f3ec4d7077b9cd8403ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T06:57:40.938 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92887]: 2024-04-22 06:57:40.859389118 +0000 UTC m=+0.357236000 container start 809df6c79caecf34f75f32cbb2889027bdd0a9a3e606f3ec4d7077b9cd8403ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:40.938 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:40 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:40.855+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:40.938 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:40 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:40.818+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:40.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:40 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:40.572+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:41.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:40.728+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:41.240 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:40 smithi079 podman[92887]: 2024-04-22 06:57:40.939672876 +0000 UTC m=+0.437519768 container attach 809df6c79caecf34f75f32cbb2889027bdd0a9a3e606f3ec4d7077b9cd8403ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:41.240 INFO:journalctl@ceph.osd.3.smithi079.stdout:Apr 22 06:57:41 smithi079 podman[92887]: 2024-04-22 06:57:41.150706138 +0000 UTC m=+0.648553022 container died 809df6c79caecf34f75f32cbb2889027bdd0a9a3e606f3ec4d7077b9cd8403ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:57:41.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:41.345+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:41.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T06:57:41.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:41.817 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T06:57:41.818 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T06:57:41.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.4 2024-04-22T06:57:41.843 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:41 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:41.566+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:42.129 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:41 smithi190 systemd[1]: Stopping Ceph osd.4 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:41.698+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:42.239 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:41 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:41.809+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:42.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:41 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:41.798+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:42.452 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:42 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4[42955]: 2024-04-22T06:57:42.130+0000 7f0890b1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:42.453 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:42 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4[42955]: 2024-04-22T06:57:42.130+0000 7f0890b1e700 -1 osd.4 121 *** Got signal Terminated *** 2024-04-22T06:57:42.453 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:42 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4[42955]: 2024-04-22T06:57:42.130+0000 7f0890b1e700 -1 osd.4 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:42.670 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:42.377+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:42.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:42.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:42.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:42 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:42.578+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:43.239 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:42 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:42.835+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:43.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:42 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:42.806+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:43.669 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:43.376+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:43.854 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:43 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:43.592+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:43.671+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:44.239 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:43 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:43.787+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:44.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:43 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:43.830+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:44.687 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:44 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:44.348+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:44.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:44 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:44.689+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:44.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:44 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:44.627+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:44.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:44 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:44.627+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:45.141 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:44.790+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:45.141 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:44 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:44.815+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:45.664 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:45 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:45.385+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:45.664 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:45 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:45.385+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:45.664 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:45 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:45.415+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:45.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:45 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:45.666+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:45.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:45 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:45.666+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:45.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:45 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:45.569+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:45.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:45 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:45.592+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:45.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:45 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:45.592+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:46.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:45 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:45.808+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:46.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:45 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:45.815+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:46.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:45 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:45.815+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:46.452 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:46 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4[42955]: 2024-04-22T06:57:46.162+0000 7f088910a700 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:22.467945+0000 front 2024-04-22T06:57:22.467979+0000 (oldest deadline 2024-04-22T06:57:45.368139+0000) 2024-04-22T06:57:46.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:46 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:46.402+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:46.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:46 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:46.379+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:46.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:46 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:46.379+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:46.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:46 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:46.568+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:46.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:46 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:46.555+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:46.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:46 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:46.555+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:47.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:46 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:46.710+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:47.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:46 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:46.710+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:47.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:46.762+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:47.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:46.813+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:47.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:46.813+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:47.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:47 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:47.338+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:47.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:47 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:47.338+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:47.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:47 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:47.412+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:47.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:47 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:47.542+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:47.953 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:47 smithi190 podman[69723]: 2024-04-22 06:57:47.654277587 +0000 UTC m=+5.648123286 container died 705880ad4b0254aa158e653b0912c4b8a9fcbc5c7963d29eeeac10e25c0923b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:57:47.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:47 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:47.560+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:47.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:47 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:47.560+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:48.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:47 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:47.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:48.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:47 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:47.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:48.242 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:47.851+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:48.243 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:47.851+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:48.243 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:47.749+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:48.243 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:47 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:47.749+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:48.408 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:48 smithi190 podman[69723]: 2024-04-22 06:57:48.176236974 +0000 UTC m=+6.170082675 container cleanup 705880ad4b0254aa158e653b0912c4b8a9fcbc5c7963d29eeeac10e25c0923b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:57:48.408 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:48 smithi190 bash[69723]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4 2024-04-22T06:57:48.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:48 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:48.377+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:48.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:48 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:48.366+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:48.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:48 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:48.366+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:48.702 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:48 smithi190 podman[69752]: 2024-04-22 06:57:48.411021901 +0000 UTC m=+0.757855701 container remove 705880ad4b0254aa158e653b0912c4b8a9fcbc5c7963d29eeeac10e25c0923b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T06:57:48.702 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:48 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:48.527+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:48.703 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:48 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:48.537+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:48.703 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:48 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:48.537+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:48.989 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:48.856+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:48.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:48.856+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:48.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:48.708+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:48.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:48 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:48.708+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:49.126 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:48 smithi190 podman[69843]: 2024-04-22 06:57:48.734895174 +0000 UTC m=+0.018159883 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:49.126 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:48 smithi190 podman[69843]: 2024-04-22 06:57:48.854123464 +0000 UTC m=+0.137388158 container create 685db354c9b6cfbae5ab02c19a3bae29241f5a377da222ac405b1bb129677432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T06:57:49.126 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:49 smithi190 podman[69843]: 2024-04-22 06:57:49.128909658 +0000 UTC m=+0.412174360 container init 685db354c9b6cfbae5ab02c19a3bae29241f5a377da222ac405b1bb129677432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T06:57:49.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:48 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:48.728+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:49.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:48 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:48.728+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:49.425 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:49 smithi190 podman[69843]: 2024-04-22 06:57:49.132493872 +0000 UTC m=+0.415758567 container start 685db354c9b6cfbae5ab02c19a3bae29241f5a377da222ac405b1bb129677432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:57:49.426 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:49 smithi190 podman[69843]: 2024-04-22 06:57:49.223560584 +0000 UTC m=+0.506825281 container attach 685db354c9b6cfbae5ab02c19a3bae29241f5a377da222ac405b1bb129677432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-04-22T06:57:49.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:49 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:49.393+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:49.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:49 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:49.400+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:49.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:49 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:49.400+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:49.699 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:49 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:49.522+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:49.699 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 22 06:57:49 smithi190 podman[69843]: 2024-04-22 06:57:49.428849965 +0000 UTC m=+0.712114657 container died 685db354c9b6cfbae5ab02c19a3bae29241f5a377da222ac405b1bb129677432 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T06:57:49.700 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:49 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:49.550+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:49.700 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:49 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:49.550+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:49.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T06:57:49.989 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:49.860+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:49.989 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:49.860+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:49.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:49.694+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:49.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:49 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:49.694+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:50.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:50.066 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T06:57:50.066 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T06:57:50.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.5 2024-04-22T06:57:50.093 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:49 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:49.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:50.093 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:49 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:49.693+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:50.362 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:50 smithi193 systemd[1]: Stopping Ceph osd.5 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:50.651 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:50.422+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:50.652 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:50.363+0000 7fea308cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:50.652 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:50.363+0000 7fea308cb700 -1 osd.5 121 *** Got signal Terminated *** 2024-04-22T06:57:50.652 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:50.363+0000 7fea308cb700 -1 osd.5 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:50.652 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:50.379+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:50.652 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:50.379+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:50.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:50.653+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:50.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:50 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:50.653+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:50.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:50 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:50.479+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:50.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:50 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:50.558+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:50.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:50 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:50.558+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:50.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:50 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:50.815+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:50.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:50 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:50.815+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:50.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:50 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:50.689+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:50.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:50 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:50.689+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:51.673 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:51 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:51.389+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:51.674 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:51 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:51.377+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:51.674 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:51 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:51.377+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:51.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:51 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:51.675+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:51.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:51 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:51.675+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:51.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:51 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:51.473+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:51.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:51 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:51.547+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:51.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:51 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:51.547+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:51.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:51 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:51.829+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:51.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:51 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:51.829+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:51.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:51 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:51.686+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:51.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:51 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:51.686+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:52.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:52 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:52.376+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:52.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:52 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:52.376+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:52.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:52 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:52.351+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:52.702 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:52 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:52.445+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:52.702 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:52 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:52.445+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:52.703 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:52 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:52.563+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:52.703 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:52 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:52.563+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:52.703 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:52 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:52.563+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:52.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:52.680+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:52.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:52.680+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:52.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:52.680+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:53.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:52 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:52.693+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:53.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:52 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:52.693+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:53.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:52.816+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:53.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:52.816+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:53.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:53.333+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:53.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:53.412+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:53.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:53.412+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:53.690 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:53.412+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:28.649727+0000 front 2024-04-22T06:57:28.649753+0000 (oldest deadline 2024-04-22T06:57:52.750014+0000) 2024-04-22T06:57:53.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:53 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:53.485+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:53.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:53 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:53.485+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:53.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:53 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:53.588+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:53.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:53 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:53.588+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:53.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:53 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:53.588+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:53.989 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:53 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:53.833+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:53.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:53 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:53.833+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:53.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:53 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:53.681+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:53.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:53 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:53.681+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:53.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:53 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:53.681+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:54.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:53.719+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:54.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:53.719+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:54.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:53 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:53.719+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:54.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:54.294+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:54.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:54.418+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.647376+0000 front 2024-04-22T06:57:14.647565+0000 (oldest deadline 2024-04-22T06:57:40.547424+0000) 2024-04-22T06:57:54.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:54.418+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.548030+0000 front 2024-04-22T06:57:20.548096+0000 (oldest deadline 2024-04-22T06:57:44.548712+0000) 2024-04-22T06:57:54.689 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5[43924]: 2024-04-22T06:57:54.418+0000 7fea2cebf700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:28.649727+0000 front 2024-04-22T06:57:28.649753+0000 (oldest deadline 2024-04-22T06:57:52.750014+0000) 2024-04-22T06:57:54.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:54.498+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:54.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:54.498+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:54.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:54.626+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:54.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:54.626+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:54.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:54 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:54.626+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:54.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:54 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:54.797+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:54.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:54 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:54.797+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:54.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:54 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:54.689+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:54.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:54 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:54.689+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:54.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:54 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:54.689+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:55.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:54.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:55.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:54.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:55.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:54 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:54.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:55.670 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:55 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:55.251+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:55.939 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:55 smithi193 podman[71241]: 2024-04-22 06:57:55.89967117 +0000 UTC m=+5.638612839 container died 8029dde9e57b2f9e6282122ae68251c6856657bd9b6abb49e80facf9d64d79e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, io.buildah.version=1.33.5) 2024-04-22T06:57:55.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:55 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:55.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:55.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:55 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:55.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:55.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:55 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:55.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:55.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:55 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:55.540+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:55.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:55 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:55.540+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:55.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:55 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:55.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:55.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:55 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:55.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:55.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:55 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:55.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:55.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:55.749+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:55.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:55.749+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:55.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:55.749+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:57:55.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:55.695+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:55.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:55.695+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:55.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:55 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:55.695+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:56.672 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:56 smithi193 podman[71241]: 2024-04-22 06:57:56.406731617 +0000 UTC m=+6.145673274 container cleanup 8029dde9e57b2f9e6282122ae68251c6856657bd9b6abb49e80facf9d64d79e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:57:56.672 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:56 smithi193 bash[71241]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5 2024-04-22T06:57:56.672 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:56 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:56.294+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:56.672 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:56 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:56.294+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:57:56.939 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:56 smithi193 podman[71258]: 2024-04-22 06:57:56.674951337 +0000 UTC m=+0.773388325 container remove 8029dde9e57b2f9e6282122ae68251c6856657bd9b6abb49e80facf9d64d79e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:57:56.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:56 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:56.682+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:56.940 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:56 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:56.682+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:56.940 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:56 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:56.682+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:56.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:56 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:56.574+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:56.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:56 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:56.574+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:56.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:56 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:56.617+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:56.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:56 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:56.617+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:56.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:56 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:56.617+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:56.989 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:56 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:56.696+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:56.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:56 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:56.696+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:56.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:56 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:56.696+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:56.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:56 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:56.784+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:56.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:56 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:56.784+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:56.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:56 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:56.784+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:57:57.409 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:57 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:57.327+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:57.410 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:57 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:57.327+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:57:57.410 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:57 smithi193 podman[71349]: 2024-04-22 06:57:56.985689643 +0000 UTC m=+0.018206454 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:57.410 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:57 smithi193 podman[71349]: 2024-04-22 06:57:57.109358843 +0000 UTC m=+0.141875660 container create 5eb43da0fa56e07b977f9796524ced275469c325c345f4b3f4b9254b507c7b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:57.410 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:57 smithi193 podman[71349]: 2024-04-22 06:57:57.328677497 +0000 UTC m=+0.361194313 container init 5eb43da0fa56e07b977f9796524ced275469c325c345f4b3f4b9254b507c7b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T06:57:57.410 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:57 smithi193 podman[71349]: 2024-04-22 06:57:57.331750475 +0000 UTC m=+0.364267286 container start 5eb43da0fa56e07b977f9796524ced275469c325c345f4b3f4b9254b507c7b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T06:57:57.410 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:57 smithi193 podman[71349]: 2024-04-22 06:57:57.412348274 +0000 UTC m=+0.444865095 container attach 5eb43da0fa56e07b977f9796524ced275469c325c345f4b3f4b9254b507c7b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T06:57:57.687 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 22 06:57:57 smithi193 podman[71349]: 2024-04-22 06:57:57.62241229 +0000 UTC m=+0.654929092 container died 5eb43da0fa56e07b977f9796524ced275469c325c345f4b3f4b9254b507c7b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:57:57.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:57 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:57.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:57.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:57 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:57.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:57.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:57 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:57.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:57.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:57 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:57.624+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:57.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:57 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:57.624+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:57.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:57 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:57.585+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:57.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:57 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:57.585+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:57.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:57 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:57.585+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:57.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:57.810+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:57.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:57.810+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:57.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:57.810+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:57:57.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:57.722+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:57.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:57.722+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:57.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:57.722+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:58.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T06:57:58.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:58.383 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T06:57:58.383 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T06:57:58.383 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.6 2024-04-22T06:57:58.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:58 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:58.283+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:58.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:58 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:58.283+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:57:58.701 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 systemd[1]: Stopping Ceph osd.6 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:58.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:58 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:58.671+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:58.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:58 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:58.671+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:58.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:58 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:58.552+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:58.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:58 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:58.552+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:58.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:58 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:58.552+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:58.734+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:58.734+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:58.734+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:58.703+0000 7f363fe41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:58.703+0000 7f363fe41700 -1 osd.6 121 *** Got signal Terminated *** 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:58.703+0000 7f363fe41700 -1 osd.6 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:58.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:58.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:58.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:58.991 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:58 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:58.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:57:59.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:58 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:58.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:59.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:58 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:58.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:59.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:58 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:58.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:59.670 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:59 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:59.302+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:57:59.670 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:57:59 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:57:59.302+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:57:59.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:59 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:59.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:57:59.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:59 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:59.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:57:59.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:57:59 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:57:59.672+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:57:59.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:59.674+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:57:59.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:57:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:57:59.674+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:57:59.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:59.538+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:57:59.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:59.538+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:57:59.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:57:59 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:57:59.538+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:59.725+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:59.725+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:57:59.725+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:59.796+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:59.796+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:59.796+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:57:59.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:57:59 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:57:59.796+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.640983+0000 front 2024-04-22T06:57:35.640914+0000 (oldest deadline 2024-04-22T06:57:59.741088+0000) 2024-04-22T06:58:00.679 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:00 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:00.293+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:00.679 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:00 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:00.293+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:00.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:00 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:00.681+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:00.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:00 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:00.681+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:00.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:00 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:00.681+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:00.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:00 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:00.684+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:00.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:00 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:00.684+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:00.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:00 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:00.518+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:00.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:00 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:00.518+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:00.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:00 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:00.518+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:00.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:00 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:00.518+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:01.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:00.773+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:01.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:00.773+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:01.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:00.773+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:01.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:00.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:58:01.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:00.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:58:01.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:00.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:58:01.241 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:00 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:00.775+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.640983+0000 front 2024-04-22T06:57:35.640914+0000 (oldest deadline 2024-04-22T06:57:59.741088+0000) 2024-04-22T06:58:01.637 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:01.338+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:01.637 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:01.338+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:01.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:01.639+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:01.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:01.639+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:01.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:01 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:01.639+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:01.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:01 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:01.649+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:01.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:01 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:01.649+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:01.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:01 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:01.545+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:01.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:01 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:01.545+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:01.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:01 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:01.545+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:01.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:01 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:01.545+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:01.780+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:01.780+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:01.780+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:01.757+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:01.757+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:01.757+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:58:02.240 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:01.757+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.640983+0000 front 2024-04-22T06:57:35.640914+0000 (oldest deadline 2024-04-22T06:57:59.741088+0000) 2024-04-22T06:58:02.687 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:02.301+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:02.687 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:02.301+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:02.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:02.689+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:02.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:02.689+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:02.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:02 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:02.689+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:02.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:02 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:02.667+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:02.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:02 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:02.667+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:02.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:02 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:02.579+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:02.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:02 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:02.579+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:02.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:02 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:02.579+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:02.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:02 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:02.579+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:02.989 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:02.720+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:14.538170+0000 front 2024-04-22T06:57:14.537999+0000 (oldest deadline 2024-04-22T06:57:38.038210+0000) 2024-04-22T06:58:02.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:02.720+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:20.339718+0000 front 2024-04-22T06:57:20.339704+0000 (oldest deadline 2024-04-22T06:57:45.639558+0000) 2024-04-22T06:58:02.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:02.720+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:25.640044+0000 front 2024-04-22T06:57:25.640157+0000 (oldest deadline 2024-04-22T06:57:55.640617+0000) 2024-04-22T06:58:02.990 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6[53363]: 2024-04-22T06:58:02.720+0000 7f363842d700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.640983+0000 front 2024-04-22T06:57:35.640914+0000 (oldest deadline 2024-04-22T06:57:59.741088+0000) 2024-04-22T06:58:02.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:02.780+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:02.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:02.780+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:02.990 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:02 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:02.780+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:03.655 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:03.274+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:03.655 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:03.274+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:03.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:03.657+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:03.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:03.657+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:03.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:03 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:03.657+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:03.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:03.696+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:03.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:03.696+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:03.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:03.696+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:03.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:03.606+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:03.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:03.606+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:03.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:03.606+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:03.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:03 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:03.606+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:04.168 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:03 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:03.801+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:04.168 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:03 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:03.801+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:04.168 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:03 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:03.801+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:04.489 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:04 smithi079 podman[93005]: 2024-04-22 06:58:04.170721252 +0000 UTC m=+5.589151307 container died a3c201c66633c3c499085d3d654d5ca99df80380ebbfca782151c7dc6df9091e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:58:04.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:04.258+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:04.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:04.258+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:04.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:04.258+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:04.843 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:04 smithi079 podman[93005]: 2024-04-22 06:58:04.511534347 +0000 UTC m=+5.929964400 container cleanup a3c201c66633c3c499085d3d654d5ca99df80380ebbfca782151c7dc6df9091e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:58:04.843 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:04 smithi079 bash[93005]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6 2024-04-22T06:58:04.843 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:04 smithi079 podman[93020]: 2024-04-22 06:58:04.720929159 +0000 UTC m=+0.548773291 container remove a3c201c66633c3c499085d3d654d5ca99df80380ebbfca782151c7dc6df9091e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:04.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:04.650+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:04.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:04.650+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:04.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:04.650+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:04.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:04.612+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:04.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:04.612+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:04.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:04.612+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:04.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:04 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:04.612+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:05.111 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:04 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:04.844+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:05.111 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:04 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:04.844+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:05.111 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:04 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:04.844+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:05.111 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:04 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:04.844+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:05.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:04.705+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:05.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:04.705+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:05.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:04.705+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:05.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:04 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:04.705+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:05.371 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:05 smithi079 podman[93112]: 2024-04-22 06:58:05.113486904 +0000 UTC m=+0.102624904 container create 4f5ca281e698a689f1f15c192ad1c22055f1e410fe9e6c8a5a8856d38dff7dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T06:58:05.372 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:05 smithi079 podman[93112]: 2024-04-22 06:58:05.028449496 +0000 UTC m=+0.017587511 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:05.372 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:05 smithi079 podman[93112]: 2024-04-22 06:58:05.290699346 +0000 UTC m=+0.279837346 container init 4f5ca281e698a689f1f15c192ad1c22055f1e410fe9e6c8a5a8856d38dff7dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T06:58:05.372 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:05 smithi079 podman[93112]: 2024-04-22 06:58:05.294743173 +0000 UTC m=+0.283881173 container start 4f5ca281e698a689f1f15c192ad1c22055f1e410fe9e6c8a5a8856d38dff7dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5) 2024-04-22T06:58:05.685 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:05.283+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:05.685 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:05.284+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:05.685 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:05.284+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:05.740 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:05 smithi079 podman[93112]: 2024-04-22 06:58:05.37418884 +0000 UTC m=+0.363326863 container attach 4f5ca281e698a689f1f15c192ad1c22055f1e410fe9e6c8a5a8856d38dff7dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-22T06:58:05.740 INFO:journalctl@ceph.osd.6.smithi079.stdout:Apr 22 06:58:05 smithi079 podman[93112]: 2024-04-22 06:58:05.588478559 +0000 UTC m=+0.577616672 container died 4f5ca281e698a689f1f15c192ad1c22055f1e410fe9e6c8a5a8856d38dff7dfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:58:05.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:05.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:05.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:05.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:05.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:05.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:05.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:05 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:05.688+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:05.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:05.654+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:05.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:05.654+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:05.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:05.654+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:05.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:05.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:05.954 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:05.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:05.954 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:05.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:05.954 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:05 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:05.653+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:06.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:05 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:05.835+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:06.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:05 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:05.835+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:06.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:05 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:05.835+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:06.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:05 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:05.835+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:06.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T06:58:06.352 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:06.352 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T06:58:06.352 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T06:58:06.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.7 2024-04-22T06:58:06.653 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 systemd[1]: Stopping Ceph osd.7 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:06.653 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:06.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:06.653 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:06.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:06.653 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:06.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:06.682 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:06.271+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:06.682 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:06.271+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:06.682 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:06.271+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:06.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:06.684+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:06.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:06.684+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:06.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:06.684+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:06.940 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:06 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:06.684+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:06.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:06.684+0000 7fbd1ef86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:06.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:06.684+0000 7fbd1ef86700 -1 osd.7 121 *** Got signal Terminated *** 2024-04-22T06:58:06.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:06.684+0000 7fbd1ef86700 -1 osd.7 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:06.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:06.655+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:06.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:06.655+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:06.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:06.655+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:06.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:06.655+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:07.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:06 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:06.861+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:07.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:06 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:06.861+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:07.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:06 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:06.861+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:07.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:06 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:06.861+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:07.666 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:07.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:07.666 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:07.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:07.666 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:07.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:07.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:07.668+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:07.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:07.668+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:07.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:07.668+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:07.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:07 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:07.668+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:07.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:07.576+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:07.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:07.576+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:07.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:07.576+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:07.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:07.645+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:07.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:07.645+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:07.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:07.645+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:07.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:07 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:07.645+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:08.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:07 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:07.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:08.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:07 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:07.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:08.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:07 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:07.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:08.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:07 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:07.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:08.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:08.282+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:08.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:08.282+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:08.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:08.282+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:08.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:08.658+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:08.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:08.658+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:08.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:08.658+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:08.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:08.658+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:08.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:08.555+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:08.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:08.555+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:08.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:08.555+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:08.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:08 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:08.555+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:42.959581+0000 front 2024-04-22T06:57:42.959715+0000 (oldest deadline 2024-04-22T06:58:08.259713+0000) 2024-04-22T06:58:09.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:08.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:09.200 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:08.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:09.200 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:08.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:09.200 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:08 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:08.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:09.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:08 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:08.803+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:09.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:08 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:08.803+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:09.241 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:08 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:08.803+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:09.241 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:08 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:08.803+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:09.241 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:08 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:08.803+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:09.469 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:09.244+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:09.469 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:09.244+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:09.469 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:09.244+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:09.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:09.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:09.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:09.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:09.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:09.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:09.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:09.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:09.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:09.580+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:09.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:09.580+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:09.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:09.580+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:09.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:09 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:09.580+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:42.959581+0000 front 2024-04-22T06:57:42.959715+0000 (oldest deadline 2024-04-22T06:58:08.259713+0000) 2024-04-22T06:58:10.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:09.737+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:10.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:09.737+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:10.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:09.737+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:10.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:09 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:09.737+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:10.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:09.831+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:10.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:09.831+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:10.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:09.831+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:10.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:09.831+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:10.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:09 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:09.831+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:10.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:10.253+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:10.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:10.253+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:10.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:10.253+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:10.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:10.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:10.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:10.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:10.608+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:42.959581+0000 front 2024-04-22T06:57:42.959715+0000 (oldest deadline 2024-04-22T06:58:08.259713+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:10.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:10.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:10.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:10.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:10 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:10.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:11.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:10.717+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:11.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:10.717+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:11.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:10.717+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:11.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:10.717+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:11.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:10 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:10.717+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:11.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:10.825+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:11.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:10.825+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:11.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:10.825+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:11.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:10.825+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:11.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:10 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:10.825+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:11.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:11.302+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:11.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:11.302+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:11.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:11.302+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:11.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:11.302+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:11.952 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:11.596+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.356369+0000 front 2024-04-22T06:57:21.356534+0000 (oldest deadline 2024-04-22T06:57:45.456476+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:11.596+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.657964+0000 front 2024-04-22T06:57:30.657900+0000 (oldest deadline 2024-04-22T06:57:52.357920+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:11.596+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.259324+0000 front 2024-04-22T06:57:38.259127+0000 (oldest deadline 2024-04-22T06:58:02.959102+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7[49657]: 2024-04-22T06:58:11.596+0000 7fbd17572700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:42.959581+0000 front 2024-04-22T06:57:42.959715+0000 (oldest deadline 2024-04-22T06:58:08.259713+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:11.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:11.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:11.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:11.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:11.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:12.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:11.733+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:12.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:11.733+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:12.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:11.733+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:12.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:11.733+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:12.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:11 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:11.733+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:12.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:11.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:12.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:11.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:12.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:11.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:12.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:11.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:12.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:11 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:11.852+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:12.579 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:12 smithi190 podman[69949]: 2024-04-22 06:58:12.267569081 +0000 UTC m=+5.715496456 container died 73760e0490252214f443c5d4e729b1df1a2950a9134dd17937c2b4dbed9821c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T06:58:12.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:12.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:12.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:12.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:12.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:12.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:12.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:12.261+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:12 smithi190 podman[69949]: 2024-04-22 06:58:12.68919605 +0000 UTC m=+6.137123425 container cleanup 73760e0490252214f443c5d4e729b1df1a2950a9134dd17937c2b4dbed9821c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:12 smithi190 bash[69949]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:12 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:12.717+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:12 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:12.717+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:12 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:12.717+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:12 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:12.717+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:12.938 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:12 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:12.717+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:13.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:12.740+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:13.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:12.740+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:13.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:12.740+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:13.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:12.740+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:13.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:12 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:12.740+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:13.202 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:12 smithi190 podman[69967]: 2024-04-22 06:58:12.940575612 +0000 UTC m=+0.670690362 container remove 73760e0490252214f443c5d4e729b1df1a2950a9134dd17937c2b4dbed9821c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:13.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:12.875+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:13.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:12.875+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:13.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:12.875+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:13.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:12.875+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:13.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:12 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:12.875+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:13.666 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:13 smithi190 podman[70060]: 2024-04-22 06:58:13.26990695 +0000 UTC m=+0.017698261 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:13.666 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:13 smithi190 podman[70060]: 2024-04-22 06:58:13.391549245 +0000 UTC m=+0.139340590 container create 901a763f7f5593feacb37d34048af754ef5353bddb4444249a9d810d7be350a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T06:58:13.666 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:13 smithi190 podman[70060]: 2024-04-22 06:58:13.5856691 +0000 UTC m=+0.333460400 container init 901a763f7f5593feacb37d34048af754ef5353bddb4444249a9d810d7be350a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:58:13.666 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:13 smithi190 podman[70060]: 2024-04-22 06:58:13.588774887 +0000 UTC m=+0.336566186 container start 901a763f7f5593feacb37d34048af754ef5353bddb4444249a9d810d7be350a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:13.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:13.254+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:13.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:13.254+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:13.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:13.255+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:13.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:13.255+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:13 smithi190 podman[70060]: 2024-04-22 06:58:13.694705094 +0000 UTC m=+0.442496404 container attach 901a763f7f5593feacb37d34048af754ef5353bddb4444249a9d810d7be350a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 22 06:58:13 smithi190 podman[70060]: 2024-04-22 06:58:13.887625368 +0000 UTC m=+0.635416673 container died 901a763f7f5593feacb37d34048af754ef5353bddb4444249a9d810d7be350a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:13 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:13.668+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:13 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:13.668+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:13 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:13.668+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:13 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:13.668+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:13.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:13 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:13.668+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:14.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:13.724+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:14.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:13.724+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:14.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:13.724+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:14.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:13.724+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:14.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:13 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:13.724+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:14.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:13.912+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:14.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:13.912+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:14.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:13.912+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:14.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:13.912+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:14.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:13 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:13.912+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:14.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T06:58:14.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:14.562 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T06:58:14.562 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-22T06:58:14.562 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.8 2024-04-22T06:58:14.590 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.227+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:14.590 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.227+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:14.590 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.227+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:14.590 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.227+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:14.775 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:14 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:14.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:14.775 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:14 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:14.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:14.776 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:14 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:14.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:14.776 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:14 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:14.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:14.776 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:14 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:14.667+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:14.883 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 systemd[1]: Stopping Ceph osd.8 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:14.884 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:14.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:14.884 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:14.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:14.884 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:14.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:14.884 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:14.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:14.884 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:14.699+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:15.189 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.885+0000 7f74ee0d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:15.189 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.885+0000 7f74ee0d0700 -1 osd.8 121 *** Got signal Terminated *** 2024-04-22T06:58:15.189 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:14 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:14.885+0000 7f74ee0d0700 -1 osd.8 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:15.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:14.941+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:15.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:14.941+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:15.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:14.941+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:15.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:14.941+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:15.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:14 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:14.941+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:15.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:15.224+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:15.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:15.224+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:15.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:15.224+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:15.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:15.224+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:15.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:15.641+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:15.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:15.641+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:15.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:15.641+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:15.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:15.641+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:15.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:15.641+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:16.161 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:15.894+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:16.161 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:15.894+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:16.162 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:15.894+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:16.162 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:15.894+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:16.162 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:15 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:15.894+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:16.182 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:15.706+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:16.182 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:15.706+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:16.182 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:15.706+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:16.182 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:15.706+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:16.182 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:15 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:15.706+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:16.439 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:16.183+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:16.439 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:16.183+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:16.439 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:16.183+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:16.439 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:16.183+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:16.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:16.682+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:16.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:16.682+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:16.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:16.682+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:16.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:16.682+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:16.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:16 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:16.682+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:17.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:16.752+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:17.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:16.752+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:17.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:16.752+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:17.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:16.752+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:17.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:16.752+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:17.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:16 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:16.752+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:17.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:16.917+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:17.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:16.917+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:17.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:16.917+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:17.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:16.917+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:17.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:16.917+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:17.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:16 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:16.917+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:17.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:17.198+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:17.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:17.198+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:17.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:17.198+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:17.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:17.198+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:17.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:17.198+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:50.515571+0000 front 2024-04-22T06:57:50.515592+0000 (oldest deadline 2024-04-22T06:58:16.415808+0000) 2024-04-22T06:58:17.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:17.700+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:17.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:17.700+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:17.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:17.700+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:17.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:17.700+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:17.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:17.700+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:17.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:17 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:17.700+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:18.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:17.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:18.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:17.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:18.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:17.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:18.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:17.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:18.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:17.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:18.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:17 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:17.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:18.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:17.882+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:18.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:17.882+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:18.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:17.882+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:18.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:17.882+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:18.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:17.882+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:18.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:17 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:17.882+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:18.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:18.245+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:18.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:18.245+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:18.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:18.245+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:18.690 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:18.245+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:18.690 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:18.245+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:50.515571+0000 front 2024-04-22T06:57:50.515592+0000 (oldest deadline 2024-04-22T06:58:16.415808+0000) 2024-04-22T06:58:19.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:18.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:19.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:18.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:19.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:18.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:19.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:18.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:19.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:18.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:19.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:18 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:18.708+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:19.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:18.720+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:19.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:18.720+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:19.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:18.720+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:19.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:18.720+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:19.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:18.720+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:19.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:18 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:18.720+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:19.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:18.909+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:19.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:18.909+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:19.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:18.909+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:19.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:18.909+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:19.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:18.909+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:19.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:18 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:18.909+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:19.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:19.218+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.212239+0000 front 2024-04-22T06:57:21.212274+0000 (oldest deadline 2024-04-22T06:57:44.712197+0000) 2024-04-22T06:58:19.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:19.218+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.013269+0000 front 2024-04-22T06:57:30.013312+0000 (oldest deadline 2024-04-22T06:57:55.313418+0000) 2024-04-22T06:58:19.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:19.218+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.313994+0000 front 2024-04-22T06:57:35.314029+0000 (oldest deadline 2024-04-22T06:58:04.114393+0000) 2024-04-22T06:58:19.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:19.218+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:45.215443+0000 front 2024-04-22T06:57:45.215489+0000 (oldest deadline 2024-04-22T06:58:10.515350+0000) 2024-04-22T06:58:19.689 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8[50778]: 2024-04-22T06:58:19.218+0000 7f74e66bc700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:50.515571+0000 front 2024-04-22T06:57:50.515592+0000 (oldest deadline 2024-04-22T06:58:16.415808+0000) 2024-04-22T06:58:19.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:19.684+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:19.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:19.684+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:19.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:19.684+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:19.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:19.684+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:19.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:19.684+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:19.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:19.684+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:20.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:19.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:20.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:19.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:20.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:19.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:20.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:19.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:20.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:19.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:20.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:19 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:19.692+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:20.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:19.881+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:20.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:19.881+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:20.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:19.881+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:20.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:19.881+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:20.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:19.881+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:20.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:19 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:19.881+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:20.776 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:20 smithi193 podman[71465]: 2024-04-22 06:58:20.49734357 +0000 UTC m=+5.731841052 container died a56c72d62d3298af3de38f74b573c0f774c1b6e1fd6e3e55b004b11f0694321d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:58:20.776 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:20.695+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:20.776 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:20.695+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:20.776 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:20.695+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:20.776 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:20.695+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:20.777 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:20.695+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:20.777 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:20 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:20.695+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:21.177 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:20 smithi193 podman[71465]: 2024-04-22 06:58:20.88637548 +0000 UTC m=+6.120872953 container cleanup a56c72d62d3298af3de38f74b573c0f774c1b6e1fd6e3e55b004b11f0694321d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T06:58:21.177 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:20 smithi193 bash[71465]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8 2024-04-22T06:58:21.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:20.716+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:21.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:20.716+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:21.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:20.716+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:21.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:20.716+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:21.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:20.716+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:21.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:20 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:20.716+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:21.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:20.926+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:21.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:20.926+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:21.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:20.926+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:21.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:20.926+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:21.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:20.926+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:21.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:20 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:20.926+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:21.439 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:21 smithi193 podman[71482]: 2024-04-22 06:58:21.179510029 +0000 UTC m=+0.680507647 container remove a56c72d62d3298af3de38f74b573c0f774c1b6e1fd6e3e55b004b11f0694321d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T06:58:21.813 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:21.664+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:21.813 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:21.664+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:21.664+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:21.664+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:21.664+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:21 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:21.664+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:21 smithi193 podman[71568]: 2024-04-22 06:58:21.488677201 +0000 UTC m=+0.109783821 container create 165eda2210edcf8ce1189694c69e8f18bf7b8804991c9d3bdcb7de0fae4c7908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:21 smithi193 podman[71568]: 2024-04-22 06:58:21.397048222 +0000 UTC m=+0.018154858 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:21 smithi193 podman[71568]: 2024-04-22 06:58:21.699621453 +0000 UTC m=+0.320728073 container init 165eda2210edcf8ce1189694c69e8f18bf7b8804991c9d3bdcb7de0fae4c7908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:21 smithi193 podman[71568]: 2024-04-22 06:58:21.702684278 +0000 UTC m=+0.323790900 container start 165eda2210edcf8ce1189694c69e8f18bf7b8804991c9d3bdcb7de0fae4c7908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T06:58:21.814 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:21 smithi193 podman[71568]: 2024-04-22 06:58:21.816226158 +0000 UTC m=+0.437332780 container attach 165eda2210edcf8ce1189694c69e8f18bf7b8804991c9d3bdcb7de0fae4c7908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:58:21.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:21.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:21.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:21.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:21.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:21.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:21.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:21.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:21.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:21.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:21.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:21 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:21.690+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:22.189 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 22 06:58:22 smithi193 podman[71568]: 2024-04-22 06:58:22.001570907 +0000 UTC m=+0.622677527 container died 165eda2210edcf8ce1189694c69e8f18bf7b8804991c9d3bdcb7de0fae4c7908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T06:58:22.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:21.953+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:22.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:21.953+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:22.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:21.953+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:22.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:21.953+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:22.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:21.953+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:22.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:21 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:21.953+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:22.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.8.service' 2024-04-22T06:58:22.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:22.756 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-22T06:58:22.756 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-22T06:58:22.756 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.9 2024-04-22T06:58:22.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:22.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:22.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:22.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:22.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:22.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:22.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:22.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:22.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:22.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:22.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:22 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:22.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:22.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:22.659+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:22.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:22.659+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:22.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:22.659+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:22.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:22.659+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:22.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:22.659+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:22.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:22 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:22.659+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:23.077 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:22 smithi079 systemd[1]: Stopping Ceph osd.9 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:23.077 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:22.999+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:23.077 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:22.999+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:23.077 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:22.999+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:23.078 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:22.999+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:23.078 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:22.999+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:23.078 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:22.999+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:23.078 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:23.078+0000 7fc962b43700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:23.078 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:23.078+0000 7fc962b43700 -1 osd.9 121 *** Got signal Terminated *** 2024-04-22T06:58:23.078 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:23 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:23.078+0000 7fc962b43700 -1 osd.9 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:23.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:23.637+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:23.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:23.637+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:23.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:23.637+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:23.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:23.637+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:23.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:23.637+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:23.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:23 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:23.637+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:24.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:23.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:24.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:23.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:24.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:23.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:24.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:23.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:24.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:23.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:24.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:23.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:24.489 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:24.031+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:24.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:24.031+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:24.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:24.031+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:24.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:24.031+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:24.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:24.031+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:24.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:24 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:24.031+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:24.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:24 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:24.683+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:25.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:24.712+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:25.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:24.712+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:25.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:24.712+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:25.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:24.712+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:25.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:24.712+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:25.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:24 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:24.712+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:25.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:25.040+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:25.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:25.040+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:25.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:25.040+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:25.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:25.040+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:25.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:25.040+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:25.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:25 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:25.040+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:26.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:25 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:25.760+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:26.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:25 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:25.760+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:26.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:25 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:25.760+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:26.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:25 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:25.760+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:26.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:25 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:25.760+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:26.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:25 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:25.760+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:26.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:26.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:26.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:26.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:26.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:26.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:26.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:25 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:25.721+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:26.489 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:26.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:26.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:26.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:26.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:26.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:26.490 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.000+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:58:00.065544+0000 front 2024-04-22T06:58:00.065563+0000 (oldest deadline 2024-04-22T06:58:25.965566+0000) 2024-04-22T06:58:26.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:26.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:26.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:26.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:26.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:26.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:26.953 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:26 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:26.694+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:27.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:26.720+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:27.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:26.720+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:27.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:26.720+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:27.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:26.720+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:27.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:26.720+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:27.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:26 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:26.720+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:27.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:26 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:26.970+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:58:00.065544+0000 front 2024-04-22T06:58:00.065563+0000 (oldest deadline 2024-04-22T06:58:25.965566+0000) 2024-04-22T06:58:27.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:27.685+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:27.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:27.685+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:27.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:27.685+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:27.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:27.685+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:27.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:27.685+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:27.940 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:27.685+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:28.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:28.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:28.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:28.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:28.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:28.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:28.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:27 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:27.732+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:28.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.357090+0000 front 2024-04-22T06:57:11.357129+0000 (oldest deadline 2024-04-22T06:57:37.257166+0000) 2024-04-22T06:58:28.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.858797+0000 front 2024-04-22T06:57:21.858784+0000 (oldest deadline 2024-04-22T06:57:47.158679+0000) 2024-04-22T06:58:28.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:29.459931+0000 front 2024-04-22T06:57:29.459792+0000 (oldest deadline 2024-04-22T06:57:51.759903+0000) 2024-04-22T06:58:28.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:35.161259+0000 front 2024-04-22T06:57:35.161319+0000 (oldest deadline 2024-04-22T06:58:03.961785+0000) 2024-04-22T06:58:28.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.763429+0000 front 2024-04-22T06:57:46.763347+0000 (oldest deadline 2024-04-22T06:58:07.863232+0000) 2024-04-22T06:58:28.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:54.864546+0000 front 2024-04-22T06:57:54.864630+0000 (oldest deadline 2024-04-22T06:58:16.564543+0000) 2024-04-22T06:58:28.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:27 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9[60371]: 2024-04-22T06:58:27.924+0000 7fc95b12f700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:58:00.065544+0000 front 2024-04-22T06:58:00.065563+0000 (oldest deadline 2024-04-22T06:58:25.965566+0000) 2024-04-22T06:58:28.739 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:28 smithi079 podman[93228]: 2024-04-22 06:58:28.463511249 +0000 UTC m=+5.507867912 container died 52583edaeda5c33517feccc3662eff4f6c5fa697f390431fe00f848e9f18c8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T06:58:29.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:28.702+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:29.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:28.702+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:29.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:28.702+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:29.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:28.702+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:29.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:28.702+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:29.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:28 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:28.702+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:29.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:29.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:29.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:29.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:29.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:29.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:29.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:28 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:28.763+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:29.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:28 smithi079 podman[93228]: 2024-04-22 06:58:28.77852768 +0000 UTC m=+5.822884344 container cleanup 52583edaeda5c33517feccc3662eff4f6c5fa697f390431fe00f848e9f18c8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:29.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:28 smithi079 bash[93228]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9 2024-04-22T06:58:29.239 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93247]: 2024-04-22 06:58:29.013162826 +0000 UTC m=+0.547806766 container remove 52583edaeda5c33517feccc3662eff4f6c5fa697f390431fe00f848e9f18c8d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T06:58:29.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93333]: 2024-04-22 06:58:29.438941469 +0000 UTC m=+0.111080740 container create 8caca9c38f99e8b897f00baec1e9a66ab30f3da6db7b8c1d8e6c9d468d04ac6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T06:58:29.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93333]: 2024-04-22 06:58:29.345440055 +0000 UTC m=+0.017579291 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:29.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93333]: 2024-04-22 06:58:29.675019254 +0000 UTC m=+0.347158480 container init 8caca9c38f99e8b897f00baec1e9a66ab30f3da6db7b8c1d8e6c9d468d04ac6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T06:58:29.740 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93333]: 2024-04-22 06:58:29.678639349 +0000 UTC m=+0.350778579 container start 8caca9c38f99e8b897f00baec1e9a66ab30f3da6db7b8c1d8e6c9d468d04ac6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:30.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:29.704+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:30.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:29.704+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:30.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:29.704+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:30.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:29.704+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:30.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:29.704+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:30.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:29 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:29.704+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:30.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:30.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:30.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:30.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:30.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:30.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:30.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:29 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:29.806+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:30.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93333]: 2024-04-22 06:58:29.766013063 +0000 UTC m=+0.438152289 container attach 8caca9c38f99e8b897f00baec1e9a66ab30f3da6db7b8c1d8e6c9d468d04ac6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:58:30.240 INFO:journalctl@ceph.osd.9.smithi079.stdout:Apr 22 06:58:29 smithi079 podman[93333]: 2024-04-22 06:58:29.973899855 +0000 UTC m=+0.646039075 container died 8caca9c38f99e8b897f00baec1e9a66ab30f3da6db7b8c1d8e6c9d468d04ac6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:30.556 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.9.service' 2024-04-22T06:58:30.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:30.594 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-22T06:58:30.594 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-22T06:58:30.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.10 2024-04-22T06:58:30.891 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 systemd[1]: Stopping Ceph osd.10 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:30.891 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:30.891 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:30.891 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:30.891 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:30.891 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:30.892 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:30.892 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.795+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:31.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:30.754+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:31.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:30.754+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:31.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:30.754+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:31.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:30.754+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:31.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:30.754+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:31.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:30 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:30.754+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:31.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.892+0000 7fc856b75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:31.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.892+0000 7fc856b75700 -1 osd.10 121 *** Got signal Terminated *** 2024-04-22T06:58:31.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:30 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:30.892+0000 7fc856b75700 -1 osd.10 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:32.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:31.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:32.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:31.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:32.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:31.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:32.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:31.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:32.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:31.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:32.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:31.725+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:32.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:32.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:32.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:32.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:32.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:32.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:32.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:31 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:31.842+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:33.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:32.735+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:33.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:32.735+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:33.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:32.735+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:33.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:32.735+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:33.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:32.735+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:33.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:32 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:32.735+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:33.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:33.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:33.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:33.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:33.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:33.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:33.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:32 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:32.800+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:34.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:33.776+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:34.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:33.776+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:34.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:33.776+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:34.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:33.776+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:34.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:33.776+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:34.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:33 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:33.776+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:34.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:34.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:34.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:34.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:34.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:34.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:34.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:33 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:33.779+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:35.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:34 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:34.784+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:35.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:34 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:34.784+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:35.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:34 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:34.784+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:35.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:34 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:34.784+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:35.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:34 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:34.784+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:35.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:34 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:34.784+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:35.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:35.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:35.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:35.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:35.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:35.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:35.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:34 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:34.746+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:36.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:35.806+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:36.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:35.806+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:36.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:35.806+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:36.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:35.806+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:36.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:35.806+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:36.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:35.806+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:36.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:11.077881+0000 front 2024-04-22T06:57:11.077857+0000 (oldest deadline 2024-04-22T06:57:36.977490+0000) 2024-04-22T06:58:36.202 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.078643+0000 front 2024-04-22T06:57:21.078598+0000 (oldest deadline 2024-04-22T06:57:43.978726+0000) 2024-04-22T06:58:36.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:26.279723+0000 front 2024-04-22T06:57:26.279733+0000 (oldest deadline 2024-04-22T06:57:52.179891+0000) 2024-04-22T06:58:36.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:36.280855+0000 front 2024-04-22T06:57:36.280812+0000 (oldest deadline 2024-04-22T06:58:00.380770+0000) 2024-04-22T06:58:36.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:46.782332+0000 front 2024-04-22T06:57:46.782287+0000 (oldest deadline 2024-04-22T06:58:12.082488+0000) 2024-04-22T06:58:36.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:52.083349+0000 front 2024-04-22T06:57:52.083364+0000 (oldest deadline 2024-04-22T06:58:17.383131+0000) 2024-04-22T06:58:36.203 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:35 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10[56674]: 2024-04-22T06:58:35.735+0000 7fc853169700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.79:6822 osd.6 since back 2024-04-22T06:57:59.084196+0000 front 2024-04-22T06:57:59.084188+0000 (oldest deadline 2024-04-22T06:58:24.383994+0000) 2024-04-22T06:58:36.692 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:36 smithi190 podman[70179]: 2024-04-22 06:58:36.421052465 +0000 UTC m=+5.630320484 container died 6cee66cf8276bbee48ba47e62564e30969da41c98d0f2b4dc08b3e97ca1a28ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T06:58:36.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:36 smithi190 podman[70179]: 2024-04-22 06:58:36.793998252 +0000 UTC m=+6.003266273 container cleanup 6cee66cf8276bbee48ba47e62564e30969da41c98d0f2b4dc08b3e97ca1a28ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:36.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:36 smithi190 bash[70179]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:37.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:36 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:36.772+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:37.404 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70198]: 2024-04-22 06:58:37.070426214 +0000 UTC m=+0.646198099 container remove 6cee66cf8276bbee48ba47e62564e30969da41c98d0f2b4dc08b3e97ca1a28ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T06:58:37.679 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70289]: 2024-04-22 06:58:37.310275385 +0000 UTC m=+0.018080592 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:37.680 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70289]: 2024-04-22 06:58:37.413316027 +0000 UTC m=+0.121121213 container create 0aed6d22d4be24993aa1b40b6a5d17227e9147403c24b2f645dbfa47bd64907d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T06:58:37.680 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70289]: 2024-04-22 06:58:37.615721222 +0000 UTC m=+0.323526407 container init 0aed6d22d4be24993aa1b40b6a5d17227e9147403c24b2f645dbfa47bd64907d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:58:37.680 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70289]: 2024-04-22 06:58:37.618820702 +0000 UTC m=+0.326625884 container start 0aed6d22d4be24993aa1b40b6a5d17227e9147403c24b2f645dbfa47bd64907d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T06:58:37.680 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70289]: 2024-04-22 06:58:37.68232804 +0000 UTC m=+0.390133229 container attach 0aed6d22d4be24993aa1b40b6a5d17227e9147403c24b2f645dbfa47bd64907d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:58:37.952 INFO:journalctl@ceph.osd.10.smithi190.stdout:Apr 22 06:58:37 smithi190 podman[70289]: 2024-04-22 06:58:37.9160426 +0000 UTC m=+0.623847792 container died 0aed6d22d4be24993aa1b40b6a5d17227e9147403c24b2f645dbfa47bd64907d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:38.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:37 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:37.812+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:38.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.10.service' 2024-04-22T06:58:38.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:38.559 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-22T06:58:38.559 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-22T06:58:38.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.11 2024-04-22T06:58:38.870 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 systemd[1]: Stopping Ceph osd.11 for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.764+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.872+0000 7fb3c69df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.872+0000 7fb3c69df700 -1 osd.11 121 *** Got signal Terminated *** 2024-04-22T06:58:38.871 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:38 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:38.872+0000 7fb3c69df700 -1 osd.11 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:40.090 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:40.090 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:40.091 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:40.091 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:40.091 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:40.091 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:40.091 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:39 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:39.723+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:41.110 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:41.110 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:41.111 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:41.111 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:41.111 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:41.111 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:41.111 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:40 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:40.703+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:42.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:41 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:41.707+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:43.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:43.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:43.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:43.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:43.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:43.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:43.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:42 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:42.700+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-04-22T06:57:13.090675+0000 front 2024-04-22T06:57:13.090572+0000 (oldest deadline 2024-04-22T06:57:38.291070+0000) 2024-04-22T06:58:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6804 osd.1 since back 2024-04-22T06:57:21.792309+0000 front 2024-04-22T06:57:21.792213+0000 (oldest deadline 2024-04-22T06:57:44.692204+0000) 2024-04-22T06:58:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-22T06:57:30.493536+0000 front 2024-04-22T06:57:30.493638+0000 (oldest deadline 2024-04-22T06:57:52.813168+0000) 2024-04-22T06:58:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.79:6814 osd.3 since back 2024-04-22T06:57:38.714130+0000 front 2024-04-22T06:57:38.714164+0000 (oldest deadline 2024-04-22T06:58:04.614017+0000) 2024-04-22T06:58:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6812 osd.4 since back 2024-04-22T06:57:44.614642+0000 front 2024-04-22T06:57:44.614577+0000 (oldest deadline 2024-04-22T06:58:10.514532+0000) 2024-04-22T06:58:43.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-22T06:57:55.115719+0000 front 2024-04-22T06:57:55.115699+0000 (oldest deadline 2024-04-22T06:58:16.215807+0000) 2024-04-22T06:58:43.940 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:43 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11[57492]: 2024-04-22T06:58:43.676+0000 7fb3befcb700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.190:6820 osd.7 since back 2024-04-22T06:58:10.718787+0000 front 2024-04-22T06:58:10.718820+0000 (oldest deadline 2024-04-22T06:58:36.018730+0000) 2024-04-22T06:58:44.596 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:44 smithi193 podman[71687]: 2024-04-22 06:58:44.31800892 +0000 UTC m=+5.563833946 container died b4320fd88cd46470b94b667f3f09eea3234b3bed481f8ea995bc098e6ebeb4d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:44.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:44 smithi193 podman[71687]: 2024-04-22 06:58:44.781386492 +0000 UTC m=+6.027211560 container cleanup b4320fd88cd46470b94b667f3f09eea3234b3bed481f8ea995bc098e6ebeb4d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T06:58:44.939 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:44 smithi193 bash[71687]: ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11 2024-04-22T06:58:45.405 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71717]: 2024-04-22 06:58:45.065998211 +0000 UTC m=+0.740827349 container remove b4320fd88cd46470b94b667f3f09eea3234b3bed481f8ea995bc098e6ebeb4d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:58:45.689 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71809]: 2024-04-22 06:58:45.311727788 +0000 UTC m=+0.018347476 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:45.689 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71809]: 2024-04-22 06:58:45.408525885 +0000 UTC m=+0.115145588 container create fbcf4cdc647a1dc37e7c13fb9514a994a51576e16e7479e8ea73a4b5123fc936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:45.689 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71809]: 2024-04-22 06:58:45.619392305 +0000 UTC m=+0.326011978 container init fbcf4cdc647a1dc37e7c13fb9514a994a51576e16e7479e8ea73a4b5123fc936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:45.689 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71809]: 2024-04-22 06:58:45.62237937 +0000 UTC m=+0.328999032 container start fbcf4cdc647a1dc37e7c13fb9514a994a51576e16e7479e8ea73a4b5123fc936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T06:58:46.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71809]: 2024-04-22 06:58:45.731018132 +0000 UTC m=+0.437637804 container attach fbcf4cdc647a1dc37e7c13fb9514a994a51576e16e7479e8ea73a4b5123fc936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T06:58:46.189 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:45 smithi193 podman[71809]: 2024-04-22 06:58:45.920454371 +0000 UTC m=+0.627074036 container died fbcf4cdc647a1dc37e7c13fb9514a994a51576e16e7479e8ea73a4b5123fc936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:58:46.475 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 22 06:58:46 smithi193 podman[71835]: 2024-04-22 06:58:46.464635791 +0000 UTC m=+0.538778492 container remove fbcf4cdc647a1dc37e7c13fb9514a994a51576e16e7479e8ea73a4b5123fc936 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T06:58:46.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@osd.11.service' 2024-04-22T06:58:46.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:46.518 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-22T06:58:46.518 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-22T06:58:46.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.a 2024-04-22T06:58:46.843 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:58:46 smithi079 systemd[1]: Stopping Ceph mds.a for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:47.171 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:58:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a[66922]: 2024-04-22T06:58:46.845+0000 7fe39277a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:47.171 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:58:46 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a[66922]: 2024-04-22T06:58:46.845+0000 7fe39277a700 -1 mds.a *** got signal Terminated *** 2024-04-22T06:58:51.151 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:58:50 smithi079 podman[93452]: 2024-04-22 06:58:50.854934649 +0000 UTC m=+4.142926758 container died 4dd988509095d178e2681aeb34539a19d62ba5161abcbf80288088f5b5796292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T06:58:51.489 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:58:51 smithi079 podman[93452]: 2024-04-22 06:58:51.236143408 +0000 UTC m=+4.524135516 container cleanup 4dd988509095d178e2681aeb34539a19d62ba5161abcbf80288088f5b5796292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:51.490 INFO:journalctl@ceph.mds.a.smithi079.stdout:Apr 22 06:58:51 smithi079 bash[93452]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-a 2024-04-22T06:58:51.647 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.a.service' 2024-04-22T06:58:51.682 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:51.682 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-22T06:58:51.682 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-22T06:58:51.682 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.d 2024-04-22T06:58:51.989 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:58:51 smithi079 systemd[1]: Stopping Ceph mds.d for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:52.490 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:58:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d[65971]: 2024-04-22T06:58:52.006+0000 7f0a342ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:52.490 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:58:52 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d[65971]: 2024-04-22T06:58:52.006+0000 7f0a342ad700 -1 mds.d *** got signal Terminated *** 2024-04-22T06:58:56.154 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:58:55 smithi079 podman[93586]: 2024-04-22 06:58:55.798751666 +0000 UTC m=+3.904893482 container died 833ef07e27f13a3f06a3e92d217ffa23361b80afb12f332759eaeedaae470cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T06:58:56.489 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:58:56 smithi079 podman[93586]: 2024-04-22 06:58:56.247909422 +0000 UTC m=+4.354051235 container cleanup 833ef07e27f13a3f06a3e92d217ffa23361b80afb12f332759eaeedaae470cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T06:58:56.489 INFO:journalctl@ceph.mds.d.smithi079.stdout:Apr 22 06:58:56 smithi079 bash[93586]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-d 2024-04-22T06:58:56.668 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.d.service' 2024-04-22T06:58:56.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:56.701 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-22T06:58:56.701 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-22T06:58:56.701 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.g 2024-04-22T06:58:57.239 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:58:56 smithi079 systemd[1]: Stopping Ceph mds.g for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:57.240 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:58:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g[66576]: 2024-04-22T06:58:57.026+0000 7f78d9857700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:57.240 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:58:57 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g[66576]: 2024-04-22T06:58:57.026+0000 7f78d9857700 -1 mds.g *** got signal Terminated *** 2024-04-22T06:59:00.990 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:59:00 smithi079 podman[93707]: 2024-04-22 06:59:00.695148443 +0000 UTC m=+3.769616408 container died 23d4fa0bb5372a69dc3d50260b42408ca7a8876d0a340bfee03bc6e31f8c593b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T06:59:01.366 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:59:01 smithi079 podman[93707]: 2024-04-22 06:59:01.084622039 +0000 UTC m=+4.159090012 container cleanup 23d4fa0bb5372a69dc3d50260b42408ca7a8876d0a340bfee03bc6e31f8c593b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T06:59:01.366 INFO:journalctl@ceph.mds.g.smithi079.stdout:Apr 22 06:59:01 smithi079 bash[93707]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-g 2024-04-22T06:59:01.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.g.service' 2024-04-22T06:59:01.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:01.535 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-22T06:59:01.535 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-22T06:59:01.535 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.j 2024-04-22T06:59:01.739 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:59:01 smithi079 systemd[1]: Stopping Ceph mds.j for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:02.239 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:59:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j[65623]: 2024-04-22T06:59:01.846+0000 7feaea0a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:02.239 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:59:01 smithi079 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j[65623]: 2024-04-22T06:59:01.846+0000 7feaea0a0700 -1 mds.j *** got signal Terminated *** 2024-04-22T06:59:05.169 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:59:04 smithi079 podman[93827]: 2024-04-22 06:59:04.908815703 +0000 UTC m=+3.156337806 container died 291f0baadb8bf8e12b0b2abff11033ad3a100ed7fcad460e74a538b013faab72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T06:59:05.490 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:59:05 smithi079 podman[93827]: 2024-04-22 06:59:05.262812218 +0000 UTC m=+3.510334308 container cleanup 291f0baadb8bf8e12b0b2abff11033ad3a100ed7fcad460e74a538b013faab72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:59:05.490 INFO:journalctl@ceph.mds.j.smithi079.stdout:Apr 22 06:59:05 smithi079 bash[93827]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-j 2024-04-22T06:59:05.694 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.j.service' 2024-04-22T06:59:05.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:05.724 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-22T06:59:05.724 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-22T06:59:05.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.b 2024-04-22T06:59:06.035 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:05 smithi190 systemd[1]: Stopping Ceph mds.b for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:06.406 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: 2024-04-22T06:59:06.036+0000 7f1bf3aa0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:06.407 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:06 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b[62333]: 2024-04-22T06:59:06.036+0000 7f1bf3aa0700 -1 mds.b *** got signal Terminated *** 2024-04-22T06:59:10.352 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:10 smithi190 podman[70397]: 2024-04-22 06:59:10.057950698 +0000 UTC m=+4.134719503 container died bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:59:10.702 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:10 smithi190 podman[70397]: 2024-04-22 06:59:10.545089247 +0000 UTC m=+4.621858062 container cleanup bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0) 2024-04-22T06:59:10.703 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:10 smithi190 bash[70397]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b 2024-04-22T06:59:10.969 INFO:journalctl@ceph.mds.b.smithi190.stdout:Apr 22 06:59:10 smithi190 podman[70423]: 2024-04-22 06:59:10.721726613 +0000 UTC m=+0.664936836 container remove bdeb2a276527411adec259563de3d6b829e9add5364718c08f948c86209882e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:59:10.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.b.service' 2024-04-22T06:59:11.008 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:11.008 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-22T06:59:11.009 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-22T06:59:11.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e 2024-04-22T06:59:11.347 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:59:11 smithi190 systemd[1]: Stopping Ceph mds.e for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:11.702 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:59:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[68125]: 2024-04-22T06:59:11.348+0000 7fa0b4445700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:11.702 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:59:11 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e[68125]: 2024-04-22T06:59:11.348+0000 7fa0b4445700 -1 mds.e *** got signal Terminated *** 2024-04-22T06:59:14.681 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:59:14 smithi190 podman[70531]: 2024-04-22 06:59:14.358946961 +0000 UTC m=+3.133286019 container died ea1358230f531fba5e6e823a4339b85959f89944b3f4e4b1ec56a11f186aafbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:59:14.952 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:59:14 smithi190 podman[70531]: 2024-04-22 06:59:14.765562534 +0000 UTC m=+3.539901582 container cleanup ea1358230f531fba5e6e823a4339b85959f89944b3f4e4b1ec56a11f186aafbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:59:14.952 INFO:journalctl@ceph.mds.e.smithi190.stdout:Apr 22 06:59:14 smithi190 bash[70531]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-e 2024-04-22T06:59:15.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.e.service' 2024-04-22T06:59:15.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:15.228 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-22T06:59:15.229 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-22T06:59:15.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h 2024-04-22T06:59:15.702 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:59:15 smithi190 systemd[1]: Stopping Ceph mds.h for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:15.702 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:59:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[67485]: 2024-04-22T06:59:15.535+0000 7efe808d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:15.702 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:59:15 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h[67485]: 2024-04-22T06:59:15.535+0000 7efe808d8700 -1 mds.h *** got signal Terminated *** 2024-04-22T06:59:18.702 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:59:18 smithi190 podman[70654]: 2024-04-22 06:59:18.368051248 +0000 UTC m=+2.925599960 container died 282b8251bd3244d42e2f064d4a3de60d5607e44ecf56735abca31eb4610bf4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T06:59:19.093 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:59:18 smithi190 podman[70654]: 2024-04-22 06:59:18.844597873 +0000 UTC m=+3.402146585 container cleanup 282b8251bd3244d42e2f064d4a3de60d5607e44ecf56735abca31eb4610bf4ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:59:19.093 INFO:journalctl@ceph.mds.h.smithi190.stdout:Apr 22 06:59:18 smithi190 bash[70654]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-h 2024-04-22T06:59:19.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.h.service' 2024-04-22T06:59:19.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:19.247 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-22T06:59:19.247 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-22T06:59:19.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.k 2024-04-22T06:59:19.452 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:59:19 smithi190 systemd[1]: Stopping Ceph mds.k for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:19.952 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:59:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k[61398]: 2024-04-22T06:59:19.556+0000 7fdb2de19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:19.952 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:59:19 smithi190 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k[61398]: 2024-04-22T06:59:19.556+0000 7fdb2de19700 -1 mds.k *** got signal Terminated *** 2024-04-22T06:59:22.202 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:59:21 smithi190 podman[70775]: 2024-04-22 06:59:21.789499907 +0000 UTC m=+2.326832736 container died 77a8771c3b90284d404df7fa1329be9c793bc49274620cc7a95a78e825f9b723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:59:22.621 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:59:22 smithi190 podman[70775]: 2024-04-22 06:59:22.356751783 +0000 UTC m=+2.894084617 container cleanup 77a8771c3b90284d404df7fa1329be9c793bc49274620cc7a95a78e825f9b723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T06:59:22.622 INFO:journalctl@ceph.mds.k.smithi190.stdout:Apr 22 06:59:22 smithi190 bash[70775]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-k 2024-04-22T06:59:22.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.k.service' 2024-04-22T06:59:22.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:22.773 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-22T06:59:22.773 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-22T06:59:22.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c 2024-04-22T06:59:23.161 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:59:22 smithi193 systemd[1]: Stopping Ceph mds.c for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:23.416 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:59:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[69808]: 2024-04-22T06:59:23.162+0000 7f3f0666b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:23.416 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:59:23 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c[69808]: 2024-04-22T06:59:23.162+0000 7f3f0666b700 -1 mds.c *** got signal Terminated *** 2024-04-22T06:59:26.689 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:59:26 smithi193 podman[71915]: 2024-04-22 06:59:26.359929654 +0000 UTC m=+3.390387557 container died 0bbcacc172bdf61c59f511c804c5de427223b56c8b5791dbc626887ae2a141c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:59:27.103 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:59:26 smithi193 podman[71915]: 2024-04-22 06:59:26.879524059 +0000 UTC m=+3.909981962 container cleanup 0bbcacc172bdf61c59f511c804c5de427223b56c8b5791dbc626887ae2a141c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:59:27.103 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 22 06:59:26 smithi193 bash[71915]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-c 2024-04-22T06:59:27.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.c.service' 2024-04-22T06:59:27.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:27.233 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-22T06:59:27.233 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-22T06:59:27.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f 2024-04-22T06:59:27.439 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:59:27 smithi193 systemd[1]: Stopping Ceph mds.f for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:27.940 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:59:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[69527]: 2024-04-22T06:59:27.533+0000 7fd48d715700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:27.940 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:59:27 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f[69527]: 2024-04-22T06:59:27.533+0000 7fd48d715700 -1 mds.f *** got signal Terminated *** 2024-04-22T06:59:30.632 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:59:30 smithi193 podman[72047]: 2024-04-22 06:59:30.349765362 +0000 UTC m=+2.899977216 container died c795b8988ff894629852104cd73c34143b817b975b9811096144a00de581786c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-22T06:59:30.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:59:30 smithi193 podman[72047]: 2024-04-22 06:59:30.734216868 +0000 UTC m=+3.284428727 container cleanup c795b8988ff894629852104cd73c34143b817b975b9811096144a00de581786c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T06:59:30.939 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 22 06:59:30 smithi193 bash[72047]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-f 2024-04-22T06:59:31.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.f.service' 2024-04-22T06:59:31.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:31.144 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-22T06:59:31.144 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-22T06:59:31.145 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i 2024-04-22T06:59:31.689 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:59:31 smithi193 systemd[1]: Stopping Ceph mds.i for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:31.689 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:59:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[68710]: 2024-04-22T06:59:31.461+0000 7fe043bd9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:31.689 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:59:31 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i[68710]: 2024-04-22T06:59:31.461+0000 7fe043bd9700 -1 mds.i *** got signal Terminated *** 2024-04-22T06:59:34.652 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:59:34 smithi193 podman[72167]: 2024-04-22 06:59:34.371052047 +0000 UTC m=+3.013739238 container died 3323c3a73c73d8bdbe5cf94a182950b337b599e3055c480da0e5ffc313536bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T06:59:34.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:59:34 smithi193 podman[72167]: 2024-04-22 06:59:34.754112387 +0000 UTC m=+3.396799566 container cleanup 3323c3a73c73d8bdbe5cf94a182950b337b599e3055c480da0e5ffc313536bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T06:59:34.939 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 22 06:59:34 smithi193 bash[72167]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-i 2024-04-22T06:59:35.099 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.i.service' 2024-04-22T06:59:35.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:35.131 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-22T06:59:35.131 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-22T06:59:35.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.l 2024-04-22T06:59:35.439 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:59:35 smithi193 systemd[1]: Stopping Ceph mds.l for 146994a8-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:35.939 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:59:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l[62320]: 2024-04-22T06:59:35.465+0000 7f58ae82e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:35.939 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:59:35 smithi193 ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l[62320]: 2024-04-22T06:59:35.465+0000 7f58ae82e700 -1 mds.l *** got signal Terminated *** 2024-04-22T06:59:39.123 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:59:38 smithi193 podman[72290]: 2024-04-22 06:59:38.786190014 +0000 UTC m=+3.435951721 container died 54adb40b1d86167d7baf35d9d721d502f12028f143fc747fe34fb0e56c1f8bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-22T06:59:39.389 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:59:39 smithi193 podman[72290]: 2024-04-22 06:59:39.233328859 +0000 UTC m=+3.883090554 container cleanup 54adb40b1d86167d7baf35d9d721d502f12028f143fc747fe34fb0e56c1f8bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:59:39.389 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 22 06:59:39 smithi193 bash[72290]: ceph-146994a8-0072-11ef-bc93-c7b262605968-mds-l 2024-04-22T06:59:39.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-146994a8-0072-11ef-bc93-c7b262605968@mds.l.service' 2024-04-22T06:59:39.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:39.629 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-22T06:59:39.629 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 146994a8-0072-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:59:39.816 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:59:42.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 146994a8-0072-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:59:42.537 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:59:45.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 146994a8-0072-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:59:45.239 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T06:59:47.563 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:59:47.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:59:47.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:59:47.653 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T06:59:47.655 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi079/crash 2024-04-22T06:59:47.656 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:59:47.683 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:59:47.683 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:59:47.685 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi190/crash 2024-04-22T06:59:47.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:59:47.806 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:59:47.806 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:59:47.808 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi193/crash 2024-04-22T06:59:47.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:59:47.847 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/146994a8-0072-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:59:47.847 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:59:47.848 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T06:59:47.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-22T06:59:47.890 INFO:tasks.cephadm:Compressing logs... 2024-04-22T06:59:47.891 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:59:47.933 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:59:47.936 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:59:47.959 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:59:47.959 INFO:teuthology.orchestra.run.smithi193.stderr:: No such file or directory 2024-04-22T06:59:47.960 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:59:47.960 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T06:59:47.960 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-22T06:59:47.961 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr: 90.4% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:59:47.966 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-22T06:59:47.967 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.2% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:59:47.967 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-22T06:59:47.976 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-22T06:59:47.976 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T06:59:47.977 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-22T06:59:47.978 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:59:47.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.a.log: 84.4%gzip -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log.gz -5 2024-04-22T06:59:47.980 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-22T06:59:47.980 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-22T06:59:47.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.d.log: 87.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:59:47.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.9.log: 90.3% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:59:47.985 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-22T06:59:48.006 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.6% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-22T06:59:48.016 INFO:teuthology.orchestra.run.smithi193.stderr: 91.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:59:48.035 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:59:48.065 INFO:teuthology.orchestra.run.smithi079.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:59:48.072 INFO:teuthology.orchestra.run.smithi190.stderr: 92.0% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T06:59:48.080 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:59:48.099 INFO:teuthology.orchestra.run.smithi193.stderr: 92.3% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-22T06:59:48.112 INFO:teuthology.orchestra.run.smithi193.stderr: 91.9% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-22T06:59:48.191 INFO:teuthology.orchestra.run.smithi079.stderr: 94.7% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-22T06:59:48.212 INFO:teuthology.orchestra.run.smithi079.stderr: 94.4% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-22T06:59:48.216 INFO:teuthology.orchestra.run.smithi079.stderr: 94.5% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-22T06:59:48.242 INFO:teuthology.orchestra.run.smithi079.stderr: 91.0% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:59:48.266 INFO:teuthology.orchestra.run.smithi079.stderr: 94.6% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-22T06:59:48.326 INFO:teuthology.orchestra.run.smithi190.stderr: 84.3% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:59:48.337 INFO:teuthology.orchestra.run.smithi193.stderr: 91.9% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T06:59:48.364 INFO:teuthology.orchestra.run.smithi190.stderr: 87.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:59:48.377 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:59:48.763 INFO:teuthology.orchestra.run.smithi079.stderr: 90.3% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T06:59:48.961 INFO:teuthology.orchestra.run.smithi079.stderr: 90.4% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T06:59:50.168 INFO:teuthology.orchestra.run.smithi190.stderr: 91.9% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T06:59:50.186 INFO:teuthology.orchestra.run.smithi190.stderr: 91.9% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-22T06:59:50.193 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-22T06:59:57.181 INFO:teuthology.orchestra.run.smithi193.stderr: 97.5% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-22T07:00:22.380 INFO:teuthology.orchestra.run.smithi193.stderr: 93.2% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-22T07:00:28.248 INFO:teuthology.orchestra.run.smithi193.stderr: 93.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T07:00:28.855 INFO:teuthology.orchestra.run.smithi193.stderr: 93.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-22T07:00:29.680 INFO:teuthology.orchestra.run.smithi079.stderr: 93.2% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-22T07:00:29.842 INFO:teuthology.orchestra.run.smithi193.stderr: 93.2% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T07:00:29.844 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-04-22T07:00:29.844 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m41.894s 2024-04-22T07:00:29.844 INFO:teuthology.orchestra.run.smithi193.stderr:user 2m41.424s 2024-04-22T07:00:29.844 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m5.659s 2024-04-22T07:00:30.551 INFO:teuthology.orchestra.run.smithi079.stderr: 93.0% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T07:00:38.081 INFO:teuthology.orchestra.run.smithi079.stderr: 93.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T07:00:45.286 INFO:teuthology.orchestra.run.smithi079.stderr: 93.0% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T07:00:45.288 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-04-22T07:00:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m57.342s 2024-04-22T07:00:45.288 INFO:teuthology.orchestra.run.smithi079.stderr:user 2m28.481s 2024-04-22T07:00:45.289 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m5.289s 2024-04-22T07:00:47.504 INFO:teuthology.orchestra.run.smithi190.stderr: 93.2% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-22T07:00:49.058 INFO:teuthology.orchestra.run.smithi190.stderr: 93.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T07:01:01.356 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T07:01:02.575 INFO:teuthology.orchestra.run.smithi190.stderr: 93.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T07:01:04.552 INFO:teuthology.orchestra.run.smithi190.stderr: 94.1% -- replaced with /var/log/ceph/146994a8-0072-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-22T07:01:04.555 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-04-22T07:01:04.555 INFO:teuthology.orchestra.run.smithi190.stderr:real 1m16.604s 2024-04-22T07:01:04.555 INFO:teuthology.orchestra.run.smithi190.stderr:user 4m9.339s 2024-04-22T07:01:04.555 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m8.556s 2024-04-22T07:01:04.555 INFO:tasks.cephadm:Archiving logs... 2024-04-22T07:01:04.556 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi079/log 2024-04-22T07:01:04.557 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:01:28.966 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi190/log 2024-04-22T07:01:28.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:02:04.289 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi193/log 2024-04-22T07:02:04.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:02:28.712 INFO:tasks.cephadm:Removing cluster... 2024-04-22T07:02:28.712 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 146994a8-0072-11ef-bc93-c7b262605968 --force 2024-04-22T07:02:28.902 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T07:02:29.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 146994a8-0072-11ef-bc93-c7b262605968 --force 2024-04-22T07:02:29.514 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T07:02:30.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 146994a8-0072-11ef-bc93-c7b262605968 --force 2024-04-22T07:02:30.221 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 146994a8-0072-11ef-bc93-c7b262605968 2024-04-22T07:02:30.922 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T07:02:30.922 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T07:02:30.941 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T07:02:30.960 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T07:02:30.975 INFO:tasks.cephadm:Teardown complete 2024-04-22T07:02:30.976 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T07:02:30.996 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T07:02:30.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:02:30.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:02:31.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:02:31.171 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T07:02:31.199 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T07:02:31.199 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:02:31.212 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:02:31.229 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:02:31.230 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:02:31.232 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:02:31.232 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-04-22T07:02:31.232 INFO:teuthology.orchestra.run.smithi079.stdout:^- hv01.front.sepia.ceph.com 2 6 377 61 +1241us[+1241us] +/- 583ms 2024-04-22T07:02:31.233 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -51us[ -34us] +/- 55ms 2024-04-22T07:02:31.233 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +105us[ +105us] +/- 90ms 2024-04-22T07:02:31.233 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:02:31.244 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:02:31.249 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:02:31.253 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:02:31.253 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-04-22T07:02:31.253 INFO:teuthology.orchestra.run.smithi193.stdout:^- hv01.front.sepia.ceph.com 2 6 377 63 +1223us[+1226us] +/- 583ms 2024-04-22T07:02:31.253 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -35us[ -32us] +/- 55ms 2024-04-22T07:02:31.253 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +84us[ +87us] +/- 90ms 2024-04-22T07:02:31.253 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:02:31.269 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:02:31.270 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-04-22T07:02:31.270 INFO:teuthology.orchestra.run.smithi190.stdout:^- hv01.front.sepia.ceph.com 2 6 377 61 +1249us[+1249us] +/- 583ms 2024-04-22T07:02:31.270 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -24us[+5391ns] +/- 55ms 2024-04-22T07:02:31.270 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +108us[ +108us] +/- 90ms 2024-04-22T07:02:31.270 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:02:31.271 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T07:02:31.347 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T07:02:31.348 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T07:02:31.372 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:02:31.519 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:02:31.604 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:02:31.669 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:02:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:02:31.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:02:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:02:31.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:02:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:02:31.769 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T07:02:32.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T07:02:32.302 INFO:teuthology.task.internal:Duration was 2183.518777 seconds 2024-04-22T07:02:32.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T07:02:32.400 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T07:02:32.400 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:02:32.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:02:32.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:02:32.449 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:02:32.469 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:02:32.480 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:02:32.793 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T07:02:32.793 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-04-22T07:02:32.793 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:02:32.815 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-04-22T07:02:32.815 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:02:32.849 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-04-22T07:02:32.849 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:02:32.872 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T07:02:32.872 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:02:32.875 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:02:32.891 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:02:32.938 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T07:02:32.939 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:02:32.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:02:32.960 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:02:33.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T07:02:33.350 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T07:02:33.350 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:02:33.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:02:33.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:02:33.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T07:02:33.447 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:02:33.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:02:33.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:02:33.472 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-04-22T07:02:33.474 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-04-22T07:02:33.496 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-04-22T07:02:33.516 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:02:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:02:33.540 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:02:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:02:33.555 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:02:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:02:33.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T07:02:33.583 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T07:02:33.584 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi079 2024-04-22T07:02:33.584 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:02:33.624 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi190 2024-04-22T07:02:33.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:02:33.664 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137/remote/smithi193 2024-04-22T07:02:33.664 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:02:33.700 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T07:02:33.700 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:02:33.702 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:02:33.704 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:02:33.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T07:02:33.765 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T07:02:33.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T07:02:33.784 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T07:02:33.784 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:02:33.786 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:02:33.789 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:02:33.798 INFO:teuthology.orchestra.run.smithi079.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:02 /home/ubuntu/cephtest 2024-04-22T07:02:33.801 INFO:teuthology.orchestra.run.smithi190.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:02 /home/ubuntu/cephtest 2024-04-22T07:02:33.803 INFO:teuthology.orchestra.run.smithi193.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:02 /home/ubuntu/cephtest 2024-04-22T07:02:33.808 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T07:02:33.835 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T07:02:33.968 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T07:02:34.000 DEBUG:teuthology.nuke:shortname: smithi079 2024-04-22T07:02:34.000 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:02:34.038 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:30.323444', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDqpX1tV8qLz7FHOEAaPXsOUj0+aWaXrJ228ZMBOOU9N1EmTyApBwBxcMNvotZwmp6kHpWAM8HaXOR5PmRjQVs='} 2024-04-22T07:02:34.050 DEBUG:teuthology.nuke:shortname: smithi190 2024-04-22T07:02:34.050 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:02:34.089 DEBUG:teuthology.nuke:shortname: smithi193 2024-04-22T07:02:34.090 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:02:34.102 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:30.325273', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+xG0cSY8uVsMvim/zyVATK+HGTebLR/+oGDEHCkRIx9f4OI9qQgyY8f7FpixNJFFKsZ+dqFo3a1l5pEK7rnRw='} 2024-04-22T07:02:34.146 INFO:teuthology.orchestra.console.smithi079:Power off 2024-04-22T07:02:34.146 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:02:34.157 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668137', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:30.327447', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhUBvAv2vykhAHKa3RdmGhLlQ+UDmD0N+2VX5/o9iSH01EPOkFlDB0qtr26IK1ElKTOyluERVaJ7jo3IWabb3U='} 2024-04-22T07:02:34.200 INFO:teuthology.orchestra.console.smithi190:Power off 2024-04-22T07:02:34.201 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:02:34.222 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-04-22T07:02:34.222 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:34.249 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-04-22T07:02:34.250 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:34.264 INFO:teuthology.orchestra.console.smithi193:Power off 2024-04-22T07:02:34.264 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:02:34.273 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-22T07:02:34.282 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-04-22T07:02:34.311 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-04-22T07:02:34.311 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:34.342 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-04-22T07:02:38.274 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:38.282 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:38.344 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:38.384 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-04-22T07:02:38.392 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-04-22T07:02:38.453 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-04-22T07:02:42.389 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:42.397 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:42.454 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:42.500 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-04-22T07:02:42.506 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-04-22T07:02:42.563 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-04-22T07:02:42.601 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-04-22T07:02:42.607 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-04-22T07:02:46.568 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:02:46.682 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-04-22T07:02:46.783 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-04-22T07:02:46.889 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} duration: 2183.518776655197 failure_reason: 'error during scrub thrashing: Command failed on smithi079 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls''' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-22T07:02:46.889 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T07:02:46.983 INFO:teuthology.run:FAIL